...
- Identify tests to be evaluated (all tests that execute modulestore code) and categorize them into: integration tests where mocking might make sense, modulestore-specific tests with limited scope, and anything else
- See "Tests that need mocking" in the cms / lms sections
- Estimate of time taken by all tests that touch any modulestore code
- In CMS at least 68% of test time is happening in modulestore code (361 out of 529 secs). This is probably higher, but it's difficult to estimate how much.
- LMS is challenging to profile, but contains 2-5x times more calls to the most time-consuming calls present in the CMS profile so efforts to improve those times should have an even greater impact on LMS
- In just the LMS djangoapps 848 seconds out of 1243 was spent in modulestore code
- Profiled runs of cms and a subset of lms paver tests can be found in the attachments to this page!
- Estimate of time taken by tests that span multiple subsystems but not the modulestore specifically
- This is captured at a high level in in the timings. To really figure this out we'd need to look through hundreds or thousands of tests to see what the specific intent of each test was. Looking at about 100 random-ish test cases across some of the top time consuming subsystems it looks like almost all of the tests are using modulestore incidentally and are not testing integration.
- The subsystems I looked at were ones that showed up as having taken a lot of time in the unit tests:
- lms/ccx
- core/content
- cms/djangoapps/contentstore
- lms/djangoapps/certificates
- common/djangoapps/util
- openedx/core/djangoapps/course_groups
- openedx/core/djangoapps/models
- openedx/core/djangoapps/bookmarks
- openedx/core/djangoapps/maintenance
- At least ccx and contentstore did have integration tests, however, and we would need to make allowances for subsystems to bypass any mocking and use a real modulestore for integration test purposes
- Estimate of time taken by tests that are limited to testing modulestore and closely-related code
- About 410 seconds are spent on a local run of just the modulestore tests (not under profiler)
- The most time consuming tests inside ccx seemed to not be the integration tests
- Significant time is also taken by contentstore tests that relate to moduestore
- Capture which tests specify use of old mongo versus split, or neither
- See "Tests that need mocking" in the cms / lms sections below
...