...

• Identify tests to be evaluated (all tests that execute modulestore code) and categorize them into: integration tests where mocking might make sense, modulestore-specific tests with limited scope, and anything else
  • See "Tests that need mocking" in the cms / lms sections
• Estimate of time taken by all tests that touch any modulestore code
  • In CMS at least 68% of test time is happening in modulestore code (361 out of 529 secs). This is probably higher, but it's difficult to estimate how much.
  • LMS is challenging to profile, but contains several 2-5x times more calls to the most time-consuming calls present in the CMS profile :so efforts to improve those times should have an even greater impact on LMS.
• Estimate of time taken by tests that span multiple subsystems but not the modulestore specifically
  • This is captured at a high level in in the CMS timing, and hopefully I'll be able to get it for LMS as well.
• Estimate of time taken by tests that are limited to testing modulestore and closely-related code
  • About 410 seconds are spent on a local run of just the modulestore tests (not under profiler)
  • Significant time is also taken by contentstore tests that relate to moduestore 
• Capture which tests specify use of old mongo versus split, or neither
  • See "Tests that need mocking" in the cms / lms sections below

...

Both of which are modulestore / Mongo related and would be improved by a mocking solution somewhere in there. Not all of the factories.py calls are from setUp methods, but a significant portion of them are. The factories.py calls also contribute significantly to the overall modulestore time. Teardown times seemed negligible.

Options for speeding up setup include:

• Mocking out modulestore at various levels from the API down to Mongo
• Creating an original of the test data and copying that instead of creating it from scratch every time?
  • Probably not much of a gain, depending on how much time is being spent in Mongo

Bonus Results

26% of CMS test time under profiler (138 seconds) and 38% of modulestore time was spent in Mongo calls. 

...