There are a handful of places where we might be able to mock out modulestore to save unit test time. Here are the results of my investigation into those places and the estimated work / time savings for each. Due to time limitations and the limitations of the hotshot profile that Nose uses I restricted my benchmarking to CMS tests.

Mongomock at the split mongo layer

I was able to test this by hacking a mongomock client to replace self.database in mongo_connection.py (MongoConnection.__init__) fairly easily and making some changes in mongomock. All but a dozen CMS tests succeeded with these changes in place. Unfortunately benefits weren't great due to it only impacting split mongo tests, not mocking out GridFS, etc. 

Work

• Write a mock to to monkey patch the hack I put in - 1d
• Submit PRs to mongomock or fork and make changes - 1-2d
• Identify tests that should be mocked and add mocking to them - 2d (there are hundreds) 
• Investigate and fix up the dozen or so tests that were still failing (optional, could just not mock those tests) - 1d

Savings

• 10-20 seconds on a local run of CMS-only tests
• Estimated total across all paver tests ~80 seconds.

Advantages

• Still uses a lot of modulestore code, tests more authentic
• Conceptually simple
• Mongomock handles almost everything we need for this

Disadvantages

• Need to submit some PRs to, or fork, mongomock to make some necessary bug fixes (one example)
• New tests might require incur more mongomock support costs as it is still fairly young
• Leaves a lot of time on the table (modulestore, but not mongo, not in split, gridfs, etc)
• Potentially hard to segregate tests w and w/o mock

Mongomock at the mongo connection layer

I was able to mostly test this by hacking a mongomock client into the return of mongo_utils.py (connect_to_mongodb) fairly easily. With some additional hacking of GridFS and mongomock I was able to get 90% of CMS tests to run successfully, and I'm fairly confident another day's work would pretty much close that gap as the vast majority of failing tests were DDT tests that should be fixed with the same change, or could simply not utilize the mocking.

Work

• Write a mock to patch connect_to_mongodb - 1d
• Submit PRs to mongomock or fork and make changes - 2-3d
• Submit PRs to pymongo or fork and make changes - 1-2d
• Identify tests that should be mocked and add mocking to them - 3d (there are hundreds) 
• Investigate and fix up the tests that were failing or in CMS etc. (optional, could just not mock those tests) - 3d

Savings

• ~70 seconds on a local run of CMS-only tests
• Estimated total savings across all paver tests ~280 secs

Advantages

• Saves quite a bit more time than mocking at the split layer
• Still uses a lot of modulestore code, tests more authentic
• Mongomock handles a lot of what we need for this
• Clear boundaries

Disadvantages

• Need to submit some PRs to, or fork, mongomock to make some necessary bug fixes (one example), new tests might require incur more mongomock support costs
• Need some assistance from Mongo for GridFS fixup, or to fork pymongo and fix it ourselves
• Still leaves a lot of time on the table (modulestore code outside of mongo)
• Potentially hard to segregate tests w and w/o mock

New modulestore backend

Work

• Create a new (in local memory?) modulestore backend that can behave like old mongo and split mongo as necessary (~15 days?)
  
  • ~40 public methods in old mongo backend, ~60 in split
  • May take additional time / iterations to realize the full savings
• Identify tests that should be mocked and add mocking to them - 3d (there are hundreds) 

Savings

• Potential savings of 10 min plus on a full paver test run
  • Depending on how much savings we can squeeze from skipping various to/from Mongo transformations, and whether we can fit everything we need to in local memory

Advantages

• Saves the most test time

Disadvantages

• Huge undertaking due to wide API and potential for some direct calls in to internals that may need to be mocked out or changed
• Likely to grow in scope as new access patterns / edge cases are discovered
• Needs to be maintained along with other storage engines
• Less authentic tests, may need to beef up integration tests for explicit backend old / split mongo testing

K/V level mock

Mocking out the k/v store was suggested as a possible place to look at. Presuming I'm looking at the right places (InheritanceKeyValueStore / MongoKeyValueStore / SplitMongoKVS) it doesn't seem to be a place where we are spending much time. Further it seems like there are no clear optimizations to be made in here as the data seems to already be stored fundamentally in local memory as a dict. The time spent in all of the KVSs is somewhere in the ~10-15 second range for CMS tests. It's unclear how we could improve on that in a mock or what the gains might be.