...

I was able to mostly test this by hacking a mongomock client into the return of mongo_utils.py (connect_to_mongodb) fairly easily. With some additional hacking of GridFS and mongomock I was able to get 90% of CMS tests to run successfully, and I'm fairly confident another day's work would pretty much close that gap as the vast majority of failing tests were DDT tests that should be fixed with the same change, or could simply not utilize the mocking.

Work

• Write a mock to to monkey patch the hacks I put in patch connect_to_mongodb - 1d
• Submit PRs to mongomock or fork and make changes - 2-3d
• Submit PRs to pymongo or fork and make changes - 1-2d
• Identify tests that should be mocked and add mocking to them - 3d (there are hundreds) 
• Investigate and fix up the tests that were failing or in CMS etc. (optional, could just not mock those tests) - 3d

...

• Huge undertaking due to wide API and potential for some direct calls in to internals that may need to be mocked out or changed
• Likely to grow in scope as new access patterns / edge cases are discovered
• Needs to be maintained along with other storage engines
• Less authentic tests, may need to beef up integration tests for explicit backend old / split mongo testing

K/V level mock

...

Work

Savings

 The work here is to replace the CRUD components at a level above Mongo, the work is simpler and the results should hopefully be able the same as mongomock.

Work

• Mock out MongoKeyValueStore - 1d
• Mock out SplitMongoKVS - 1d
• Identify tests that should be mocked and add mocking to them - 3d (there are hundreds) 

Savings

• Estimated total savings across all paver tests ~280 secs, presuming we can get the same kinds of gains as we did with mongomock.

Advantages

• Still uses a lot of modulestore code, tests more authentic
• Probably saves about the same amount of time as mocking Mongo
• Much smaller API footprint to mock out than a full new backend
• No new depsdependencies, no libraries to fork / edit / maintain
• Conceptually simple
• Clear boundaries

Disadvantages

• Leaves a lot of modulestore test timeProbably still a fair amount of work
• Depending on usage may need an outside memory store (or just use memcached if it's there?)
• May need different implementations for different modulestore backends to get full benefits, potentially tripling the work

Mid-level mock (one of the mixins?) (3, 4)

Work

Savings

Advantages

• Saves more test time than k/v or Mongo
• Perhaps more flexibility?

Disadvantages

• Boundaries fuzzy, possibility of more unintended side-effects including mock/non-mock conflicts
• Saves less time than a new modulestore backend
• May just be a superset of the k/v workNeed to maintain two different sets of mocks