There are a handful of places where we might be able to mock out modulestore to save unit test time. Here are the results of my investigation into those places and the estimated work / time savings for each. Due to time limitations and the limitations of the hotshot profile that Nose uses I restricted my benchmarking to CMS tests.
| Table of Contents |
|---|
Mongomock at the split mongo layer
...
- Need to submit some PRs to, or fork, mongomock to make some necessary bug fixes (one example), new tests might require incur more mongomock support costs
- Need some assistance from Mongo for GridFS fixup, or to fork pymongo and fix it ourselves
- Still leaves a lot of time on the table (modulestore code outside of mongo)
- Potentially hard to segregate tests w and w/o mock
New modulestore backend
...
Work
Savings
Advantages
...
- Saves the most test time
...
- Clear boundaries
...
Disadvantages
...
- Huge undertaking due to wide API and some direct calls to internals
...
- Needs to be maintained along with other engines
...
- Less authentic tests, may need to beef up integration tests for explicit backend testing
K/V level mock (2, 3)
Work
Savings
Advantages
...
- Still uses a lot of modulestore code, tests more authentic
...
- Probably saves about the same amount of time as mocking Mongo
...
- Much smaller API footprint to mock out1
...
- Can be used selectively (and explicitly) in a decorator or context manager
...
- No new deps
Disadvantages
...
- Leaves a lot of modulestore test time
...
- Probably still a fair amount of work
...
- Depending on usage may need an outside memory store (or just use memcached if it's there?)
...
- May need different implementations for different modulestore backends to get full benefits, potentially tripling the work
Mid-level mock (one of the mixins?) (3, 4)
Work
Savings
Advantages
...
- Saves more test time than k/v or Mongo
...
- Perhaps more flexibility?
Disadvantages
...
- Boundaries fuzzy, possibility of more unintended side-effects including mock/non-mock conflicts
...
- Saves less time than a new modulestore backend
...
- May just be a superset of the k/v work