Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

There are a handful of places where we might be able to mock out modulestore to save unit test time. Here are the results of my investigation into those places and the estimated work / time savings for each. Due to time limitations and the limitations of the hotshot profile that Nose uses I restricted my benchmarking to CMS tests.

Mongomock at the split mongo layer

I was able to test this by hacking a mongomock client to replace self.database in mongo_connection.py (MongoConnection.__init__) fairly easily and making some changes in mongomock. All but a dozen CMS tests succeeded with these changes in place. Unfortunately benefits weren't great due to it only impacting split mongo tests, not mocking out GridFS, etc. 

Work

  • Write a mock to to monkey patch the hack I put in - 1d
  • Submit PRs to mongomock or fork and make changes - 1-2d
  • Identify tests that should be mocked and add mocking to them - 2d (there are hundreds) 
  • Investigate and fix up the dozen or so tests that were failing (optional, could just not mock those tests) - 1d

Savings

  • 10-20 seconds on a local run of CMS-only tests
  • Estimated total across all paver tests ~80 seconds.

Advantages

  • Still uses a lot of modulestore code, tests more authentic
  • Conceptually simple
  • Mongomock handles almost everything we need for this
  • Clear boundaries

Disadvantages

  • Need to submit some PRs to, or fork, mongomock to make some necessary bug fixes (one example)
  • New tests might require incur more mongomock support costs as it is still fairly young
  • Leaves a lot of time on the table (modulestore, but not mongo, not in split, gridfs, etc)
  • Potentially hard to segregate tests w and w/o mock


Mongomock at the mongo connection layer

I was able to mostly test this by hacking a mongomock client into the return of mongo_utils.py (connect_to_mongodb) fairly easily. With some additional hacking of GridFS and mongomock I was able to get 90% of CMS tests to run successfully, and I'm fairly confident another day's work would pretty much close that gap as the vast majority of failing tests were DDT tests that should be fixed with the same change, or could simply not utilize the mocking.

Work

  • Write a mock to to monkey patch the hacks I put in - 1d
  • Submit PRs to mongomock or fork and make changes - 2-3d
  • Submit PRs to pymongo or fork and make changes - 1-2d
  • Identify tests that should be mocked and add mocking to them - 3d (there are hundreds) 
  • Investigate and fix up the tests that were failing or in CMS etc. (optional, could just not mock those tests) - 3d

Savings

  • ~70 seconds on a local run of CMS-only tests
  • Estimated total savings across all paver tests ~280 secs

Advantages

  • Saves quite a bit more time than mocking at the split layer
  • Still uses a lot of modulestore code, tests more authentic
  • Conceptually simple
  • Mongomock handles a lot of what we need for this
  • Clear boundaries

Disadvantages

  • Need to submit some PRs to, or fork, mongomock to make some necessary bug fixes (one example), new tests might require incur more mongomock support costs
  • Need some assistance from Mongo for GridFS fixup, or to fork pymongo and fix it ourselves
  • Still leaves a lot of time on the table (modulestore code outside of mongo)
  • Potentially hard to segregate tests w and w/o mock


New modulestore backend

Work


Savings


Advantages

  • Saves the most test time
  • Clear boundaries

Disadvantages

  • Huge undertaking due to wide API and some direct calls to internals
  • Needs to be maintained along with other engines
  • Less authentic tests, may need to beef up integration tests for explicit backend testing


K/V level mock (2, 3)


Work


Savings


Advantages

  • Still uses a lot of modulestore code, tests more authentic
  • Probably saves about the same amount of time as mocking Mongo
  • Much smaller API footprint to mock out1
  • Can be used selectively (and explicitly) in a decorator or context manager
  • No new deps

Disadvantages

  • Leaves a lot of modulestore test time
  • Probably still a fair amount of work
  • Depending on usage may need an outside memory store (or just use memcached if it's there?)
  • May need different implementations for different modulestore backends to get full benefits, potentially tripling the work


Mid-level mock (one of the mixins?) (3, 4)

Work


Savings


Advantages

  • Saves more test time than k/v or Mongo
  • Perhaps more flexibility?

Disadvantages

  • Boundaries fuzzy, possibility of more unintended side-effects including mock/non-mock conflicts
  • Saves less time than a new modulestore backend
  • May just be a superset of the k/v work
  • No labels