Test Goals

• Decouple
  
  • With the "Tame the Monolith" project, our long-term vision is to make apps self-contained and loosely coupled, with the eventual goal of moving (some) into separate repos, and possibly into independent deployment units if desired.
  • To achieve this end, we need to decouple our tests so that tests within a particular self-contained app are just for that app.
• Reduction
  • The "Tame the Monolith" project forces us to proactively define seams and boundaries and therefore, also ownership of tests.  In so doing, we hope to assess individual tests and determine whether they provide any additional value over other tests.  This directly addresses an existing developer pain point with an over abundance of tests that need to be maintained.
  • This reduction process would be part of a deeper cleaning effort and can be done on an app-by-app basis, with adequate tools and best practices.

Proposal for discussion

Extracted from a previous brainstorming session (edx-platform Code Structure: Hackathon XIV):

• Unit tests 
  • Properties
    • Do not have dependencies on peer or higher layers.
    • May depend on low-level core libraries.
    • White-box tests of all edge conditions and paths.
  • Guidelines
    • Can use/test internal methods of the module.
    • Use override_settings instead of requiring settings in a central envs/tests.py.
• Integration tests
  • Properties  (Note: this differs from testing.rst.)
    • End-to-end python integration tests that test dependencies between apps.
    • At least one inter-dependency is not mocked.
    • May test multiple paths, including non-happy paths if impacted by inter-dependencies.
  • Guidelines
    
    • Stored centrally
    • Only use exposed public interfaces
• Acceptance tests
  • Properties
    
    • End-to-end UI-level black-box tests.
    • Mostly only happy paths are tested.  Regularly occurring exceptional paths that are integral to the user-facing feature may also be tested.