Tests: Pain Points Discovery, 2020-03-06

Story Mapping based on Discovery Workshop

Mural With Story Mapping of Action Plan

Discovery workshop with Stacey

Full Board

Affinitized Categories

What’s going well is…

I Don’t Trust

What’s slowing me down is…

I want to help with…

Action Plan

Major Pains (Summary)

  • Lack of overall test strategy makes things hard.

  • Lack of documentation of existing features makes it hard to make changes.

    • "How is this supposed to work?"

  • Don't know when to use what kind of tests? How do I know what I need in
    terms of testing?

  • How do I run tests locally? There are so many different kinds of tests and
    it's not clear how to run them all locally.

  • Things are so different across repos and even within them, I'm not sure
    what the right way forward is.

  • Non-Production Environments are not a good proxies for the production environment.

    • Both in terms of data and which features are enabled.

  • Waiting for CI

  • Not Trusting CI

Potential Solutions

  • Re-write testing.rst in edx-platform to be very clear about how to run any
    given singular or small groups of tests.

  • Document via ADR/OEP what kinds of testing is available and when to use any
    given kind. Including examples of the various different kinds of tests.

  • Make settings more consistent across environments.

    • Regularly review prod/stage/dev config and sync feature flags across them.

      • Remote config should make this pretty easy.

      • Over time we can automate and reduce cost of doing this.

    • Toggle visibility work we’ve already been talking about aligns well with this.

  • Improve UX of feedback from Jenkins

    • Provide a test summary on the PR of which tests failed so people don’t have to go to jenkins.

  • Work with product to document how features should work.

    • Build out some examples of how this would look and push for more of this documentation from product/ux?

    • Provide a template for how features should work.