Release Testing Strategy Brainstorm
Let's analyze the current community release testing strategy and decide whether, based on both product and technical requirements, it warrants improvement. If so, how can we assist the community in doing so? And, in particular, what can we do to:
Help guarantee that every feature in each release is as well-tested as possible, in such a way to minimize the introduction of last-minute bugs;
Help the Product Working Group communicate with whoever is doing the testing, so that the latter effort can be factored into the plan for upcoming releases.
And finally:
How much of the testing should be delegated to the development teams? To the reviewers? To a dedicated testing team?
General testing categories
Feature evaluation/user acceptance testing
Feature validation
Testing at scale
Edge cases
Performance
Part 0. Overview of current edx.org testing
Feature shipped but off by default
“Real world” beta testing
Via course waffle flags
Subset of users that are beta testers
Uses actual courses, with real users
Multi-stage roll-out of new features (for bigger features, usually)
Subset of orgs/courses
Allowlist/denylist for defaulting it to on
Extreme Example: Learning MFE (6 months - 1 year: an extreme case)
End-to-end testing
Cypress
Reverting commits
Because they’re running on master and have control of it
Monitoring + Continuous Delivery
Detecting new errors against specific release versions (down to specific PRs)
No specific philosophy/industry pattern adopted
Coursegraph:
Graph database where modulestore data is dumped
Page of canned queries
Example: “How many people use an LTI tool with this exact URL?”
Evaluate the priority/impact of features/bugs
Possible to replicate variety of content
Bug reporting pipeline (CRs)
Part I. Overview of the current community testing strategy
Links to existing documentation:
Announcement: Join the Test Team for Quince and Meet the Test Champions from Palm
Test Sheet: https://docs.google.com/spreadsheets/d/1oVhrnRjk7zxUHMNyU8bJ-TLKyaSukLmN3jWom17O7pI/edit#gid=0
Project Board: Build-Test-Release Working Group • openedx
Example release blocker: [Test failure] TC_LEARNER_7: Can't create an account · Issue #318 · openedx/wg-build-test-release
Part II. What should probably continue for Redwood?
List the things that have worked well so far, and likely shouldn’t be messed with.
BTR should continue to function as gatekeepers for a release
Part III. What should we change for Redwood?
Product should take ownership of a “master” test spreadsheet
The master version should be ready by the cutoff date
The test manager still instantiates the release-specific version of the spreadsheet
Cut the official testing period in half
Pros: more features land in Redwood
“If the test cycle was smaller, we’d increase the chance of features getting into each release”
We (Axim) know BTR has taken the brunt of it, and are willing to help find resources to get testing done faster (more automation, more documentation, etc)
Do more point releases
On a cadence, or at every (important) backport?
Codify the process on docs.openedx.org
What the test manager needs to do: write this doc proactively (this Friday)
Move from spreadsheet to Github issues?
One issue for each test case
Tracks the whole lifecycle of the issue
Script to convert the product-owned master spreadsheet to issues
Happens at every release
Smart enough to create new issues if new rows are added
Process suggestions
Critical BTR roles should probably rotate mandatorily every two terms
Critical roles should probably have a primary and secondary
Make the test plan/state more visible/discoverable
A “folder” pinned to the Slack channel would be great
Milestones (with dates) should be set