...
The Product Core Working Group should take ownership of a “master” test spreadsheet. By “ownership”, we mean that the Product Core Working Group should be the group ultimately responsible for making sure that the ways in which we test the release align with the Product Core Working Group’s definition of “core product”. That means defining the tests and their relative priorities. BTR is still responsible for building, testing, and releasing the release.
The Product Team proposes to take on the task of writing coherent and sane tests that align with the highest priority features
This should help alleviate the testing burden that has previously been solely on BTR’s plate
The Product Team will prioritize defining test plans through the most important paths through the core product features.
Teams outside of BTR are welcome to make their own test plans for features outside of the core product that they care about; however, those tests should be executed by the team that has an interest in the feature, and any bugs found will not be considered release blockers.
The master version should be ready by the cutoff date
The test manager still instantiates the release-specific version of the spreadsheet
Prototype here: https://docs.google.com/spreadsheets/d/1F-HIdvSh2cICZIMyPXal2eqogMi4m76av8u1xsNpVNk/edit#gid=0
Cut the official testing period in half
Pros: more features land in Redwood
“If the test cycle was smaller, we’d increase the chance of features getting into each release”
We (Axim) know BTR has taken the brunt of it, and are willing to help coordinate efforts to get testing done faster (for example, connecting the right community members together for needed work items, such as more automation, more documentation, etc)
Note: Expected new features/updates and statuses will be known long before the testing cutoff (tracked in the Community Release Plan) and much documentation will be available before the cutoff (see Appendix below).
Codify the process on http://docs.openedx.org - the Docs site is a semi-permanent record of how our software and community works. Making official BTR documentation ensures all stakeholders agree on the processes and that it’s clear this is an official Open edX community function.
What the test manager needs to do: write these docs proactively, including docs such as “How we do Pull Request Workflows” for PRs before the release is finished testing as well as backports after the release is out
Move from spreadsheet to Github issues
One issue for each test case
Tracks the whole lifecycle of the issue
Produce a script to convert the product-owned master spreadsheet to issues
Benefit: Happens at every release
Requirement: Must be smart enough to create new issues if new rows are added
Process suggestions
Critical BTR roles should probably rotate mandatorily every two terms, to avoid individual burnout, allow for process iterations with fresh eyes, and ensure we don’t “default” to the same set of people running the releases
Critical roles should probably have a primary and secondary
Make the test plan/state more visible/discoverable
Collect all testing docs and get as many into the http://docs.openedx.org site as possible; aggregate all pieces of important information - whether in http://docs.openedx.org , wiki, or elsewhere - in a top-level BTR wiki page.
Maintain a place where feature use experts can contribute docs easily - for now may be the wiki until we migrate all testing docs to http://docs.openedx.org
Pinning the top-level wiki page to the Slack channel would be great
Milestones (with dates) should be set
Product team will write all the tests & slot them into 4 priority buckets
Highest priority items must be completed by Week 1. Second highest in Week 2, and so on.
Maintainers are responsible for fixing bugs or getting the authors of the feature question to do so
For Redwood specifically, we recognize that the maintainers program is still gathering steam. Certainly we'll be in a situation where we'll need some heroes (unfortunately).
BTR should help maintainers that are new if they're having trouble, but in general having a concrete thing to fix or understand in a codebase is a great way to get a better understanding of it. BTR should be prepared to give the maintainers extra support for Redwood (if a maintainer exists at all). If the maintainer doesn't exist, we'll be in the situation we've been in historically.
The community needs to rally such that by Sumac, all repos have named maintainers who are ready to help with the release process.
...