Redwood Release Testing Strategy
Background: The Axim Engineering Team got together and brainstormed ways we could improve the release testing strategy around named releases. Here are our recommendations; we hope to spark discussion within BTR about the ways in which the testing process can evolve to be faster, catch more bugs, and be more comprehensive. Also, how BTR can work with the Product team to ensure the right things are being tested with the right level of scrutiny?
Overview of the current community testing strategy
Links to existing documentation:
Announcement: https://discuss.openedx.org/t/join-the-test-team-for-quince-and-meet-the-test-champions-from-palm/11503
Test Sheet: https://docs.google.com/spreadsheets/d/1oVhrnRjk7zxUHMNyU8bJ-TLKyaSukLmN3jWom17O7pI/edit#gid=0
Project Board: https://github.com/orgs/openedx/projects/28/views/5
Recommendations for change with Redwood release
The Product Core Working Group should take ownership of a “master” test spreadsheet. By “ownership”, we mean that the Product Core Working Group should be the group ultimately responsible for making sure that the ways in which we test the release align with the Product Core Working Group’s definition of “core product”. That means defining the tests and their relative priorities. BTR is still responsible for building, testing, and releasing the release.
The Product Team proposes to take on the task of writing coherent and sane tests that align with the highest priority features
This should help alleviate the testing burden that has previously been solely on BTR’s plate
The Product Team will prioritize defining test plans through the most important paths through the core product features.
Teams outside of BTR are welcome to make their own test plans for features outside of the core product that they care about; however, those tests should be executed by the team that has an interest in the feature, and any bugs found will not be considered release blockers.
The master version should be ready by the cutoff date
The test manager still instantiates the release-specific version of the spreadsheet
Prototype here:
Cut the official testing period in half
Pros: more features land in Redwood
“If the test cycle was smaller, we’d increase the chance of features getting into each release”
We (Axim) know BTR has taken the brunt of it, and are willing to help coordinate efforts to get testing done faster (for example, connecting the right community members together for needed work items, such as more automation, more documentation, etc)
Note: Expected new features/updates and statuses will be known long before the testing cutoff (tracked in the Community Release Plan) and much documentation will be available before the cutoff (see Appendix below).
Codify the process on - the Docs site is a semi-permanent record of how our software and community works. Making official BTR documentation ensures all stakeholders agree on the processes and that it’s clear this is an official Open edX community function.
What the test manager needs to do: write these docs proactively, including docs such as “How we do Pull Request Workflows” for PRs before the release is finished testing as well as backports after the release is out
Move from spreadsheet to Github issues
One issue for each test case
Tracks the whole lifecycle of the issue
Produce a script to convert the product-owned master spreadsheet to issues
Benefit: Happens at every release
Requirement: Must be smart enough to create new issues if new rows are added
Process suggestions
Critical BTR roles should probably rotate mandatorily every two terms, to avoid individual burnout, allow for process iterations with fresh eyes, and ensure we don’t “default” to the same set of people running the releases
Critical roles should probably have a primary and secondary
Make the test plan/state more visible/discoverable
Collect all testing docs and get as many into the site as possible; aggregate all pieces of important information - whether in , wiki, or elsewhere - in a top-level BTR wiki page.
Maintain a place where feature use experts can contribute docs easily - for now may be the wiki until we migrate all testing docs to
Pinning the top-level wiki page to the Slack channel would be great
Milestones (with dates) should be set
Product team will write all the tests & slot them into 4 priority buckets
Highest priority items must be completed by Week 1. Second highest in Week 2, and so on.
Maintainers are responsible for fixing bugs or getting the authors of the feature question to do so
For Redwood specifically, we recognize that the maintainers program is still gathering steam. Certainly we'll be in a situation where we'll need some heroes (unfortunately).
BTR should help maintainers that are new if they're having trouble, but in general having a concrete thing to fix or understand in a codebase is a great way to get a better understanding of it. BTR should be prepared to give the maintainers extra support for Redwood (if a maintainer exists at all). If the maintainer doesn't exist, we'll be in the situation we've been in historically.
The community needs to rally such that by Sumac, all repos have named maintainers who are ready to help with the release process.
AppendixThe branding must be site-agnostic and configurable
Definition of “Done” for new features
We are implementing a definition of “Done”, described here. We hope it helps BTR know what to expect with all new features:
Product Documentation, to be completed by the Product Lead for each feature, and the Product Delivery Team:
Definition of done for a feature includes:
Value statement/impact statement - Describing the value of the feature and the impact for users.
How-to documentation - Describing what the feature is and how to use it and configure it
Demo videos - From the product manager demonstrating how to use the feature
Technical documentation - How to set the feature up on the technical end
Case study - A real-life example of how someone is using the feature
Testing instructions (Gherkin/cucumber method) (product)
Example Aspects user story written in Gherkin method
Scenario: A user with access to a course can view total number of enrolled learners and the number and percentage of total enrollees for current enrollees and enrollees per enrollment track
Given I: am a user that has access to course data
When I: Navigate to the instructor tab for one of my courses
Then I: See the total number of learners enrolled in course
And I: See the number and percentage of total enrollees for current enrollees
And I: See the number and percentage of total enrollees per enrollment track (of the enrollment tracks available on my Open edX instance)
Other Requirements:
Features must work in Tutor
Translations requirement must be defined
The branding must be site-agnostic and configurable
Accessibility Review
General testing categories
“Tests” are not all the same. As a reminder, when we say “test” we may mean one or more of the following types of tests are involved.
Feature evaluation/user acceptance testing
Feature validation
Testing at scale
Edge cases
Performance