Release Testing for the Upcoming Release

Introduction

Every 6 months, the Open edX project creates a new release (named after trees in alphabetical order - Quince, Redwood, Sumac, Teak, etc). We need the whole community to work together not just to build new features but also to test them - and other already existing ones! - before we can declare the next release ready to go.

Product Testing is our term for the tests that we do that copy a workflow an Open edX user (such as an educator, course team member, or learner) would do as part of their experience on an Open edX site. Anyone and everyone is qualified to be a product tester, no matter your role in the community or on your Open edX team.

Important Dates

See the https://openedx.atlassian.net/wiki/spaces/OEPM/pages/4191191044 for dates on when the upcoming release will be cut and tested.

Instructions for Product Testing

Testing is fairly simple:

1. Visit the Release Spreadsheet (for Teak: https://docs.google.com/spreadsheets/d/1kDn7P4qymNGZeCjTV9pWY9026RAarGsEyMBgWghRDok/edit?gid=0#gid=0 ), and navigate to the Tests page. To claim a test, add your name to the “Assigned To” column.

   1. Note: Please only claim tests you can test within 24 hours.

   2. In some cases, more than one tester will be testing the same test. In that case, put everyone’s names in that cell.

2. The spreadsheet will have links to the testing “sandbox”. Mark your test as “In Progress”, visit the sandbox, and perform all the steps of the tests (they will be specified in the spreadsheet).

   1. For Redwood: Teak LMS, and Teak Studio

3. Update the spreadsheet with results: Put “Completed” as the test’s status, and “Passed” or “Failed” as the result. For failure, please file a report - see the next section.

Need help? Ask us in #wg-build-test-release in Slack!

Reporting Failed Tests

To report a failed test you will create a GitHub ticket. This is similar to a ticket in other tracking systems, such as Jira or Trello.

1. Determine the appropriate GitHub code repository (“repo”) to log the failed test in - this will be the repo that holds the code that failed.

   1. Don’t know the right repo? No problem! Go here: https://github.com/openedx/wg-build-test-release/issues

2. Open up a new ticket with a title that looks like this:
   [Test-ID] Description of issue
   such as: [TC_LEARNER_9] Sign in with existing user account keeps failing

3. Add as much detail as possible, including:

   1. Steps to reproduce

   2. Expected result

   3. Actual result

   4. Screenshots and/or screen recordings (Instructions on a Mac)

4. Submit the ticket

5. Add the label “release testing” to your ticket by writing - in one comment, with nothing else - the following:
   label: release testing

After you file the ticket…

Please make sure you have email notifications enabled on GitHub.

You may receive questions on your ticket from the team responsible for fixing the bug. Please reply promptly so they may finish their work.

When they’ve marked the bug as fixed, please run the test again (or indicate, on the ticket or in Slack, that you are unavailable to do so). Update the Tests spreadsheet with the result of the re-test (if Passed, you’re good to go! If it’s still failing, update your GitHub ticket - no need to file a new one unless you’re asked to do so).

Spreadsheet for the Next Release

Is here! https://docs.google.com/spreadsheets/d/1kDn7P4qymNGZeCjTV9pWY9026RAarGsEyMBgWghRDok/edit?gid=0#gid=0

I want to fix bugs!

Awesome - the project is always in need of more bug squashers! We recommend that you join the Build Test Release working group, by attending a meeting, posting in the forums, and/or joining the Slack channel, to find out where they could best use you. To get started, check out the Communication section of the BTR homepage: https://openedx.atlassian.net/wiki/spaces/COMM/pages/1022099494/Build-Test-Release+BTR+Working+Group#Communication

FAQs

Have any questions about this process? Add a comment to this page or reach out in the #wg-build-test-release Slack chat room and we’ll get back to you.