...
Note that we want to create the per-repository issues pretty early in the process, to as they serve a few important roles:
They give teams advance notice that they need to allocate some time for the work. More detailed communications about the expected level of effort and available automation will come later, but even just this heads up that “you need to leave some room in your schedule for this” is useful.
They allow teams to specify their preferred level of involvement in upgrading each repository. Some will prefer to be hands-off and let the orchestration team handle as much as practical, others will have reasons why they want to do most of the upgrade work themselves in a particular repository.
They provide an official forum for discussing repository-specific aspects of the upgrade project.
They allow the https://github.com/orgs/edx/projects/17/views/1 board to serve as a status dashboard for the upgrade project, by filtering to just the issues in that project.
Automate As Much As Practical
...
Now that we’ve done as much as we reasonably can to automate the upgrade, we need to ask the teams owning repositories impacted by the upgrade what they want their role in the upgrade to be. The orchestration team should send an announcement (as described in Upgrade-Related Announcements) asking teams to read the upgrade instructions and select one of the Upgrade Service Levels for each impacted repository. These choices should be specified in the “Owner Does” field of the corresponding issue on the https://github.com/orgs/edx/projects/17/views/1 board (this is easiest to do in the Table view).
Any external dependencies which still need updates to support the upgrade should have issues created in https://github.com/openedx/public-engineering with the “help wanted” label and “maintenance” labels added. These issues should link to the roadmap issue for the upgrade, the Handling Outdated Dependencies page, and the upgrade instructions document, and should be added to the https://github.com/orgs/edx/projects/17/views/1 board. Once these are created, the orchestration team should ask the broader Open edX developer community for assistance with these issues. These can be completed independently of the other upgrade work if done early enough, but may have a long turnaround time depending on the availability of upstream maintainers, so we really want to encourage early action on these. And the calendar time needed to complete these dependency upgrades is shortest if we have a lot of people reaching out in parallel to get this work started, rather than queuing all the upgrade communications on a small team of people.
If the deployment will require nontrivial operations work, there should also be a ticket tickets for the Build-Test-Release Working Group and the SRE team at 2U (and any other organization publicly committed to releasing from the main development branches) to prepare for any necessary database upgrades, etc. This These typically isn’t aren’t needed for Django upgrades, but is are for MongoDB, MySQL, Ubuntu, etc. This These should also go on the https://github.com/orgs/edx/projects/17/views/1 board.
...
If new information is discovered that could help with other parts of the upgrade (solutions for tricky parts of the upgrade, new codemods for handling some of the code updates, unanticipated problem points to watch out for, etc.), the project documentation should be updated and the changes should be
The status of issues on the https://github.com/orgs/edx/projects/17/views/1 board should be kept up to date as work progresses; one person on the orchestration team should be assigned ultimate responsibility for making sure this happens, although they are free to procure assistance from other team members and/or a project manager as appropriate. The project view on that board and the repository health dashboards provide a relatively real-time view of the status of the overall project, minimizing the need for manual dashboard creation in Confluence or elsewhere.
...
Update the support windows spreadsheet to correctly indicate the version now used.
Announce the successful completion of the upgrade!Make sure any deployment instructions make it into the release notes of the next Open edX named release. Communicate with the Build-Test-Release Working Group to make sure this gets done satisfactorily.
Remove any CI matrix entries for no-longer-supported versions of the dependency. For packages, this typically can’t be done until all of the services using them have completed the upgrade. Services are more free to do this relatively soon after the upgrade is successfully deployed.
Remove the https://github.com/orgs/edx/projects/17/views/1 board view created for the upgrade.
Stop running repo health checks which are no longer relevant. We may just want to skip the checks rather than deleting them, to make it easier to reuse them for the next similar upgrade.
Mark the roadmap issue for the upgrade as complete.
Do a retrospective meeting + asynchronous conversation to collect feedback on what went well, what could have been better, and what we can do even better next time. Update this document and the pages it links to if appropriate, and write and assign issues for any necessary followup work.
Move the root Confluence page for this upgrade’s docs under Past Upgrades.
If any external dependencies were discovered to be unsatisfactorily maintained, schedule work to replace them or (if necessary) take over maintenance of them ourselves. See Handling Outdated Dependencies for more details on this.