Django 3.2 Upgrade

All of the Django-using Open edX IDAs are now on Django 2.2, which will reach end of life in April 2022. edX wants to upgrade Django to version 3.2, the next LTS (long term support) release. We want this project complete by October 15, 2021, in time for the next Open edX release, Maple. Key contacts are @Jeremy Bowman (Deactivated) and @Natalia Berdnikov (Deactivated).

What is this about?

Most of the services in Open edX use the Django framework to handle all of the low-level “this is what you need to even have a functional, maintainable web application” bits. Each repository in Open edX has a designated owning team ultimately responsible for keeping it working with a still-supported version of Django at all times, but the Arch-BOM and Arbi-BOM teams will be utilizing prior experience with major Django upgrades to do much of the planning, automation, and coordination for the process. We will be upgrading essentially all Django-using code in Open edX to work not just with the prior long-term support (LTS) release (Django 2.2), but also the recently released one (Django 3.2). This is expected to require less effort than previous Django upgrades that made more backwards-incompatible changes, but still involve more code changes than upgrading most other components of our technology stack due to the fact that Django influences how we structure our code in many ways.

Why are we doing this?

Because web application security depends on effectively mitigating all currently known attack vectors, it’s pretty important to stay on a framework version that’s still getting regular updates to fix newly discovered attacks. New Django versions also often address pain points in previous releases, improve performance, and/or add useful new functionality that can accelerate future development. Django is one of the best web application frameworks with respect to facilitating upgrades and managing backwards-incompatible changes; this would be even more challenging in most available alternatives (in any programming language). In fact, many Java web applications end up forking and maintaining a specific version of their framework indefinitely because the effort to do an upgrade exceeds available development resources.

The Django 2.2.x release series will cease getting security fixes in April 2022, before the end of support for the next Open edX release (Maple). The latest, 3.2.x release we’re upgrading to will continue getting security fixes until April 2024.

Have we done this before?

We have already successfully upgraded Open edX from one Django LTS release to the next 3 times; the Django release schedule is such that we have to do this roughly every 2.5 years. Each time, both edX and Django get better at streamlining the process. The Open edX codebase is growing over time, but the effort needed to perform each upgrade is nevertheless reducing slightly over time due to these improvements. We get a lot of assistance from the Arbi-BOM team at Arbisoft and other members of the Open edX community, but previous attempts to enlist further aid from consulting firms haven’t proven successful because the Open edX codebase is too large for them to be able to scope and estimate the work involved in a timely manner.

Information about prior Open edX Django upgrades can be found at Django 2.2 Upgradearchived, Platform Django 1.8 - 1.11 Upgradearchived, and edx-platform Django Upgradesarchived . The main points we want to do differently than the previous 2.2 upgrade are:

• Analyze and track Django compatibility issues with a master list of all Django-using dependencies of Open edX, rather than on a per-IDA basis (since many packages are used in multiple IDAs, and some IDAs start the upgrade in earnest much later than others). The current state of this analysis can be found at Dependency Supported Python/Django Versions.

• Figure out early on which upgrade tasks can be effectively taken on by the broader Open edX community, and add them to a GitHub Project board so they can be efficiently distributed even to community members with little Jira experience.

• Explicitly provide a menu of options for each squad owning an IDA to decide how much of the development work to delegate to the Arbi-BOM team. We expect this to reduce confusion around who is responsible for what, and allow squad managers to better account for the upgrade in their planning process.

What do we plan to do?

We need to perform the steps in Django 3.2 Upgrade: Steps to Upgrade a Repositoryarchived for each of the IDAs in Django 3.2 Upgrade List of IDAsarchived, as well as each of their dependencies which is maintained by edX or another member of the Open edX community. The key stakeholders are identified in Django 3.2 Upgrade: Roles and Communication | Team and Rolesarchived.

Are there risks and dependencies?

Each time we perform one of these upgrades, we inevitably find a few dependencies which don’t add support for the new Django versions in a timely manner (or cease to be maintained altogether). See Handling Outdated Dependencies for our process for handling such cases.

What is the impact of this upgrade on various users of the system?

Learners and educators should see no change in site availability or functionality as a result of the upgrade. Some developers will assist in implementing it, and all should read Django 3.2 Upgrade: Key Changesarchived for a summary of minor changes in how to write new application code as a result of the upgrade. Site operators shouldn’t need to do anything beyond upgrading to the upcoming Maple release of Open edX, which most of them intend to do anyway.

How will we do it?

Each IDA can be upgraded independently whenever testing with Django 3.2 is complete. This can be done via the standard deployment pipeline and should require no downtime. If unexpected problems are hit in stage or production, a standard pipeline rollback can be used. Barring any such unexpected problems, the upgrade should be transparent to site users.

Scope

Upgrade all IDAs from Django 2.2 to 3.2 in production and Edge:

• Review dependencies and ensure all things we use are upgraded to 3.2 or we no longer use them.

• Run scripts to automatically fix known problems (apply code mods)

• Update the test suites for repositories so they can work on all of the following Django versions: 2.2, 3.0, 3.1, and 3.2. Test, and fix identified problems.

• Upgrade the default version of the IDA to be 3.2 and test

• [Can be done after Maple, if necessary] Drop testing Django 2.2, 3.0 and 3.1 in all IDAs

Notes:

• You’re least likely to get blocked by unfinished work if you first upgrade repositories that have the fewest dependencies.

• Roll out upgrades to IDAs one by one, as they are ready.

• A more detailed description of the steps involved can be found in Django 3.2 Upgrade: Steps to Upgrade a Repositoryarchived.