This is a collection of issues we encountered with the Django 2.2 upgrade and lessons learned from it, especially ones that could have been addressed earlier or should inform how we do major upgrades in the future.
Dealing with Deferred Maintenance
Although we couldn’t test with Django 2.0+ until the Python 3 upgrade was complete, much of the work was updating our code to remove usage of APIs that were already deprecated by Django 1.11 and could have been done over the past 2 years. Tickets had been made for much of that work as soon as the 1.11 upgrade was finished, but were deferred as GDPR and other crises took precedence.
Few repositories had been updated to use the newer style of Django middleware introduced in 1.10 (August 2016) which became mandatory in 2.0. We use lots of middleware, both from 3rd-party packages and written ourselves.
Our original OAuth2 library in edx-platform was obsolete and superseded even before the Django 1.11 upgrade, but in a rush to complete that upgrade we chose to maintain a fork of the library alongside its newer replacement rather than completely remove it. Two years later, we had to finish that removal after circumstances had made it even more difficult; this took a few senior engineers over two months to complete.
2 years ago we created guidelines (OEP-18) for more easily keeping Python package dependencies up to date, but dozens of our repositories had not yet been updated to follow them. We made these updates as part of the Django 2.2 upgrade to facilitate safely upgrading many dependencies in each of dozens of repositories, which should help keep things current moving forward.
We had been routinely pinning dependencies to old versions whenever upgrading to a newer one didn’t work automatically (or often just to avoid the risk of an upgrade not working out). We had to deal with dozens of such minor problems all at once when we suddenly needed to upgrade all those libraries to newer versions with support for Django 2.2. The OEP-18 guidelines from 2 years ago warned of this, but again, many repositories weren’t following those guidelines yet.
Missing Automation or Tooling
We had to make essentially the same changes to test matrix files for tox and Travis in dozens of repositories; we have since automated much of that for future Python, Django, and Django REST Framework upgrades.
Django 2.0 removed support for bytestrings from some APIs that had previously accepted them mainly in order to support Python 2. Our tests missed many of these, because they aren’t always consistent with the actual code in the type of data provided as arguments. Type annotations may have caught this much earlier, even if we just checked against the ones already provided for Django itself.
Maintenance Failures (External and Internal)
A few critical dependencies (such as django-oscar in ecommerce and wagtail in portal-designer) had major backwards-incompatibility breaks after dropping support for Django 1.11 and before adding support for Django 2.2 (which the Django project strongly discourages). Upgrading these was unusually painful, but switching to alternatives would have been even moreso.
A few dependencies (such as django-babel, djangorestframwork-jwt, and jsonfield) went through forks after the original project went dormant, and the new maintainers often made major changes to suit their needs. These again were somewhat painful upgrades, but still preferable to maintaining our own fork or switching to a less widely used alternative.
A few of the services that needed to be upgraded either had no active developers or newly assigned owners who weren’t very familiar with the code yet. These were particularly challenging to get deployed without breaking them, as in some cases they had not been deployed in months.
One-Time Problems
Django added a new default view permission for models, which conflicted with custom ones we’d created in some services with the same name. The Django code actually handled such cases pretty well, but our own database migrations didn’t.
Many of our older repositories still used the “nose” testing framework, which (among other deficiencies) has little or no support for collecting deprecation warnings that signal where changes are needed to stop using deprecated code which will be removed in an upcoming release. Upgrading to the much more versatile “pytest” framework is pretty straightforward, but doing it in dozens of repositories took some time. This test runner upgrade should pay dividends over time.