edx-platform uses Django. Django is in active development and keeps advancing with continual releases. edx-platform must incorporate these new releases:

https://www.djangoproject.com/download/

Django 1.4 → 1.8

This upgrade was a big project, taking ~5 engineers 4+ months.

For details, view the subpages under this top-level page:

/wiki/spaces/TNL/pages/34013439

Here's the notes from the actual release:

/wiki/spaces/RELEASES/pages/46794141

Here's the retro notes:

Django 1.8 Retro

Django 1.9 Upgrade Hackathon Project

As of June 2016, Django 1.9 is out and on version 1.9.7. It is not a long-term-support (LTS) release and we haven't made the decision to upgrade edx-platform's Django to it yet. However, it must happen eventually.

Release Changes

Official:

https://docs.djangoproject.com/en/1.9/releases/1.9/

Broken down onto a Confluence page with created JIRA tickets:

Django 1.9 Release Notes and Stories

Deprecation

In the current Django 1.8 that edx-platform uses, we've eliminated nearly all the deprecation messages that warn us about functionality that will not be supported in Django 1.9. However, one particularly annoying message remained:

In case that's not readable, it's this warning message:

WARNINGS:
wiki.ArticleRevision.ip_address: (fields.W900) IPAddressField has been deprecated. Support for it (except in historical migrations) will be removed in Django 1.9.
    HINT: Use GenericIPAddressField instead.

I first made a branch to get rid of this warning message - this branch:

https://github.com/edx/django-wiki/tree/jeskew/fix_model_deprecation_warning

Unfortunately, it can't be merged and used by edx-platform until "Comprehensive Theming" lands. But it's ready after that happens!

Exploration

The first thing I ran into was this Django 1.9 change:

• All models need to be defined inside an installed application or declare an explicit app_label. Furthermore, it isn’t possible to import them before their application is loaded. In particular, it isn’t possible to import models inside the root package of an application.

This subtle point means that upon Django startup - when all INSTALLED_APPS are being imported - no models can be imported from the __init__.py of those applications. Unfortunately, edx-platform and its module dependencies do this all over the place.

The errors caused by this premature model importing will look similiar to this one:

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/home/jenkins/edx-venv/local/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/jenkins/edx-venv/local/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/jenkins/edx-venv/local/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/jenkins/workspace/edx-platform-test-subset/common/lib/xmodule/xmodule/video_module/__init__.py", line 8, in <module>
    from .transcripts_utils import *
  File "/home/jenkins/workspace/edx-platform-test-subset/common/lib/xmodule/xmodule/video_module/transcripts_utils.py", line 18, in <module>
    from .bumper_utils import get_bumper_settings
  File "/home/jenkins/workspace/edx-platform-test-subset/common/lib/xmodule/xmodule/video_module/bumper_utils.py", line 16, in <module>
    import edxval.api as edxval_api
  File "/home/jenkins/edx-venv/local/lib/python2.7/site-packages/edxval/api.py", line 13, in <module>
    from edxval.models import Video, EncodedVideo, CourseVideo, Profile
  File "/home/jenkins/edx-venv/local/lib/python2.7/site-packages/edxval/models.py", line 48, in <module>
    class Profile(models.Model):
  File "/home/jenkins/edx-venv/local/lib/python2.7/site-packages/django/db/models/base.py", line 94, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/home/jenkins/edx-venv/local/lib/python2.7/site-packages/django/apps/registry.py", line 239, in get_containing_app_config
    self.check_apps_ready()
  File "/home/jenkins/edx-venv/local/lib/python2.7/site-packages/django/apps/registry.py", line 124, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
AppRegistryNotReady: Apps aren't loaded yet.

After modifying 5+ external Python modules, changing their requirements entries, and changing around the startup.py files, I was able to run the following command to completion:

python manage.py cms --settings=devstack reindex_course --setup

However, when I started up LMS or Studio, I got errors indicating that I had unapplied migrations.

When I attempted a migration, I got errors about importing South in djcelery! What??! Turns out the version that edx-platform uses has some remaining migrations that are pre-Django-1.7? I upgraded djcelery to the most recent version and tried migration again. Got a message about a table already existing. When I compared the existing schema and the new single 0001_initial.py migration in djcelery, the schema did not match. So some tricky database figuring-out will be required to proceed. The next step:

• Look at https://github.com/celery/django-celery/commits/master/djcelery/migrations
• Figure out if the new migration 0001_initial.py fully represents the state of the DB in the four older migration files 0001-0004.

The branch I created for this work is here:

https://github.com/edx/edx-platform/tree/jeskew/django_1_9_compat

And the PR I used to run tests is here:

https://github.com/edx/edx-platform/pull/12770

I also created branches/forks in other repos. To find all those changes, review the base.txt/github.txt files from the requirements files on the branch above.