Python 3.11/3.12 Major Changes

Owned by Muhammad Soban Javed
Last updated: Mar 12, 2024 by Usama Sadiq
2 min read

This list is currently in draft, if you find any change causing issues and that isn’t listed here so kindly add to the list.

 

Example Pull Request: https://github.com/openedx/edx-django-utils/pull/390

Potential Steps to perform on a repo:

  • Run both python312_tox_modernizer and python312_gh_actions_modernizer scripts from the edx-repo-tools package if not done already.

    • You can also manually remove toxenv related to django32 and add django42

      • Add python312 in the python-version matrix value in CI

  • Run make upgrade with Python 3.12 virtualenv to update dependencies for Python 3.12

  • If needed, include setuptools and wheelin tox.ini config sections.

List all those major changes that can impact us while upgrading our services and packages.

  • [Python 3.12] setuptools isn't now pre-installed in Python 3.12 virtual environments, instead it should now be installed through pip explicitly.

    • Include setuptools in tox config sections wherever needed

    • You may need to add wheel in tox config for docs checks to work as well if python setup.py sdist bdist_wheel command fails with Python 3.12.

  • [Python 3.12] pact-python

    • Python 3.12 update needs pact-python>2.1.1 which will have the support of Python 3.12. The package owner has planned to release new version within the current week which will unblock this PR. For details, see the package author's comment pact-foundation/pact-python#422 (comment).

  • [Python 3.12] pyresttest

    • pyresttest doesn’t have Python 3.12 support and it has not been updated in last 8 years.

    • Need to find any fork or alternative of this package to upgrade api-manager.

  • [Python 3.12 + Django 4.2] backports.zoneinfo

    • We haven’t migrated from pytz to zoneinfo yet so the repos, where Django 4.2 by default import backports.zoneinfo as a dependency fail tests setup with Python 3.12.

    • To fix this issue, we need to add following pin in requirements/base.in and run make upgrade again.

      • backports.zoneinfo;python_version<"3.9"  # Can be removed once we upgrade to use zoneinfo

  • [Python 3.11 & Python 3.12] Celery>=5.0

    • Celery>=5.0 is needed for the support of both Python 3.11 and Python 3.12.

    • Right now, edx-platform has already been updated but some other IDAs needs to be updated first including Registrar and Ecommerce to enable Python 3.12 support.

 

Usama Sadiq
February 28, 2024

Since Python 3.12 dropped bundling setuptools, now our only option is to install it separately.
As you can see in the linked PRs on the Python issue [https://github.com/python/cpython/issues/95299 ], the only solution community has is to install it separately.
As far as I know and could search, in our requirements files pattern, we only have the option to use --allow-unsafe to include setuptools.

Robert Raposa
February 28, 2024

I was just misinterpreting the scope of this change. I thought that was being added to all requirements files. But I see in https://github.com/openedx/edx-django-utils/blob/1916a4729dba8c576ffc75849274558034e1bd10/Makefile#L45 that this is just for pip-related dependencies. Maybe the docs should mention pip.in to make that more clear?

Usama Sadiq
February 28, 2024

Previously, we only had it in the pip.in file.
But now, setuptools needs to be added in base requirements.

Usama Sadiq
February 28, 2024

So you were right in your understanding that we need to add it in all requirements but we don’t have any other solution for now since if we don’t include setuptools, the test runs fail to execute when setting up testenv or installing packages for quality checks.

Tim McCormack
March 5, 2024

It’s unsafe-as-in-breakage, not unsafe-as-in-security, so if it’s working then I’m fine with it.

It sounds like it’s not even all that “unsafe” in terms of breakage, especially in the way we’re using it: https://github.com/jazzband/pip-tools/issues/806#issuecomment-493591664

Usama Sadiq
March 6, 2024

Got an alternative working since we only need it in testing envs, we are only adding it in tox config sections for now in packages instead of opting for this approach.
Feanil has also agreed on this approach.

Robert Raposa
March 6, 2024

Nice. If this doc is meant for others, consider the following:

  1. Removing the crossed out lines and replacing it with updated recommendations.

  2. Including links to example code where it makes sense.

Thanks @Usama Sadiq.