Mini-RCA doc: ubuntu-latest issues on edx-platform

October 9 (overnight):

edx-platform build started failing when ubuntu-latest rolled over from 22.04 to 24.04:
- successful job: Merge pull request #35618 from openedx/katrinan029/upgrade-edx-enterp… · openedx/edx-platform@f2ad108
unsuccessful jobs were running into two issues:
- undefined symbol: xmlSecOpenSSLTransformRsaMd5GetKlass
  - example job: chore: version bump · openedx/edx-platform@977e849
- mongodb-org-mongos : Depends: libssl1.1 (>= 1.1.1) but it is not installable
  - example job: chore: version bump · openedx/edx-platform@977e849

October 10:

2U fixes both issues in the same PR: feat: use jammy repositories to install mongo for testing and unpin lxml and xmlsec. by dianakhuang · Pull Request #35634 · openedx/edx-platform
- use the jammy repositories to install mongo (the noble repositories do not have mongo 7.0, which is the version we use): https://github.com/openedx/edx-platform/pull/35634/commits/126cdb14c57c1ddbc5d97937bfef263ca0d76b5e
- unpin xml-sec and lxml and upgrade all dependencies: https://github.com/openedx/edx-platform/pull/35634/commits/b0f6af92f823ca0de8f8256f29ee005f4da612e9
When attempting to deploy to edx.org, 2U runs into this error: AssertionError: Internal issue: Candidate is not for this requirement lxml[html-clean,html-clean] vs lxml[html-clean]
- this is probably already related to the breaking changes referred to in the lxml upgrade ticket: unpin lxml in edx-platform · Issue #35272 · openedx/edx-platform
2U puts together a PR to revert out the changes and pin to 22.04: Revert out updates and pins many of the jobs to Ubuntu 22.04 by dianakhuang · Pull Request #35635 · openedx/edx-platform
- This PR is blocked because the static assets check was required and still referred to ubuntu-latest

FAILED - RETRYING: install python requirements (1 retries left).
failed: [10.3.11.223] (item={'changed': False, 'stat': {'exists': True, 'path': '/edx/app/edxapp/edx-platform/requirements/edx/base.txt', 'mode': '0664', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1038, 'gid': 1039, 'size': 30361, 'inode': 4914286, 'dev': 66306, 'nlink': 1, 'atime': 1728591811.6928587, 'mtime': 1728591811.6928587, 'ctime': 1728591811.6928587, 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': True, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 64, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': False, 'pw_name': 'edxapp', 'gr_name': 'edxapp', 'checksum': '1b337c7a7d1905394f19b8f4c5b144e45ff38f42', 'mimetype': 'text/plain', 'charset': 'us-ascii', 'version': '409460603', 'attributes': ['extents'], 'attr_flags': 'e'}, 'invocation': {'module_args': {'path': '/edx/app/edxapp/edx-platform/requirements/edx/base.txt', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/edx/app/edxapp/edx-platform/requirements/edx/base.txt', 'ansible_loop_var': 'item'}) => {
    "ansible_loop_var": "item",
    "attempts": 5,
    "changed": true,
    "cmd": [
        "/edx/app/edxapp/venvs/edxapp/bin/pip",
        "install",
        "-i",
        "https://pypi.python.org/simple",
        "--exists-action",
        "w",
        "-r",
        "/edx/app/edxapp/edx-platform/requirements/edx/base.txt"
    ],
    "delta": "0:00:44.422824",
    "end": "2024-10-10 20:29:25.108155",
    "item": {
        "ansible_loop_var": "item",
        "changed": false,
        "failed": false,
        "invocation": {
            "module_args": {
                "checksum_algorithm": "sha1",
                "follow": false,
                "get_attributes": true,
                "get_checksum": true,
                "get_md5": false,
                "get_mime": true,
                "path": "/edx/app/edxapp/edx-platform/requirements/edx/base.txt"
            }
        },
        "item": "/edx/app/edxapp/edx-platform/requirements/edx/base.txt",
        "stat": {
            "atime": 1728591811.6928587,
            "attr_flags": "e",
            "attributes": [
                "extents"
            ],
            "block_size": 4096,
            "blocks": 64,
            "charset": "us-ascii",
            "checksum": "1b337c7a7d1905394f19b8f4c5b144e45ff38f42",
            "ctime": 1728591811.6928587,
            "dev": 66306,
            "device_type": 0,
            "executable": false,
            "exists": true,
            "gid": 1039,
            "gr_name": "edxapp",
            "inode": 4914286,
            "isblk": false,
            "ischr": false,
            "isdir": false,
            "isfifo": false,
            "isgid": false,
            "islnk": false,
            "isreg": true,
            "issock": false,
            "isuid": false,
            "mimetype": "text/plain",
            "mode": "0664",
            "mtime": 1728591811.6928587,
            "nlink": 1,
            "path": "/edx/app/edxapp/edx-platform/requirements/edx/base.txt",
            "pw_name": "edxapp",
            "readable": true,
            "rgrp": true,
            "roth": true,
            "rusr": true,
            "size": 30361,
            "uid": 1038,
            "version": "409460603",
            "wgrp": true,
            "woth": false,
            "writeable": true,
            "wusr": true,
            "xgrp": false,
            "xoth": false,
            "xusr": false
        }
    },
    "rc": 2,
    "start": "2024-10-10 20:28:40.685331"
}

STDOUT:

Looking in indexes: https://pypi.python.org/simple
Obtaining edx_proctoring_proctortrack from git+https://github.com/anupdhabarde/edx-proctoring-proctortrack.git@31c6c9923a51c903ae83760ecbbac191363aa2a2#egg=edx_proctoring_proctortrack (from -r /edx/app/edxapp/edx-platform/requirements/edx/base.txt (line 7))
  Skipping because already up-to-date.
Collecting py2neo@ https://github.com/overhangio/py2neo/releases/download/2021.2.3/py2neo-2021.2.3.tar.gz
  Using cached https://github.com/overhangio/py2neo/releases/download/2021.2.3/py2neo-2021.2.3.tar.gz (158 kB)
Collecting acid-xblock==0.4.1
  Using cached acid_xblock-0.4.1-py3-none-any.whl (24 kB)
Collecting aiohappyeyeballs==2.4.3
  Using cached aiohappyeyeballs-2.4.3-py3-none-any.whl (14 kB)
Collecting aiohttp==3.10.9
  Using cached aiohttp-3.10.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
Collecting aiosignal==1.3.1
  Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting algoliasearch==3.0.0
  Using cached algoliasearch-3.0.0-py2.py3-none-any.whl (33 kB)
Collecting amqp==5.2.0
  Using cached amqp-5.2.0-py3-none-any.whl (50 kB)
Collecting analytics-python==1.4.post1
  Using cached analytics_python-1.4.post1-py2.py3-none-any.whl (23 kB)
Collecting aniso8601==9.0.1
  Using cached aniso8601-9.0.1-py2.py3-none-any.whl (52 kB)
Collecting annotated-types==0.7.0
  Using cached annotated_types-0.7.0-py3-none-any.whl (13 kB)
Collecting appdirs==1.4.4
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting asgiref==3.8.1
  Using cached asgiref-3.8.1-py3-none-any.whl (23 kB)
Collecting asn1crypto==1.5.1
  Using cached asn1crypto-1.5.1-py2.py3-none-any.whl (105 kB)
Collecting attrs==24.2.0
  Using cached attrs-24.2.0-py3-none-any.whl (63 kB)
Collecting babel==2.16.0
  Using cached babel-2.16.0-py3-none-any.whl (9.6 MB)
Collecting backoff==1.10.0
  Using cached backoff-1.10.0-py2.py3-none-any.whl (31 kB)
Collecting bcrypt==4.2.0
  Using cached bcrypt-4.2.0-cp39-abi3-manylinux_2_28_x86_64.whl (273 kB)
Collecting beautifulsoup4==4.12.3
  Using cached beautifulsoup4-4.12.3-py3-none-any.whl (147 kB)
Collecting billiard==4.2.1
  Using cached billiard-4.2.1-py3-none-any.whl (86 kB)
Collecting bleach[css]==6.1.0
  Using cached bleach-6.1.0-py3-none-any.whl (162 kB)
Collecting boto==2.49.0
  Using cached boto-2.49.0-py2.py3-none-any.whl (1.4 MB)
Collecting boto3==1.35.37
  Using cached boto3-1.35.37-py3-none-any.whl (139 kB)
Collecting botocore==1.35.37
  Using cached botocore-1.35.37-py3-none-any.whl (12.6 MB)
Collecting bridgekeeper==0.9
  Using cached bridgekeeper-0.9-py3-none-any.whl (20 kB)
Collecting cachecontrol==0.14.0
  Using cached cachecontrol-0.14.0-py3-none-any.whl (22 kB)
Collecting cachetools==5.5.0
  Using cached cachetools-5.5.0-py3-none-any.whl (9.5 kB)
Collecting camel-converter[pydantic]==4.0.1
  Using cached camel_converter-4.0.1-py3-none-any.whl (6.2 kB)
Collecting celery==5.4.0
  Using cached celery-5.4.0-py3-none-any.whl (425 kB)
Collecting certifi==2024.8.30
  Using cached certifi-2024.8.30-py3-none-any.whl (167 kB)
Collecting cffi==1.17.1
  Using cached cffi-1.17.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (467 kB)
Collecting chardet==5.2.0
  Using cached chardet-5.2.0-py3-none-any.whl (199 kB)
Collecting charset-normalizer==2.0.12
  Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting chem==1.3.0
  Using cached chem-1.3.0-py3-none-any.whl (24 kB)
Collecting click==8.1.6
  Using cached click-8.1.6-py3-none-any.whl (97 kB)
Collecting click-didyoumean==0.3.1
  Using cached click_didyoumean-0.3.1-py3-none-any.whl (3.6 kB)
Collecting click-plugins==1.1.1
  Using cached click_plugins-1.1.1-py2.py3-none-any.whl (7.5 kB)
Collecting click-repl==0.3.0
  Using cached click_repl-0.3.0-py3-none-any.whl (10 kB)
Collecting code-annotations==1.8.0
  Using cached code_annotations-1.8.0-py2.py3-none-any.whl (39 kB)
Collecting codejail-includes==1.0.0
  Using cached codejail_includes-1.0.0-py2.py3-none-any.whl (25 kB)
Collecting crowdsourcehinter-xblock==0.7
  Using cached crowdsourcehinter_xblock-0.7-py3-none-any.whl (34 kB)
Collecting cryptography==43.0.1
  Using cached cryptography-43.0.1-cp39-abi3-manylinux_2_28_x86_64.whl (4.0 MB)
Collecting cssutils==2.11.1
  Using cached cssutils-2.11.1-py3-none-any.whl (385 kB)
Collecting defusedxml==0.7.1
  Using cached defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting django==4.2.16
  Using cached Django-4.2.16-py3-none-any.whl (8.0 MB)
Collecting django-appconf==1.0.6
  Using cached django_appconf-1.0.6-py3-none-any.whl (6.4 kB)
Collecting django-cache-memoize==0.2.0
  Using cached django_cache_memoize-0.2.0-py3-none-any.whl (14 kB)
Collecting django-celery-results==2.5.1
  Using cached django_celery_results-2.5.1-py3-none-any.whl (36 kB)
Collecting django-classy-tags==4.1.0
  Using cached django_classy_tags-4.1.0-py3-none-any.whl (14 kB)
Collecting django-config-models==2.7.0
  Using cached django_config_models-2.7.0-py2.py3-none-any.whl (30 kB)
Collecting django-cors-headers==4.4.0
  Using cached django_cors_headers-4.4.0-py3-none-any.whl (12 kB)
Collecting django-countries==7.6.1
  Using cached django_countries-7.6.1-py3-none-any.whl (864 kB)
Collecting django-crum==0.7.9
  Using cached django_crum-0.7.9-py2.py3-none-any.whl (4.7 kB)
Collecting django-fernet-fields-v2==0.9
  Using cached django_fernet_fields_v2-0.9-py3-none-any.whl (8.7 kB)
Collecting django-filter==24.3
  Using cached django_filter-24.3-py3-none-any.whl (95 kB)
Collecting django-ipware==7.0.1
  Using cached django_ipware-7.0.1-py2.py3-none-any.whl (6.4 kB)
Collecting django-js-asset==2.2.0
  Using cached django_js_asset-2.2.0-py3-none-any.whl (4.7 kB)
Collecting django-method-override==1.0.4
  Using cached django_method_override-1.0.4-py3-none-any.whl (5.2 kB)
Collecting django-model-utils==5.0.0
  Using cached django_model_utils-5.0.0-py3-none-any.whl (42 kB)
Collecting django-mptt==0.16.0
  Using cached django_mptt-0.16.0-py3-none-any.whl (115 kB)
Collecting django-multi-email-field==0.7.0
  Using cached django_multi_email_field-0.7.0-py3-none-any.whl (11 kB)
Collecting django-mysql==4.14.0
  Using cached django_mysql-4.14.0-py3-none-any.whl (55 kB)
Collecting django-oauth-toolkit==1.7.1
  Using cached django_oauth_toolkit-1.7.1-py3-none-any.whl (63 kB)
Collecting django-object-actions==4.3.0
  Using cached django_object_actions-4.3.0-py3-none-any.whl (19 kB)
Collecting django-pipeline==3.1.0
  Using cached django_pipeline-3.1.0-py3-none-any.whl (40 kB)
Collecting django-push-notifications==3.1.0
  Using cached django-push-notifications-3.1.0.tar.gz (54 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting django-ratelimit==4.1.0
  Using cached django_ratelimit-4.1.0-py2.py3-none-any.whl (11 kB)
Collecting django-sekizai==4.1.0
  Using cached django_sekizai-4.1.0-py3-none-any.whl (8.6 kB)
Collecting django-ses==4.2.0