Node 18 - edx-platform Retro

Theard that most of the timeline came from: https://openedx.slack.com/archives/C02QN50TYMD/p1711998235104289

Timeline

• Apr 9th 11am - First merged Node 18 upgrade PR

  • Relevant PR: https://github.com/openedx/edx-platform/pull/34311

• Apr 9th 1pm - 2U Deployment fails during production paver update_assets step.

• Apr 9th 3pm - We discovery that the edx-platform production webpack config is not being tested due to complexities in how it’s run with paver

• Apr 10th 7am - edx.org starts seeing new errors in courseware with some content refusing to load.

• Apr 10th 10am - Rollback Node 18 Upgrade from edx-platform github repo.

• Apr 19th 2pm - Brian has figured out and has a PR to fix most of the courseware loading issues. Including making some test suites run that weren’t running before.

  • Relevant PR: https://github.com/openedx/edx-platform/pull/34503

• Apr 23rd 10am - Merge new PR to edx-platform

• Apr 23rd 12pm - Issues related to proctoring JS found on edx.org staging environment.

• Apr 23rd 1pm - Brian S. tries to setup a proctoring environment locally to re-produce the issue but was unable to setup a working proctoring environment.

• Apr 23rd 2pm - After some discussion, Diana H. puts up a PR that we can merge so that we can test on the edx.org staging environment since there was no way to test locally.

• Apr 23rd 4pm - Manual testing on edx.org staging environment confirms that the proctoring issues have been fixed.

Retrospective

• Going from Webpack 2 → Webpack 5 was the most complicated part of the “Node 18” upgrade.

• Many dependencies did not get upgraded for 7 years exactly.

  • Would automation help? We were looking at major version upgrades which renovate wouldn’t help us with.

• We don’t test the prod webpack build in CI.

• We didn’t have a good way to test rendering of XBlocks

  • maybe needs to be part of the edx.org e2e tests

  • May need some CI e2e tests?

  • Run JS Xblock tests with the prod webpack config.

  • Some tests wouldn’t trigger CI failure when build failed, can we test for this like we do on the python side? https://github.com/openedx/edx-platform/issues/34550

• We don’t have any way of testing the proctoring via CI or Tutor.

  • Proctoring is tied to a specific vendor proctortrack

  • If you make JS changes and want to test them, you have to run them with proctortrack.

  • You can sort of use proctortrack but that process is not documented.

  • Proctoring Process

    • There is a JS code in a webworker that loads proctoring code from the vendor.

      • This is the only feature that has specific implementation within webpack.

    • We want to get rid of this, it is an area of tech debt.

      • The project to move proctoring to LTI has been de-prioritized but is about 80% done.

        • https://github.com/openedx/edx-proctoring/blob/master/docs/decisions/0004-exam-ida.rst

        • https://github.com/edx/edx-exams/blob/main/docs/decisions/0001-purpose-of-this-repo.rst

    • Can we do the proctoring testing with mockprock?

      • Need to make updates to mockprock to test with webworkers.

• Knowledge about the state of the node18 upgrade which had been attempted before.

Learnings

• If you start and don’t finish an upgrade, please, please, please, document how far you got and what prevented you from making further progress to help the next person who may take up the task.

  • You didn’t fail. You learned. Please share what you learned!

• We should use this process for landing big changes made by the community.

  • Flag to arch-bom in #cc-edx-platform

  • Coordinate a time to pause the pipeline and have edx eng do testing on stage.

  • What constitutes a big change?

    • Major Node or Python Version Update

    • React Major Version Update

    • Webpack Major Version Update

    • Django major version update

    • Major version update of djangorestframework

    • react-router major version update

Action Items

• Brian Smith Look into whether we can have renovate at-least make PRs for major edx-platform node package updates.
• Feanil Patel Create an issue to see if we can we setup an e2e test suite in CI for edx-platform that does the bare minimum of testing login, enroll, and submit an answer to a problem.
• Feanil Patel Document an issue for the proctoring tech debt in the edx-platform repo.
• https://github.com/openedx/edx-proctoring/blob/master/docs/decisions/0004-exam-ida.rst
• https://github.com/edx/edx-exams/blob/main/docs/decisions/0001-purpose-of-this-repo.rst
• Zach Hancock Update mockprock docs/config to help improve the manual proctoring testing story.
• Feanil Patel create an issue to build the prod webpack assets in CI
• Diana Huang socialize the process for landing big changes.