The 10th Open edX release will be named Juniper.
Put stuff here that we have to remember when we start packaging up for . Especially important is information that system installers or operators will need to know. Please include your name when you add an item, so that we can get back to you with questions.
Juniper is gated by the upgrade to Django 2.2
As of April 27, it looks like all services will be upgraded and in production this week. The master branches can be created soon after that point. We are triaging issues (
) and will decide when to make the master branches based on that triaging.
Upgrading from Ironwood
A high-level upgrade guide is being collaboratively created here.
Installation and Operations
- HTTP Cookie Sharing
- Runbook Configuration
- We are looking at doing large migrations for the
courseware_studentmodule table to prevent running out of primary keys. These will likely be large, semi-manually done migrations on CSM and it's history table(s). Nothing is settled yet, but if we get this in before Juniper we will need to share our runbooks for anyone upgrading to it. (Brian Mesick)
- Part of the django upgrade includes a migration on the auth-user table to increase the size of the last_name column from 30 to 150 characters. If your user table is above a million rows, the to make this update might be long enough to cause downtime. edX.org has opted to skip this migration and has limited the length of the last name field to the previous 30 characters in the user change forms. It's recommend that openedx do the same.
- Django Configuration and SetUp
- Remote Configuration (Cory Lee, Bill DeRusha (Deactivated))
- - Need to provide description of the impact of this change.
- lms/cms removal of aws.py
plugins should use production.py at this point.
JWT Cookies require JWK setup. This was disabled in Ironwood. Thanks to Pierre Mailhot for the following steps to generate and configure the JWT signing keys. (Robert Raposa, Nimisha Asthagiri)
Alternatively, the following piece of code (created for Tutor) can be generated to manually generate the private/public signing keys.
- Third Party Auth Organizations linking is no longer managed via ApplicationOrganization model. This model will be removed and the linking will be done by explicitly defining filters for the application. See relevant decisions:
- We upgraded
sorl-thumbnail from a version that didn't have migrations to one that does, so the initial migration needs to be faked in any environment that previously installed the old version:
./manage.py lms migrate thumbnail --fake .
- Management Commands
- The django-celery library has been removed from edx-celeryutils. The corresponding management command has now been deleted and should be removed from upgrade instructions. (Tim McCormack, Diana Huang)
- Inter-service Communication Impact
- OAuth2 DOP→DOT Migration (Nimisha Asthagiri, Julia Eskew, Troy Sankey, Robert Raposa, Diana Huang, Jeremy Bowman)
- Any OAuth Applications/Clients registered in the LMS that are using the old Django-OAuth-Provider will need to be migrated to Django-OAuth-Toolkit. These applications as well as your Open edX services (e-commerce, Insights, Credentials, etc.)
- For updating your Open edx services, see Setup OAuth Client for Internal Services (Django Oauth Toolkit version)
- Note: there exist management commands used in devstack, but edx.org did this manually.
- For updating your mobile apps:
- Create a new OAuth application at https://<lms>/admin/oauth2_provider/application/ for each of your mobile app types (iOS and Android).
- Client_id: <autogenerated>
User: select the "service user" you've created specifically for this mobile app type
Redirect Uris: <blank>
Client type: Public (differs from services above)
Authorization grant type: Resource owner password-based (differs from services above)
Client secret: <autogenerated>
Name: a unique name you choose to identify this mobile app type
- Update your mobile app configuration:
- OAUTH_CLIENT_ID should be updated to the Client_id from the step above.
- The oauth2/access_token URL defined in the code should already work as-is.
- A new OAuth2 scope, 'user_id', is required for SSO access between the LMS and satellite Open edX services (e-commerce, Insights, Credentials, etc.) For existing Open edX instances, please add the 'user_id' scope for each of your satellite services at /admin/oauth_dispatch/applicationaccess/. You may use a management command to do so (see https://github.com/edx/edx-platform/pull/20076). For newly provisioned instances, this scope will be added automatically with https://github.com/edx/devstack/pull/394. (Robert Raposa)
- Ecommerce expects to find a unique user id (LMS user id) for each user, either in a JWT or social auth. If it cannot find this id, a MissingLmsUserIdException will be raised; this exception can be silenced by enabling the allow_missing_lms_user_id waffle switch. The LMS user id can be back-filled for existing users by running the ecommerce/core/management/commands/import_user_ids.py management command. ecommerce/docs/decisions/0004-unique-identifier-for-users.rst. (crice)
- In edx-platform/lms/djangoapps/courseware/courses.py - The function
check_course_access has been renamed to
check_course_access_with_redirect. A new implementation of
check_course_access returns AccessResponse objects, rather than performing redirects when access is denied.
check_course_access_with_redirect is a drop-in replacement for the old behavior of
check_course_access. This is likely a breaking change for any code outside of edx-platform (such as plugin apps) that was using the
check_course_access function directly. See https://github.com/edx/edx-platform/pull/23651 for more information.
- The MONGOID_AUTH_MECH environment variable needs to be defined, even if just empty. Otherwise the following error will be triggered: "in `encoded_name': undefined method `encode' for nil:NilClass(NoMethodError)".
- Database migration:
- Due to a migration in "SiteConfiguration", it is not possible to have an Ironwood instance serving traffic while a Juniper one is being set up with zero downtime. It will result with
OperationalError exceptions being raised until the Ironwood instance is shutdown. (Slack thread)
Impact on Your Code
- The courseware Django app is now only imported as "lms.djangoapps.courseware". If you import it as "courseware", you may get errors. Fix the imports.
- Comprehensive theming is now disabled by default. To enable it, make sure to add
ENABLE_COMPREHENSIVE_THEMING = True in the edx-platform settings of both the LMS and the CMS.
- Paragon enhancements and edx-bootstrap (Adam Butterworth)
- The Paragon reusable component and pattern library has seen significant investment and refinement since the last Open edX release. A number of components have been re-engineered for ease of use and flexibility. We’ve also launched an interactive documentation site (https://edx.github.io/paragon) where you can explore the components, their implementation status, as well as the library’s foundational CSS utility classes.
- frontend-platform (David Joy)
The Frontend Platform is a framework for Open edX micro-frontend applications. It provides a number of common services used by frontend apps, including analytics (Segment), logging (New Relic), authentication and HTTP/AJAX request client (Axios/JWT), and internationalization (React Intl). It also includes a number of helpful React components and utilities. The framework is highly configurable and extensible, allowing override implementations of analytics and logging, as well as a number of customization hooks for the application. It should be used as a basis for new micro-frontends. A detailed documentation site can be found here: https://edx.github.io/frontend-platform
- Frontend Repos
- We have created a new Microfrontend named Publisher (https://github.com/edx/frontend-app-publisher/) for creating courses and course runs. Although primarily used to keep track of course metadata (such as marketing information), Publisher can also help manage ecommerce products. Publisher integrates between edx-platform, ecommerce, and course-discovery. (Dillon Dumesnil)
- Note: At this point in time, there is no standard process for installing Microfrontends, but Publisher is still being provided so the community can become more familiar with it and possibly configure and install it on their own.
- Python-3: OEP-7, Python 3 How-To Note: Ubuntu 16.04 (and thus the ansible playbooks) ships with Python 3.5.1, but Python 3.5.9 was found to work in Tutor, and it works "better").
- Django-2: Django 2.2 Upgrade Plan
- Ruby was upgraded to v2.5.7 in cs_comments_service (forum). Bundler v1.11.2 and Rake v10.4.2 versions remain identical in the configuration repo, but I (regisb) am completely unable to get them to work in Tutor. I had to upgrade to Bundler v1.17.3 and Rake v13.0.1.
- MongoDb: upgraded from 3.2.16 to 3.6.17
- Pip: upgraded from 19.3.1 to 20.0.2
- Setuptools: upgraded from 39.0.1 to 44.1.0
- DEPRs (see below)
Deprecations and Removals
Based on Juniper Dates: 2019-01-17 to 2020-05-27
- Removal of microsites includes the removal of several tables that were in this djangoapp: https://github.com/edx/edx-platform/pull/21355/
- We have removed the entire djangoapp, but if installations would like to clean up these tables after upgading, they can drop all the tables prefixed by
- Removed now unused /login_post endpoint, which was just used by Studio login. Note: /login_ajax points to the same login code.
- Several breaking changes were made to clean-up
login_user, used by our login POST endpoints.
- JSON is now always returned, including for the third-party auth errors.
- 400 status is now used for errors instead of 200. JSON included
'success': 'failure' both before and after the change.
: https://github.com/edx/edx-platform/pull/22006 (Bill DeRusha (Deactivated))
edx-certificates is no longer installed. This was an error. edx-certificates is still part of Juniper.
- Will Public Course Content be on by default?
- Learner Portal. learner-portal (https://github.com/edx/frontend-app-learner-portal) and it's accompanying configuration service (https://github.com/edx/portal-designer) were created
- SAML. Irfan Ahmad: As part of https://github.com/edx/edx-platform/pull/20665 we have updated the username match criterion for SAML SSO users to be based on email only instead of both email & username.
- As part of https://github.com/edx/ecommerce/pull/2621, the payment page microfrontend is the default payment page for edx ecommerce. To use your own microfrontend, you need to add enable_microfrontend_for_basket_page to your site configuration as well as payment_microfrontend_url . If you do not use the microfrontend, you can use the waffle flag disable_microfrontend_for_basket_page to see the old basket page for testing, etc. (Brandon Baker (Deactivated))
- Performance improvements, etc. (Emma Green)
- TBD - Need to provide a description of feature changes in ecommerce since April 2019 as it impacts the community. (Perhaps links to ADRs.)
- Learner Dashboard. Adeel Ehsan: As part of https://github.com/edx/edx-platform/pull/20439 by default LMS dashboard will show only limited courses based on the course limit set in configuration. Show all the courses link will be at bottom if enrolled courses count is greater than course limit.
- Instructor Dashboard
- TBD - Need to provide a description of changes in gradebook since April 2019 as it impacts the community.
- TBD - Need to provide a description of changes in proctoring since April 2019 as it impacts the community.
- TBD - Need to provide a description of changes in ORA-2 since April 2019 as it impacts the community.
- TBD - Need to provide a description of changes to course dates since April 2019 as it impacts the community.
- SGA xBlock
- Python3 and Django 2.2 compatibility added in version 0.11.0. This version is not backward compatible with Python2 and Ironwood.
- Learning Sequence Authoring Speed (David St. Germain (Deactivated))
- A new OLX REST API can be used to get the OLX "source code" of any XBlock in a course without having to first export the whole course as a tarball.
- New course design interface with horizontal tabs
- Course Discovery
- Course Discovery + Studio
- Changes have been made to how organizations are managed in the edX infrastructure. When Organizations are saved in Course Discovery, we sync the information to edx-platform so Course Discovery is able to be a single source of edit. See https://openedx.atlassian.net/browse/DISCO-1414 and the associated Pull Requests linked in the ticket for more details. (Dillon Dumesnil)
- We have added a new flag ENABLE_PUBLISHER to edx-platform and course-discovery that should both be set if you are using the Publisher Microfrontend. Setting this flag will hide certain fields from Studio and make others read-only. This is to enable Publisher to be the only source of edit for these fields. Setting the flag in course-discovery prevents Refresh Course Metadata from pulling in some fields that become no longer editable in Studio. See details in https://github.com/edx/edx-platform/pull/21516 (Dillon Dumesnil)
- Note: if setting the ENABLE_PUBLISHER flag, you must also define a Publisher URL for your Partner object in the Course Discovery service.