The 10th Open edX release will be named Juniper.

Put stuff here that we have to remember when we start packaging up for Juniper.  Especially important is information that system installers or operators will need to know. Please include your name when you add an item, so that we can get back to you with questions.

Timeline

Juniper is gated by the upgrade to Django 2.2

As of April 27, it looks like all services will be upgraded and in production this week.  The master branches can be created soon after that point.  We are triaging issues ( CRI-171 - Getting issue details... STATUS ) and will decide when to make the master branches based on that triaging.

Installation and Operations

• HTTP Cookie Sharing
  • Studio redirects to the LMS for login and registration. (Robert Raposa, Nimisha Asthagiri (Deactivated)) 
    • The DISABLE_STUDIO_SSO_OVER_LMS setting used in Ironwood has been removed. See https://github.com/edx/edx-platform/pull/22416 for background on why this change was made.
    • Note: for community members who will have difficulty sharing cookies across Studio and LMS, see https://github.com/edx/edx-platform/pull/19845#issuecomment-559154256 for possible alternative solutions from Felipe Montoya. These solutions can be discussed here: https://discuss.openedx.org/t/deprecation-removal-complete-removal-of-studio-signin-and-signup-pages-relates-to-depr-6/962
      
      
• Runbook Configuration
  • We are looking at doing large migrations for the courseware_studentmodule table to prevent running out of primary keys. These will likely be large, semi-manually done migrations on CSM and it's history table(s). Nothing is settled yet, but if we get this in before Juniper we will need to share our runbooks for anyone upgrading to it. (Brian Mesick (Deactivated))
    
    
  • Part of the django upgrade includes a migration on the auth-user table to increase the size of the last_name column from 30 to 150 characters.  If your user table is above a million rows, the to make this update might be long enough to cause downtime.  edX.org has opted to skip this migration and has limited the length of the last name field to the previous 30 characters in the user change forms.  It's recommend that openedx do the same.

    • The relevant command to run before you run the rest of the migration is the following: 

      sudo -HEu www-data /edx/bin/python.edxapp /edx/bin/manage.edxapp lms migrate --fake auth 0009

      • Run this command after loading the new code but before running the migrations.
• Django Configuration and SetUp
• Remote Configuration (Cory Lee (Deactivated), Bill DeRusha (Deactivated))
  • TBD - Need to provide description of the impact of this change.
  • lms/cms removal of aws.py DEPR-14 - Getting issue details... STATUS plugins should use production.py at this point.
    
    

• JWT Cookies require JWK setup. This was disabled in Ironwood.  Thanks to Pierre Mailhot for the following steps to generate and configure the JWT signing keys. (Robert Raposa, Nimisha Asthagiri (Deactivated))

   Click here to expand...

  sudo -H -u edxapp bash
  cd ~
  source edxapp_env
  source venvs/edxapp/bin/activate
  cd edx-platform
  python manage.py lms generate_jwt_signing_key

  The last command generates values for JWT_PUBLIC_SIGNING_JWK_SET, JWT_PRIVATE_SIGNING_JWK and JWT_SIGNING_ALGORITHM that needs to be filled in into /edx/etc/lms.yml. After restarting the LMS and the CMS, you will be able to login.

  Alternatively, the following piece of code (created for Tutor) can be generated to manually generate the private/public signing keys.

• Third Party Auth Organizations linking is no longer managed via ApplicationOrganization model.  This model will be removed and the linking will be done by explicitly defining filters for the application.  See relevant decisions:
  • https://github.com/edx/edx-platform/blob/master/openedx/core/djangoapps/oauth_dispatch/docs/decisions/0011-scope-filter-support.rst
  • https://github.com/edx/edx-platform/blob/master/openedx/core/djangoapps/oauth_dispatch/docs/decisions/0012-scope-and-filter-for-third-party-auth.rst
• We upgraded sorl-thumbnail from a version that didn't have migrations to one that does, so the initial migration needs to be faked in any environment that previously installed the old version: ./manage.py lms migrate thumbnail --fake .
• Management Commands
  • As part of https://github.com/edx/edx-platform/pull/20698, send_verification_expiry_email command arguments, --resend-days and --days-range were removed from the management command and moved to lms common settings. (Zainab Amir)
    
    
• Celery
  • The django-celery library has been removed from edx-celeryutils. The corresponding management command has now been deleted and should be removed from upgrade instructions. (Tim McCormack, Diana Huang)

• Inter-service Communication Impact
  • OAuth2 DOP→DOT Migration (Nimisha Asthagiri (Deactivated), Julia Eskew (Deactivated), Troy Sankey, Robert Raposa, Diana Huang, Jeremy Bowman (Deactivated))
    • DEPR-47 - Getting issue details... STATUS
    • Any OAuth Applications/Clients registered in the LMS that are using the old Django-OAuth-Provider will need to be migrated to Django-OAuth-Toolkit.  These applications include your Open edX Mobile Apps (iOS, Android) as well as your Open edX services (e-commerce, Insights, Credentials, etc.) 
      • See Setup OAuth Client for Internal Services (Django Oauth Toolkit version)
        • Note: there exist management commands used in devstack, but edx.org did this manually.
  • A new OAuth2 scope, 'user_id', is required for SSO access between the LMS and satellite Open edX services (e-commerce, Insights, Credentials, etc.)  For existing Open edX instances, please add the 'user_id' scope for each of your satellite services at /admin/oauth_dispatch/applicationaccess/.  You may use a management command to do so (see https://github.com/edx/edx-platform/pull/20076).  For newly provisioned instances, this scope will be added automatically with https://github.com/edx/devstack/pull/394. (Robert Raposa)
    
    
  • Ecommerce expects to find a unique user id (LMS user id) for each user, either in a JWT or social auth. If it cannot find this id, a MissingLmsUserIdException will be raised; this exception can be silenced by enabling the allow_missing_lms_user_id waffle switch. The LMS user id can be back-filled for existing users by running the ecommerce/core/management/commands/import_user_ids.py management command. See ecommerce/docs/decisions/0004-unique-identifier-for-users.rst. (crice (Deactivated))
    
    
  • In edx-platform/lms/djangoapps/courseware/courses.py - The function check_course_access has been renamed to check_course_access_with_redirect.  A new implementation of check_course_access returns AccessResponse objects, rather than performing redirects when access is denied.  check_course_access_with_redirect is a drop-in replacement for the old behavior of check_course_access.  This is likely a breaking change for any code outside of edx-platform (such as plugin apps) that was using the check_course_access function directly.  See https://github.com/edx/edx-platform/pull/23651 for more information.

• Mobile Apps
  

   Click here to expand...

  • Android
    • Android 2.20.2 Added a testing feature of the 'Upgrade to Verified' course.
    • Android 2.20.1
      • You can now view the edX app in French, German, Portuguese, Chinese, and Turkish. The application uses your device language to specify this setting.
      • It's now clearer when there are areas of a course that cannot be completed in the app or are unavailable for audit track learners.
      • We have fixed the ‘Downloading Videos’ to phone memory issue. All users having any OS and device model should be able to download them on phone memory now.
      • We have hidden/removed the ‘Downloading Videos’ to SD-Card feature for OS 9 and above for now (as this beta feature is having many issues on them and team will prioritize its fixes in near future). While all users having any device and OS below 9 can enable the feature from settings and should face no issue.
    • Android 2.19.0
      • Watch and learn at your own pace by adjusting the video play speed controls.
      • You can now choose to store downloaded videos in your SD card from the settings area.
    • Android 2.18.2 Facebook login/register button brand enforcement.
    • Android 2.18.1 You can now search edX programs and degrees using the mobile application.
    • Android 2.17.1 Accessibility improvements
  • iOS
    • iOS 2.20.0 You can now view the edX app in German, Hebrew, French, Portuguese, Chinese, and Turkish. The application uses your device language to specify this setting.
    • iOS 2.18.1 You can now search edX programs and degrees using the mobile application!
    • iOS 2.17.1 Accessibility improvements.
• Forum:
  • The MONGOID_AUTH_MECH environment variable needs to be defined, even if just empty. Otherwise the following error will be triggered: "in `encoded_name': undefined method `encode' for nil:NilClass(NoMethodError)".
• Database migration:
  • Due to a migration in "SiteConfiguration", it is not possible to have an Ironwood instance serving traffic while a Juniper one is being set up with zero downtime. It will result with OperationalError exceptions being raised until the Ironwood instance is shutdown. (Slack thread)

Impact on Your Code

• edx-platform
  • The courseware Django app is now only imported as "lms.djangoapps.courseware".  If you import it as "courseware", you may get errors.  Fix the imports.
  • DEPR-16 - Getting issue details... STATUS
    
    
  • Comprehensive theming is now disabled by default. To enable it, make sure to add ENABLE_COMPREHENSIVE_THEMING = True in the edx-platform settings of both the LMS and the CMS.
• Frontend
  • Paragon enhancements and edx-bootstrap (Adam Butterworth (Deactivated))
    • The Paragon reusable component and pattern library has seen significant investment and refinement since the last Open edX release. A number of components have been re-engineered for ease of use and flexibility. We’ve also launched an interactive documentation site (https://edx.github.io/paragon) where you can explore the components, their implementation status, as well as the library’s foundational CSS utility classes.
  • frontend-platform (David Joy (Deactivated))

    • The Frontend Platform is a framework for Open edX micro-frontend applications. It provides a number of common services used by frontend apps, including analytics (Segment), logging (New Relic), authentication and HTTP/AJAX request client (Axios/JWT), and internationalization (React Intl). It also includes a number of helpful React components and utilities.  The framework is highly configurable and extensible, allowing override implementations of analytics and logging, as well as a number of customization hooks for the application. It should be used as a basis for new micro-frontends. A detailed documentation site can be found here: https://edx.github.io/frontend-platform

  • Frontend Repos
    
    
  • We have created a new Microfrontend named Publisher (https://github.com/edx/frontend-app-publisher/) for creating courses and course runs. Although primarily used to keep track of course metadata (such as marketing information), Publisher can also help manage ecommerce products. Publisher integrates between edx-platform, ecommerce, and course-discovery. (Dillon Dumesnil)
    • Note: At this point in time, there is no standard process for installing Microfrontends, but Publisher is still being provided so the community can become more familiar with it and possibly configure and install it on their own.
• Upgrades
  • Python-3: OEP-7, Python 3 How-To Note: Ubuntu 16.04 (and thus the ansible playbooks) ships with Python 3.5.1, but Python 3.5.9 was found to work in Tutor, and it works "better").
    
  • Django-2: Django 2.2 Upgrade Plan
  • Node-12
  • Ruby was upgraded to v2.5.7 in cs_comments_service (forum). Bundler v1.11.2 and Rake v10.4.2 versions remain identical in the configuration repo, but I (Régis Behmo) am completely unable to get them to work in Tutor. I had to upgrade to Bundler v1.17.3 and Rake v13.0.1.
  • MongoDb: upgraded from 3.2.16 to 3.6.17
  • Pip: upgraded from 19.3.1 to 20.0.2
  • Setuptools: upgraded from 39.0.1 to 44.1.0
• DEPRs (see below)
  
  

• Relevant ADRs
  

   Click here to expand...

  • Discovery/Publisher
    • https://github.com/edx/course-discovery/blob/master/docs/decisions/0005-salesforce-integration.rst
    • https://github.com/edx/course-discovery/blob/master/docs/decisions/0006-disallow-course-number-changes-in-publisher.rst
    • https://github.com/edx/course-discovery/blob/master/docs/decisions/0009-LMS-types-in-course-metadata.rst
  • edx-platform
    • Platform-wide
      • https://github.com/edx/edx-platform/blob/master/docs/decisions/0001-courses-in-lms.rst
      • https://github.com/edx/edx-platform/blob/master/docs/decisions/0002-inter-app-apis.rst
    • Access and Permissions
      • https://github.com/edx/edx-platform/blob/master/openedx/features/course_duration_limits/docs/decisions/0001-access-messages.rst
      • https://github.com/edx/edx-platform/blob/master/lms/djangoapps/courseware/docs/decisions/0003-permissions-via-bridgekeeper.rst
    • Course Dates
      • https://github.com/edx/edx-platform/blob/master/common/lib/xmodule/xmodule/docs/decisions/0001-move-due-dates-to-db.rst
      • https://github.com/edx/edx-platform/blob/master/lms/djangoapps/courseware/docs/decisions/0004-personalized-relative-dates.rst
  • e-Commerce
    • https://github.com/edx/edx-platform/blob/master/openedx/features/discounts/docs/decisions/0001-determining-discount-location.rst
    • https://github.com/edx/ecommerce/blob/b127ac52f19c49cf8571af2d80ae6e7e20167302/docs/decisions/0004-unique-identifier-for-users.rst
    • https://github.com/edx/ecommerce/blob/58446dc91e1ef266b499b2ebf559de8001ec8234/docs/decisions/0005-payment-microfrontend-support-and-bff-apis.rst

Deprecations and Removals

Based on Juniper Dates: 2019-01-17 to 2020-05-27

Deprecations

Removals

• DEPR-12 - Getting issue details... STATUS
  • Removal of microsites includes the removal of several tables that were in this djangoapp: https://github.com/edx/edx-platform/pull/21355/
  • We have removed the entire djangoapp, but if installations would like to clean up these tables after upgading, they can drop all the tables prefixed by microsite_configuration.
• DEPR-52 - Getting issue details... STATUS
  • Removed now unused /login_post endpoint, which was just used by Studio login.  Note: /login_ajax points to the same login code.  BOM-1096 - Getting issue details... STATUS (was ARCH-1253)
  • Several breaking changes were made to clean-up login_user, used by our login POST endpoints.
    • JSON is now always returned, including for the third-party auth errors.
    • 400 status is now used for errors instead of 200. JSON included 'success': 'failure' both before and after the change.
• DEPR-51 - Getting issue details... STATUS  : https://github.com/edx/edx-platform/pull/22006 (Bill DeRusha (Deactivated))
• edx-certificates is no longer installed. This was an error.  edx-certificates is still part of Juniper.

Feature Changes

• Will Public Course Content be on by default?
  
  
• Learner Portal. learner-portal (https://github.com/edx/frontend-app-learner-portal) and it's accompanying configuration service (https://github.com/edx/portal-designer) were created
  
  
• SAML. Irfan Uddin Ahmad: As part of https://github.com/edx/edx-platform/pull/20665 we have updated the username match criterion for SAML SSO users to be based on email only instead of both email & username.
  
  
• e-Commerce
  • As part of https://github.com/edx/ecommerce/pull/2621, the payment page microfrontend is the default payment page for edx ecommerce. To use your own microfrontend, you need to add enable_microfrontend_for_basket_page to your site configuration as well as payment_microfrontend_url . If you do not use the microfrontend, you can use the waffle flag disable_microfrontend_for_basket_page to see the old basket page for testing, etc. (Brandon Baker (Deactivated))
    
  • Performance improvements, etc. (Emma Green (Deactivated))
    • TBD - Need to provide a description of feature changes in ecommerce since April 2019 as it impacts the community. (Perhaps links to ADRs.)
      
      
• LMS
  • Learner Dashboard. Adeel Ehsan (Deactivated): As part of https://github.com/edx/edx-platform/pull/20439 by default LMS dashboard will show only limited courses based on the course limit set in configuration. Show all the courses link will be at bottom if enrolled courses count is greater than course limit.
    
    
  • Instructor Dashboard
    • Problem Response Report. As part of (https://github.com/edx/edx-platform/pull/19781) the problem response report API was enhanced with support for generating combined reports for multiple blocks and for limiting reports to specified block types. Additionally, the block paths in reports are now absolute and start at the root/course level.
    • Course Data Researcher Role. https://github.com/edx/edx-platform/pull/22336 (David St. Germain (Deactivated))
      • https://partners.edx.org/announcements/coming-soon-enhanced-protections-learner-data
    • Profile Report. It's now possible to include last login date and creation date in the student profile CSV export, for instance to find out which students never logged in. See related PR. Daniel Clemente
      
      
  • Gradebook
    • TBD - Need to provide a description of changes in gradebook since April 2019 as it impacts the community.
      
      
  • Proctoring
    • TBD - Need to provide a description of changes in proctoring since April 2019 as it impacts the community.
      
      
  • ORA-2
    • TBD - Need to provide a description of changes in ORA-2 since April 2019 as it impacts the community.
       
  • Course Dates
    • TBD - Need to provide a description of changes to course dates since April 2019 as it impacts the community.
      
      
  • SGA xBlock
    • Python3 and Django 2.2 compatibility added in version 0.11.0. This version is not backward compatible with Python2 and Ironwood.
      

• Studio
  • Learning Sequence Authoring Speed (David St. Germain (Deactivated))
    • https://partners.edx.org/announcements/coming-soon-authoring-speed-quick-wins
      
      
  • A new OLX REST API can be used to get the OLX "source code" of any XBlock in a course without having to first export the whole course as a tarball.
    • https://github.com/edx/edx-platform/pull/23068
      
      
  • New course design interface with horizontal tabs
• Course Discovery
  • We introduced a few new models (CourseType, CourseRunType, Track, Mode) to Course Discovery as a way of managing LMS tracks and Ecommerce Products for Courses and Course Runs. Details can be found here: https://github.com/edx/course-discovery/blob/master/docs/decisions/0009-LMS-types-in-course-metadata.rst. (Dillon Dumesnil)
    • If converting to using Publisher and wanting your courses set up with types, use the backpopulate-course-type management command found here: https://github.com/edx/course-discovery/blob/master/course_discovery/apps/course_metadata/management/commands/backpopulate_course_type.py. Please pay attention to the section on mismatches in the decision document above if running into issues with the management command. Alternatively, the Refresh Course Metadata command will try and match Courses and Course Runs to their types when run.
      
      
• Course Discovery + Studio
  • Changes have been made to how organizations are managed in the edX infrastructure. When Organizations are saved in Course Discovery, we sync the information to edx-platform so Course Discovery is able to be a single source of edit. See https://openedx.atlassian.net/browse/DISCO-1414 and the associated Pull Requests linked in the ticket for more details. (Dillon Dumesnil)
    
    
  • We have added a new flag ENABLE_PUBLISHER to edx-platform and course-discovery that should both be set if you are using the Publisher Microfrontend. Setting this flag will hide certain fields from Studio and make others read-only. This is to enable Publisher to be the only source of edit for these fields. Setting the flag in course-discovery prevents Refresh Course Metadata from pulling in some fields that become no longer editable in Studio. See details in https://github.com/edx/edx-platform/pull/21516 (Dillon Dumesnil)
    • Note: if setting the ENABLE_PUBLISHER flag, you must also define a Publisher URL for your Partner object in the Course Discovery service.