/
Ginkgo

Ginkgo

Dec 18, 2017

The seventh Open edX release.  Please add information here that will be useful when it comes time to package the release.  Please include your name when you add an item, so that we can get back to you with questions.

New Services & Major Upgrades Planned for This Release

What new products, applications, services, etc have we built since Ginkgo?

Installation changes

Things you'd like to complete before Ginkgo

Things to cherry-pick onto Ginkgo

The open-release/ginkgo.master branch has been created in each repo.  Changes for Ginkgo will now have to be cherry-picked onto that branch.  This raises the bar for changes: they must be important, and they must not be unduly disruptive.

Cherry-picks for Ginkgo.2

Future cherry-picks 

Deployment Issues to Keep in Mind

What has changed in the devops/configuration world? Are there manual steps needed to move from Ficus to Ginkgo?

  • ElasticSearch will be upgraded, probably to 1.5 which is what we use in production.  This will coincide with changes to the forums service to support the newer version.

    • Robert Raposa: In addition to migration documentation, we should remind about "rake search:initialize" and "rake search:validate_index".

  • Credentials Service

    • The service now requires Python 3.5

    • The service has been upgraded to Django 1.11.x

  • E-Commerce Service

    • Receipt Page Updates

      • We now only support the receipt page integrated into Otto. Redirecting to the LMS/shopping_cart receipt page is no longer enabled.

    • CyberSource Updates

      • Secure Acceptance Silent Order POST is now the only supported integration. This integration will require creating a new profile in the business center.

      • The merchant notifications endpoint has been removed.

    • The service has been upgraded to Django 1.10.x.

      • This change includes a migration that affects the user table. If your users table is large, consider faking migration core 0032. (@ClintonB (Deactivated))

    • Django Oscar has been upgraded to 1.4. (The steps below only apply to existing installations, and can be ignored for new installations.)

      • This upgrade requires faking migrations for the thumbnail app: ./manage.py migrate thumbnail --fake

      • Site maintainers should be aware that one of the migrations includes a change on the guest_email column in the orders table. If your orders table is large 1M+ rows, this migration may lock the table for an extended amount of time. The E-Commerce Service does not normally use the guest_email column. If you have not modified your system to use this column, and wish to avoid the table lock, it is recommended that you fake this migration:
        ./manage.py migrate orders 0012

        ./manage.py migrate orders 0013 --fake

      • @Robert Raposa A "fake thumbnails" Ansible task was added to resolve this issue in devstack and on sandboxes.  I cannot speak about Production or other environments, and whether or not this task could be used in those cases.

    • The Course Administration Tool (CAT) has been updated to use the user account associated with the OAuth credentials rather than individual users' accounts. The user associated with the OAuth credentials—we at edx.org use the username ecommerce_worker—should have full CRUD permissions for the following models on LMS:

      • commerce.CommerceConfiguration

      • course_modes.CourseMode

      • credit.CreditCourse

      • credit.CreditRequest

    • Additionally, ecommerce_worker should have the student:userprofile:can_deactivate_users permission if you are using SDN verification.

  • Catalog Service

    • The service is in the process of been upgraded to Django 1.11.X

  • Waffle is used by many repositories to control feature rollout (@Andy Armstrong (Deactivated))

  • Will catalog service be required?

  • Are there required upgrades for Django, pip, ...

  • Will there be a significant change to the asset pipeline in time for Ginkgo? (Probably not but something to keep an eye on) 

  • Upgrade to RabbitMQ https://github.com/edx/configuration/pull/3297 (@Kevin Falcone (Deactivated))

  • Insights (@Tyler Hallada (Deactivated))

    • We're planning on upgrading Insights to Django 1.11.  As part of this upgrade, we need to follow a delicate procedure to update python-social-auth from the old version in Eucalyptus.  Essentially, what we think needs to be scripted in the upgrade is a sequence of migrations documented in python-social-auth/MIGRATING_TO_SOCIAL:

      • upgrade to python-social-auth==0.2.21

      • run migrations

      • upgrade edx-auth-backends, which in turn upgrades a bunch of social-auth-* packages

      • run migrations

    • In other words, upgrade Insights to version 0.33.0, run migrations, then upgrade Insights to the Ginkgo release, and run migrations again.

  • python-social-auth has been upgraded to social-auth-app-django 1.2.0 and social-auth-core 1.4.0.

    • In some cases, this may cause migration issues.

      • For the initial release of Ficus, we were at python-social-auth 0.2.12.

      • This is not true:

        • For a later version of Ficus, we moved to python-social-auth 0.2.21 as a preliminary step to migrating to the split social architecture.

      • The upgrade to 0.2.21 included a number of database migrations that essentially brought our database into the same state as would be required by social-auth-app-django. Once that was performed, we were able to move to social-auth-app-django without additional database migrations.

      • In cases where the Open edX host has not first moved to a version of Ficus that includes python-social-auth 0.2.21, an intermediate step will be necessary to perform those database migrations, as a direct database migration from PSA 0.2.12 to social-app-django is not possible.

  • django-celery was updated from a version that used South migrations to a version that used django migrations, and the structure changed in the process so --fake-initial doesn't help.

    • Please run the drop_djcelery_tables management command (from edx-celeryutils) after you update code, but before running migrations. If not, djcelery_0001 will yield a "table already exists" error.

    • ./manage.py lms drop_djcelery_tables --settings=aws

    • This command assumes the tables are empty, as no code in edx-platform was using them previously. If there is data in your tables, the management command will exit without making any changes and you'll need to fix up the relevant tables manually.

  • The edx-platform heartbeat (courses(LMS)/studio) format has changed, it is now:

    • { "modulestore": { "status": true, "message": "OK" }, "sql": { "status": true, "message": "OK" } }

      We recommend automated checks relying on this endpoint focus on the HTTP status code, and that this output be used primarily for diagnostics

Monthly product notes

Any unusual things changed in the product that adopters or developers should know about?

Deprecated Features

  • Programs was deprecated in Ficus and will be replaced by the Catalog service in Ginkgo

  • Credentials API V1 is replaced by V2 (V1 has been removed)

  • The waffle flag "unified_course_view" is deprecated in Ginkgo, which can be used for the new view of the course outline on a separate page.  The old sidebar navigation and this waffle flag will be fully removed in the next release. (@Robert Raposa)

    • This is currently defaulted off, but we'd like to get it defaulted on for Ginkgo.