ES7 deployment strategy

Owned by Diana Huang
Last updated: Oct 29, 2020 by Feanil Patel

Runbooks from the ES1 migration

https://openedx.atlassian.net/wiki/spaces/ArchiveEng/pages/151191637

https://openedx.atlassian.net/wiki/spaces/ArchiveEng/pages/158541260

https://openedx.atlassian.net/wiki/spaces/ArchiveEng/pages/158541200

Generalized Strategy

These will have to be done twice in most cases, first on stage and then on prod so that we can uncover any major infrastructural issues.

0. Create/Find a Test Bed on Stage and Prod

  • Test before upgrade

  • After upgrade

  • After re-index

1. Create ES7 clusters in terraform

  • An example PR:

2. Create a new setting for the ES7 configuration

  • Set this setting to point to the new clusters

3. Spin up new instances of app with ES7-compatible code

  • This is a manual process

  • eSREs will rely on SRE for assistance with how to do this

4. Index ES data from ES7-compatible instance

  • Each app has its own method for indexing data, will need to be discussed with eSRE during setup and implementation

  • Understand complications based on deploy time (i.e. missing notes that were added between indexing + deploy)

5. Merge ES7 code and configuration changes

6. Deploy using GoCD using the newly built changes

  • Communicate schedule to #support ahead of time.

7. Re-index ES7 to include any writes that were missed

8. Clean up settings and clusters

  • Remove old settings

  • Remove old clusters from terraform

 

 

Specific App Strategies

edx-notes

Plan deploy for week of Sept 7.

  • Does not have remote config, due to Kubernetes

  • Can’t send out config synchronously with image

    • Set different variables for different clusters

      • Different code uses different variables

  • Can we hide search from users temporarily?

  • Action Items:

    • @Diana Huang - try to get this working in devstack

      • PR:

      • Try adding new ELASTICSEARCH_DSL in the yaml, clean up HAYSTACK_CONNECTIONS and ELASTICSEARCH_URL after we confirm that the deploy goes smoothly

    • @Diana Huang - schedule time next week to try to deploy to stage

course-discovery

  • Instead of using the existing ELASTICSEARCH_URL var for the url for the new cluster. We’ll add a new one ELASTICSEARCH_CLUSTER_URL to make the cutover easier and less error prone.

  • Runbook

    1. Merge remote-config PR

    2. Merge course-discovery PR

    3. Deploy to stage.

      1. After Deploy

        1. manage.py search_index --create ← This is needed if it’s a fresh ES cluster.

        2. manage.py update_index --disable-change-limit

      2. Test on stage.

  • Other Useful commands:

mysqldump -u read_only_iam -h <DATABASE_HOST_NAME> --enable-cleartext-plugin --ssl-ca=/edx/bin/rds-combined-ca-bundle.pem -p"$(aws rds generate-db-auth-token --hostname <DATABSE_HOSTNAME> --port 3306 --region us-east-1 --username read_only_iam )" discovery --single-transaction course_metadata_additionalpromoarea course_metadata_backfillcourserunslugsconfig course_metadata_backpopulatecoursetypeconfig course_metadata_bulkmodifyprogramhookconfig course_metadata_collaborator course_metadata_corporateendorsement course_metadata_corporateendorsement_individual_endorsements course_metadata_course course_metadata_course_authoring_organizations course_metadata_course_collaborators course_metadata_course_expected_learning_items course_metadata_course_prerequisites course_metadata_course_sponsoring_organizations course_metadata_course_subjects course_metadata_courseeditor course_metadata_courseentitlement course_metadata_courserun course_metadata_courserun_staff course_metadata_courserun_transcript_languages course_metadata_courserun_video_translation_languages course_metadata_courseruntype course_metadata_courseruntype_tracks course_metadata_coursetype course_metadata_coursetype_course_run_types course_metadata_coursetype_entitlement_types course_metadata_coursetype_white_listed_orgs course_metadata_courseurlredirect course_metadata_courseurlslug course_metadata_curriculum course_metadata_curriculumcoursemembership course_metadata_curriculumcourserunexclusion course_metadata_curriculumprogrammembership course_metadata_dataloaderconfig course_metadata_degree course_metadata_degree_rankings course_metadata_degreecost course_metadata_degreedeadline course_metadata_deletepersondupsconfig course_metadata_drupalpublishuuidconfig course_metadata_endorsement course_metadata_expectedlearningitem course_metadata_faq course_metadata_historicalcourse course_metadata_historicalcourseentitlement course_metadata_historicalcourserun course_metadata_historicalcourseruntype course_metadata_historicalcoursetype course_metadata_historicalcurriculum course_metadata_historicalcurriculumcoursemembership course_metadata_historicalcurriculumcourserunexclusion course_metadata_historicalcurriculumprogrammembership course_metadata_historicaldegreecost course_metadata_historicaldegreedeadline course_metadata_historicalmode course_metadata_historicalorganization course_metadata_historicalprogram course_metadata_historicalprogramtype course_metadata_historicalseat course_metadata_historicaltrack course_metadata_icontextpairing course_metadata_image course_metadata_joboutlookitem course_metadata_leveltype course_metadata_leveltypetranslation course_metadata_migratecommentstosalesforce course_metadata_migratecommentstosalesforce_orgs course_metadata_migratepublishertocoursemetadataconfig course_metadata_migratepublishertocoursemetadataconfig_orgs course_metadata_mode course_metadata_organization course_metadata_pathway course_metadata_pathway_programs course_metadata_person course_metadata_personareaofexpertise course_metadata_personsocialnetwork course_metadata_position course_metadata_prerequisite course_metadata_profileimagedownloadconfig course_metadata_program course_metadata_program_authoring_organizations course_metadata_program_corporate_endorsements course_metadata_program_courses course_metadata_program_credit_backing_organizations course_metadata_program_excluded_course_runs course_metadata_program_expected_learning_items course_metadata_program_faq course_metadata_program_individual_endorsements course_metadata_program_instructor_ordering course_metadata_program_job_outlook_items course_metadata_programtype course_metadata_programtype_applicable_seat_types course_metadata_programtypetranslation course_metadata_ranking course_metadata_removeredirectsconfig course_metadata_searchdefaultresultsconfiguration course_metadata_searchdefaultresultsconfiguration_courses course_metadata_searchdefaultresultsconfiguration_programs course_metadata_seat course_metadata_seattype course_metadata_subject course_metadata_subjecttranslation course_metadata_syllabusitem course_metadata_tagcourseuuidsconfig course_metadata_topic course_metadata_topictranslation course_metadata_track course_metadata_video core_partner catalogs_catalog > coursediscovery_data.sql
mysql -h edx.devstack.mysql -u discov001 -ppassword discovery < coursediscovery_data.sql

  • Pause prod pipeline

  • Merge changes to master

  • Have revert PR available

  • After code is on stage:

    • Run these commands

      python manage.py lms reindex_course_team --all python manage.py cms reindex_course --all

  • Do testing in stage

    • Make sure to test teams.

  • If we feel confident with stage, unpause prod pipeline

  • Prevent prod/edge deploy from deploying - not added to ELB

    • Index data - prod/edge

    • This needs to be discovered and investigated - is it a checkbox? - @Fred Smith (Deactivated)

      • Checkbox - switch deploy_ami step from ‘On Success’ to 'Manual'

      • Manually set up ASGs for

  • @Diana Huang - schedule a 4 hour window for attempting to deploy

cs_comment_service

Example of running the catchup command with rake.