Runbooks from the ES1 migration

Generalized Strategy

These will have to be done twice in most cases, first on stage and then on prod so that we can uncover any major infrastructural issues.

0. Create/Find a Test Bed on Stage and Prod

Test before upgrade
After upgrade
After re-index

1. Create ES7 clusters in terraform

An example PR: https://github.com/edx/terraform/pull/2830

2. Create a new setting for the ES7 configuration

Set this setting to point to the new clusters

3. Spin up new instances of app with ES7-compatible code

This is a manual process
eSREs will rely on SRE for assistance with how to do this

4. Index ES data from ES7-compatible instance

Each app has its own method for indexing data, will need to be discussed with eSRE during setup and implementation
Understand complications based on deploy time (i.e. missing notes that were added between indexing + deploy)

5. Merge ES7 code and configuration changes

6. Deploy using GoCD using the newly built changes

Communicate schedule to #support ahead of time.

7. Re-index ES7 to include any writes that were missed

8. Clean up settings and clusters

Remove old settings
Remove old clusters from terraform

Specific App Strategies

edx-notes

Plan deploy for week of Sept 7.

Does not have remote config, due to Kubernetes
Can’t send out config synchronously with image
- Set different variables for different clusters
  - Different code uses different variables
Can we hide search from users temporarily?
Action Items:
- Diana Huang - try to get this working in devstack
  - PR: https://github.com/edx/edx-notes-api/pull/181
  - Try adding new ELASTICSEARCH_DSL in the yaml, clean up HAYSTACK_CONNECTIONS and ELASTICSEARCH_URL after we confirm that the deploy goes smoothly
- Diana Huang - schedule time next week to try to deploy to stage
- https://openedx.atlassian.net/browse/PSRE-260

course-discovery

Instead of using the existing ELASTICSEARCH_URL var for the url for the new cluster. We’ll add a new one ELASTICSEARCH_CLUSTER_URL to make the cutover easier and less error prone.
Runbook
1. Merge remote-config PR
2. Merge course-discovery PR
3. Deploy to stage.
  1. After Deploy
    1. manage.py search_index --create ← This is needed if it’s a fresh ES cluster.
    2. manage.py update_index --disable-change-limit
  2. Test on stage.
Other Useful commands:

mysqldump -u read_only_iam -h <DATABASE_HOST_NAME> --enable-cleartext-plugin --ssl-ca=/edx/bin/rds-combined-ca-bundle.pem -p"$(aws rds generate-db-auth-token --hostname <DATABSE_HOSTNAME> --port 3306 --region us-east-1 --username read_only_iam )" discovery --single-transaction course_metadata_additionalpromoarea course_metadata_backfillcourserunslugsconfig course_metadata_backpopulatecoursetypeconfig course_metadata_bulkmodifyprogramhookconfig course_metadata_collaborator course_metadata_corporateendorsement course_metadata_corporateendorsement_individual_endorsements course_metadata_course course_metadata_course_authoring_organizations course_metadata_course_collaborators course_metadata_course_expected_learning_items course_metadata_course_prerequisites course_metadata_course_sponsoring_organizations course_metadata_course_subjects course_metadata_courseeditor course_metadata_courseentitlement course_metadata_courserun course_metadata_courserun_staff course_metadata_courserun_transcript_languages course_metadata_courserun_video_translation_languages course_metadata_courseruntype course_metadata_courseruntype_tracks course_metadata_coursetype course_metadata_coursetype_course_run_types course_metadata_coursetype_entitlement_types course_metadata_coursetype_white_listed_orgs course_metadata_courseurlredirect course_metadata_courseurlslug course_metadata_curriculum course_metadata_curriculumcoursemembership course_metadata_curriculumcourserunexclusion course_metadata_curriculumprogrammembership course_metadata_dataloaderconfig course_metadata_degree course_metadata_degree_rankings course_metadata_degreecost course_metadata_degreedeadline course_metadata_deletepersondupsconfig course_metadata_drupalpublishuuidconfig course_metadata_endorsement course_metadata_expectedlearningitem course_metadata_faq course_metadata_historicalcourse course_metadata_historicalcourseentitlement course_metadata_historicalcourserun course_metadata_historicalcourseruntype course_metadata_historicalcoursetype course_metadata_historicalcurriculum course_metadata_historicalcurriculumcoursemembership course_metadata_historicalcurriculumcourserunexclusion course_metadata_historicalcurriculumprogrammembership course_metadata_historicaldegreecost course_metadata_historicaldegreedeadline course_metadata_historicalmode course_metadata_historicalorganization course_metadata_historicalprogram course_metadata_historicalprogramtype course_metadata_historicalseat course_metadata_historicaltrack course_metadata_icontextpairing course_metadata_image course_metadata_joboutlookitem course_metadata_leveltype course_metadata_leveltypetranslation course_metadata_migratecommentstosalesforce course_metadata_migratecommentstosalesforce_orgs course_metadata_migratepublishertocoursemetadataconfig course_metadata_migratepublishertocoursemetadataconfig_orgs course_metadata_mode course_metadata_organization course_metadata_pathway course_metadata_pathway_programs course_metadata_person course_metadata_personareaofexpertise course_metadata_personsocialnetwork course_metadata_position course_metadata_prerequisite course_metadata_profileimagedownloadconfig course_metadata_program course_metadata_program_authoring_organizations course_metadata_program_corporate_endorsements course_metadata_program_courses course_metadata_program_credit_backing_organizations course_metadata_program_excluded_course_runs course_metadata_program_expected_learning_items course_metadata_program_faq course_metadata_program_individual_endorsements course_metadata_program_instructor_ordering course_metadata_program_job_outlook_items course_metadata_programtype course_metadata_programtype_applicable_seat_types course_metadata_programtypetranslation course_metadata_ranking course_metadata_removeredirectsconfig course_metadata_searchdefaultresultsconfiguration course_metadata_searchdefaultresultsconfiguration_courses course_metadata_searchdefaultresultsconfiguration_programs course_metadata_seat course_metadata_seattype course_metadata_subject course_metadata_subjecttranslation course_metadata_syllabusitem course_metadata_tagcourseuuidsconfig course_metadata_topic course_metadata_topictranslation course_metadata_track course_metadata_video core_partner catalogs_catalog > coursediscovery_data.sql

mysql -h edx.devstack.mysql -u discov001 -ppassword discovery < coursediscovery_data.sql

edx-search

Pause prod pipeline
Merge changes to master
Have revert PR available

After code is on stage:

Run these commands

python manage.py lms reindex_course_team --all
python manage.py cms reindex_course --all

Do testing in stage
- Make sure to test teams.
If we feel confident with stage, unpause prod pipeline
Prevent prod/edge deploy from deploying - not added to ELB
- Index data - prod/edge
- This needs to be discovered and investigated - is it a checkbox? - Fred Smith (Deactivated)
  - Checkbox - switch deploy_ami step from ‘On Success’ to 'Manual'
  - Manually set up ASGs for
Diana Huang - schedule a 4 hour window for attempting to deploy

cs_comment_service

Example of running the catchup command with rake.

sudo -HEu forum /bin/bash -c "source /edx/app/forum/forum_env; rake search:catchup[comments_20201014184547124,comment_threads_20201014184547124,22601]"