Elasticsearch Ownership and Rollout

Owned by Diana Huang
Last updated: Jan 14, 2021
2 min read

Context

See [BD 19-20] Technical Plan (Haystack Replacement, Elasticsearch Upgrade)

Runbook

ES7 deployment strategy

Effort Coordination

This blended project upgrades our outdated and unsupported Elasticsearch from version 1.5 to version 7 in each of our services. We will need a resource from each owning squad to do a final review and deploy of the Elasticsearch upgrade for their owning service. The expected effort is:

  • Squad engineer’s review of PR: 2-5 hours depending on the complexity of the service’s use of Elasticsearch (PR will already be reviewed carefully by @Diana Huang or @Michael Terry (Deactivated))

  • Squad engineer’s deploy and monitoring of the migration: 2-3 days depending on production rollout

Service/Repo

Owning Team

PRs

Current Blocker

Status

Service/Repo

Owning Team

PRs

Current Blocker

Status

terraform

Arch-BOM

 

@Tim McCormack working on updating ES clusters to ES 7.8.

devstack

SRE / Arch-BOM

https://github.com/edx/devstack/pull/611

https://github.com/edx/devstack/pull/607

 

  • notes

  • comments-service

  • edx-platform

  • course-discovery

  • analytics

edx-notes-api

SRE
#site-reliability-engineering

Contact:@Fred Smith (Deactivated)
ARCH-BOM: @Diana Huang

[BD-20] Haystack Replacement and Elasticsearch upgrade for edx notes api by NikolayBorovenskiy · Pull Request #181 · openedx/edx-notes-api

[BD-20] Elasticsearch fix: return course-specific notes by NikolayBorovenskiy · Pull Request #200 · openedx/edx-notes-api

indexing on prod caused errors due to malformed tags - checking to see if we can/should fix this.

  • PR review

  • PR merge

  • Deploy to Stage

  • Deploy to Edge

  • Deploy to Prod (target week: Sep 21st)

course-discovery

Engage (Always Available)
#discovery

Contact: @Michael Terry (Deactivated) ARCH-BOM: @Feanil Patel (Deactivated)

[NBorovensky]Haystack migration to dsl by NikolayBorovenskiy · Pull Request #12 · raccoongang/course-discovery

[BD-20][RACCOON GANG][NBorovenskiy] Haystack replacement to es-dsl. Upgrade to ElasticSearch 7 by NikolayBorovenskiy · Pull Request #2724 · openedx/course-discovery - Merged and reverted on Stage.

Waiting for new PR with original code + fixes.

Fixes code: [BUGS][NBorovenskiy]/OeX_ES=57 problems that were discovered after the first testing of the es7 course discovery service on live data on edx stage env by NikolayBorovenskiy · Pull Request #13 · raccoongang/course-discovery

  • RG working with @Feanil Patel (Deactivated) and @Michael Terry (Deactivated) to resolve search query discrepancies.

  • PR review

  • Manual test

  • PR merge

  • Devstack PR merge

  • Deploy and Test on Stage

  • Deploy to Prod (week: Jan 4th)

edx-search

TNL
#tnl-squad

Contact: @sburch (Deactivated)
ARCH-BOM: @Tim McCormack

[BD-19] Transition to the Elasticsearch 7.8.0 version by Golub-Sergey · Pull Request #104 · openedx/edx-search

https://github.com/edx/edx-platform/pull/25081

 

 

  • PR review

  • Test confidence

  • edx-search PR merge

  • edx-platform PR merge

  • Devstack PR merge

  • Deploy and Test on Stage

  • Deploy to Prod (Actual week: Nov 16)

cs_comments_service

TNL
#tnl-squad

Contact: @David Joy (Deactivated)
ARCH-BOM: @Robert Raposa

https://github.com/edx/cs_comments_service/pull/327

 

  • PR review

  • Test confidence

  • PR merge

  • Devstack PR merge

  • Deploy and Test on Stage

  • Deploy to Prod (target week: Oct 27th)

edx-analytics-data-api
edx-analytics-pipeline

Data Engineering
#data-engineering

Contact: @Stuart Young (Deactivated)
ARCH-BOM: @Diana Huang

https://github.com/edx/edx-analytics-data-api/pull/389

https://github.com/edx/edx-analytics-pipeline/pull/835

@Stuart Young (Deactivated) review and testing

  • PR review

  • Manual test

  • PR merge

  • Devstack PR merge

  • Deploy and Test on Stage

  • Deploy to Prod (target week: Nov)