Elasticsearch Upgrade Task Force

Goals:

Establish communication between different parts of engineering that rely on ES
Make decisions about what the future of ES at edX will be
- What version will we target?
- What technologies will we use going forward? (e.g. Haystack, edx-search, elastisearch-dsl)
- How do we support these on Devstack?
  - Currently, Devstack is the limiting factor in doing service-by-service upgrades.
- Are there things we don’t want to upgrade?
Put together a plan for the upgrade
- How can we break up the work between teams?

The Current State

ES version

Version we use is 1.5

this is only supported by AWS at the moment.

Current version of ES is 7.5

Elasticsearch End of Life chart

Support Windows Spreadsheet

Services

Service Name	Technologies Used	Owner(s)	Representative

Service Name	Technologies Used	Owner(s)
edx-platform edx-enterprise	edx-search	Tech owners are listed on tech ownership spreadsheet https://docs.google.com/spreadsheets/d/1qpWfbPYLSaE_deaumWSEZfz91CshWd3v3B7xhOk5M4U/edit#gid=840878297
edx-notes	Haystack → elasticsearch-dsl-drf
forums	elasticsearch, elasticsearch-model
course-discovery	Haystack → elasticsearch-dsl-drf
edx-analytics-data-api edx-analytics-pipeline	elasticsearch-dsl
blockstore

Technologies

Haystack

current version: 2.8.1
used versions: 2.8.1 (course-discovery), 2.8.0 (edx-notes-api)
supported ES versions: 1.x, 2.x

edx-search

current version: 1.3.3
used versions: 1.3.3 (edx-platform), 1.2.2 (edx-enterprise)
supported ES versions: 1.x
Known issues:
- https://openedx.atlassian.net/browse/BOM-1036

elasticsearch-dsl

current version: 7.1.0
used versions: 0.0.11 (edx-analytics-data-api)
supported ES versions: all?

elasticsearch-model

current version: 7.x
used versions: 0.1.9(cs_comments_service)
supported ES versions: all

elasticsearch (ruby)

current version: 7.x
used versions: 1.1.2(cs_comments_service)
supported ES versions: all

Kickoff Agenda

Introductions
Mission Statement
Decision making!
- versions
  - What is the latest version that AWS supports?
    - Currently this is ES 7.1
    - Haystack dev/pre-release branch does support ES5
      - Maybe an intermediate migration
  - Let’s target ES5 for now
  - Is Haystack query language used in APIs? @Michael Terry (Deactivated)
    - Check both Discovery and Notes
- elasticsearch-dsl (y/n)
  - Can we move edx-search to using this?
  - Let’s try upgrading edx-search to see if things work?
- django-elasticsearch-dsl-drf (y/n)
Devstack
- Who can look into this work?
  - Resources will constrain what we can do
  - Maybe only have 2 ES versions running simultaneously at max
  - Set up an ES 5 container in devstack as well @Diana Huang
    - individual services can point to this
Forums (the red-headed step child)
- Let’s just try upgrading along with everything else @Alan Zarembok (Deactivated)
Start up a sandbox with a newer version and see what breaks?
- Requires configuration change to set the value of the version @Fred Smith (Deactivated)
- Need branches in each repo for the library update @Diana Huang
Open questions
- What is the pain of upgrading between 5 and 7? @Diana Huang
- What features does ES support in the analytics api and how important are they? @Stuart Young (Deactivated)

Agenda

Updates

We have an ES5 container in Devstack
Haystack -
- Notes & Discovery: we’re not using anything Haystack specific here
Minimal usage in analytics-api
- Type changes in there
- Looked into updating the version of dsl and seeing what breaks
Maybe we can attempt to make the upgrade in some of our other services as well
- Understand why vanilla devstack doesn’t work for forums
- Will try upgrade for edx-search and seeing what breaks
- notes
- discovery
Frequency of meetings
- Reduce to half an hour
Action Items
- @Stuart Young (Deactivated) to continue looking into analytics api work
- @Alan Zarembok (Deactivated) to get forums working on devstack
- @Fred Smith (Deactivated) to get sandboxes building against ES5
- @Michael Terry (Deactivated) to start work on Discovery
- @Dave Ormsbee (Deactivated) to look into what breaks with edx-search
- @Diana Huang to look into upgrade edx-notes and update meeting times.

Agenda (3/20/20) -

Forums - still not working in devstack
- @Alan Zarembok (Deactivated) - maybe ask someone on Lahore to see if they can do something about this.
@Stuart Young (Deactivated) - still working on failures on analytics api
@Diana Huang - edx-notes, still running into issues with make upgrade
@Fred Smith (Deactivated) - still working on sandboxes
@Michael Terry (Deactivated) - discovery - managed to get things to ES5 and tests fail
- Lack of good documentation
@Dave Ormsbee (Deactivated) - switched to edx-search

Agenda 4/30/20 -

Discuss Blended Brief
- Discuss the possibility of moving off Haystack as a Blended Brief
  - @Diana Huang to come up with a draft
- Scope down the work to having forums, edx-search, and analytics-data-api to latest version of ES blended brief
  - got buy-in from owners on reviewing/scoping work
Made decision to target ES7 because there are still major limitations to using ES5 with our haystack stack (see notes)

Useful notes/resources

Elasticsearch-py 5.0.0+ requires changing from using the create to index API call (see: Creating document with automatic ID generation doesn't work in 5.0 · Issue #474 · elastic/elasticsearch-py )
The docs for elasticsearch-py indicate that you must use a matching major version. And in my testing, that’s true – or at least, that the surrounding changes in our own code (renamed parameters, etc) don’t work with both server versions.
Old (server) release notes, each with breaking changes listed: https://www.elastic.co/guide/en/elasticsearch/reference/index.html
Migration helper plugin for server for 1->2 and 2->5: GitHub - elastic/elasticsearch-migration: This plugin will help you to check whether you can upgrade directly to the next major version of Elasticsearch, or whether you need to make changes to your data and cluster before doing so.
- Only warning for 1->2 upgrade locally: “Boolean fields will return 1/0 instead of T/F in scripts, aggregations, or sort values, in fields: modelresult:has_enrollable_paid_seats, modelresult:has_enrollable_seats, modelresult:hidden, modelresult:hidden_exact, modelresult:is_current_and_still_upgradeable, modelresult:is_program_eligible_for_one_click_purchase, modelresult:mobile_available, modelresult:mobile_available_exact, modelresult:published, modelresult:published_exact.” – we need to see if we rely on old behavior anywhere
- Only warning for 2->5 upgrade locally: “Indices created before v2.0.0 must be reindexed” – which means we must upgrade to 2.x, reindex, then upgrade to 5.x.
While beta versions of django-haystack 3.0 do support ES 5.x, drf-haystack (used by Discovery at least) requires the 2.8.x version of django-haystack. So we’d also need to get them to release a beta (and they are light on maintainers).

Upgrade plan

Get server ready

Make a backup of the existing elasticsearch database.
Make a cloned ES server under a new hostname.
Upgrade cloned ES server to 2.x.
Reindex everything through their migration plugin.
Make another database backup.
Upgrade cloned ES server to 5.x.
At this point, the server’s data will start becoming stale, and we should update clients in short order.

Now update code

Update devstack with a script to do the above server upgrade.
Start landing PRs to update client services to elasticsearch-py 5.x in concert with config changes that point at the new ES hostname.
Message out the changes, letting people know they should update devstack.

Cleanup

Document the change and how to upgrade servers in our Open edX release notes.
Decommision old ES 1.x servers once all services are upgraded.