Elasticsearch Upgrade Task Force
Goals:
Establish communication between different parts of engineering that rely on ES
Make decisions about what the future of ES at edX will be
What version will we target?
What technologies will we use going forward? (e.g. Haystack, edx-search, elastisearch-dsl)
How do we support these on Devstack?
Currently, Devstack is the limiting factor in doing service-by-service upgrades.
Are there things we don’t want to upgrade?
Put together a plan for the upgrade
How can we break up the work between teams?
The Current State
ES version
Version we use is 1.5
this is only supported by AWS at the moment.
Current version of ES is 7.5
Elasticsearch End of Life chart
Services
Service Name | Technologies Used | Owner(s) | Representative |
---|---|---|---|
edx-platform
| edx-search | Tech owners are listed on tech ownership spreadsheet |
|
edx-notes | Haystack → elasticsearch-dsl-drf |
|
|
forums | elasticsearch, elasticsearch-model |
|
|
course-discovery | Haystack → elasticsearch-dsl-drf |
|
|
edx-analytics-data-api
| elasticsearch-dsl |
|
|
blockstore |
|
|
|
Technologies
Haystack
current version: 2.8.1
used versions: 2.8.1 (course-discovery), 2.8.0 (edx-notes-api)
supported ES versions: 1.x, 2.x
edx-search
current version: 1.3.3
used versions: 1.3.3 (edx-platform), 1.2.2 (edx-enterprise)
supported ES versions: 1.x
Known issues:
current version: 7.1.0
used versions: 0.0.11 (edx-analytics-data-api)
supported ES versions: all?
current version: 7.x
used versions:
0.1.9
(cs_comments_service)supported ES versions: all
current version: 7.x
used versions:
1.1.2
(cs_comments_service)supported ES versions: all
Kickoff Agenda
Introductions
Mission Statement
Decision making!
versions
What is the latest version that AWS supports?
Currently this is ES 7.1
Haystack dev/pre-release branch does support ES5
Maybe an intermediate migration
Let’s target ES5 for now
Is Haystack query language used in APIs? @Michael Terry (Deactivated)
Check both Discovery and Notes
elasticsearch-dsl (y/n)
Can we move edx-search to using this?
Let’s try upgrading edx-search to see if things work?
django-elasticsearch-dsl-drf (y/n)
Devstack
Who can look into this work?
Resources will constrain what we can do
Maybe only have 2 ES versions running simultaneously at max
Set up an ES 5 container in devstack as well @Diana Huang
individual services can point to this
Forums (the red-headed step child)
Let’s just try upgrading along with everything else @Alan Zarembok (Deactivated)
Start up a sandbox with a newer version and see what breaks?
Requires configuration change to set the value of the version @Fred Smith (Deactivated)
Need branches in each repo for the library update @Diana Huang
Open questions
What is the pain of upgrading between 5 and 7? @Diana Huang
What features does ES support in the analytics api and how important are they? @Stuart Young (Deactivated)
Agenda
Updates
We have an ES5 container in Devstack
Haystack -
Notes & Discovery: we’re not using anything Haystack specific here
Minimal usage in analytics-api
Type changes in there
Looked into updating the version of dsl and seeing what breaks
Maybe we can attempt to make the upgrade in some of our other services as well
Understand why vanilla devstack doesn’t work for forums
Will try upgrade for edx-search and seeing what breaks
notes
discovery
Frequency of meetings
Reduce to half an hour
Action Items
@Stuart Young (Deactivated) to continue looking into analytics api work
@Alan Zarembok (Deactivated) to get forums working on devstack
@Fred Smith (Deactivated) to get sandboxes building against ES5
@Michael Terry (Deactivated) to start work on Discovery
@Dave Ormsbee (Deactivated) to look into what breaks with edx-search
@Diana Huang to look into upgrade edx-notes and update meeting times.
Agenda (3/20/20) -
Forums - still not working in devstack
@Alan Zarembok (Deactivated) - maybe ask someone on Lahore to see if they can do something about this.
@Stuart Young (Deactivated) - still working on failures on analytics api
@Diana Huang - edx-notes, still running into issues with make upgrade
@Fred Smith (Deactivated) - still working on sandboxes
@Michael Terry (Deactivated) - discovery - managed to get things to ES5 and tests fail
Lack of good documentation
@Dave Ormsbee (Deactivated) - switched to edx-search
Agenda 4/30/20 -
Discuss Blended Brief
Discuss the possibility of moving off Haystack as a Blended Brief
@Diana Huang to come up with a draft
Scope down the work to having forums, edx-search, and analytics-data-api to latest version of ES blended brief
got buy-in from owners on reviewing/scoping work
Made decision to target ES7 because there are still major limitations to using ES5 with our haystack stack (see notes)
Useful notes/resources
Elasticsearch-py 5.0.0+ requires changing from using the
create
toindex
API call (see: https://github.com/elastic/elasticsearch-py/issues/474#issuecomment-256903012 )The docs for elasticsearch-py indicate that you must use a matching major version. And in my testing, that’s true – or at least, that the surrounding changes in our own code (renamed parameters, etc) don’t work with both server versions.
Old (server) release notes, each with breaking changes listed: https://www.elastic.co/guide/en/elasticsearch/reference/index.html
Migration helper plugin for server for 1->2 and 2->5: https://github.com/elastic/elasticsearch-migration
Only warning for 1->2 upgrade locally: “Boolean fields will return
1/0
instead ofT/F
in scripts, aggregations, or sort values, in fields:modelresult:has_enrollable_paid_seats
,modelresult:has_enrollable_seats
,modelresult:hidden
,modelresult:hidden_exact
,modelresult:is_current_and_still_upgradeable
,modelresult:is_program_eligible_for_one_click_purchase
,modelresult:mobile_available
,modelresult:mobile_available_exact
,modelresult:published
,modelresult:published_exact
.” – we need to see if we rely on old behavior anywhereOnly warning for 2->5 upgrade locally: “Indices created before v2.0.0 must be reindexed” – which means we must upgrade to 2.x, reindex, then upgrade to 5.x.
While beta versions of django-haystack 3.0 do support ES 5.x, drf-haystack (used by Discovery at least) requires the 2.8.x version of django-haystack. So we’d also need to get them to release a beta (and they are light on maintainers).
Upgrade plan
Get server ready
Make a backup of the existing elasticsearch database.
Make a cloned ES server under a new hostname.
Upgrade cloned ES server to 2.x.
Reindex everything through their migration plugin.
Make another database backup.
Upgrade cloned ES server to 5.x.
At this point, the server’s data will start becoming stale, and we should update clients in short order.
Now update code
Update devstack with a script to do the above server upgrade.
Start landing PRs to update client services to elasticsearch-py 5.x in concert with config changes that point at the new ES hostname.
Message out the changes, letting people know they should update devstack.
Cleanup
Document the change and how to upgrade servers in our Open edX release notes.
Decommision old ES 1.x servers once all services are upgraded.