Elasticsearch 7.10

This page is under construction.

What is this about?

  • Briefly describe service and its role and impact on our systems.

  • Who owns it within edX?

  • What do we plan to do with it in a nutshell?

  • Is this a small, medium or large project (ballpark)? 

  • Any other relevant context to help understand importance, size or impact.

Why are we doing this?

  • Why does this need to be done? What is the impact of not doing this?

  • Could we be better off not doing it, postponing, retiring the service, or replacing it with another?

  • Why did we pick this version to upgrade to?

  • Why does this need to be done now?

  • Can someone else do this for us?

Have we done this before?

  • Have we done this before for in edX.org instance or for any other Open edX instance?

  • Did the last time go well?

  • Do we want to do things differently this time, and how?

  • Provide links to wiki pages of similar previous projects

What do we plan to do?

  • What is the scope of this project?

  • On a high level, what is our plan? Identify big milestones.

  • Who do we plan to involve (groups of stakeholders)?

Are there risks and dependencies?

  • Are there any known risks?

  • How can we mitigate them?

  • Have we reviewed system requirements (e.g. AWS, Python, etc.) for this upgrade?

  • Are there dependencies with other projects?

  • Are there any third-party integrations that might be affected by this upgrade? What other systems or services rely on this service?

What is the impact of this upgrade on various users of the system?

  • Which user groups or personas will this upgrade impact? E.g. developers, learners, educators, site operators, etc.

  • Are there any new features?

  • Are there any features or functionality that are being removed as a result of the upgrade?

How will we do it?



How will we communicate about it?

  • How will communication take place (internally and externally)?

  • What mode of communication will be used? E.g. regular or private Slack channel, will it be internal to edX or shared with Open edX space, email, regular meetings.

  • How often will routine communication take place? Is there a need for regularly touching base?

edX wants to upgrade Elasticsearch to version 7.10 in time for Maple Open edX release.

Diane has a lot of knowledge about the previous Elasticsearch upgrade - https://openedx.atlassian.net/wiki/spaces/COMM/pages/1678639164. It was a Blended project with Racoon Gang.

This time we expect owning squads to do the upgrade, after which SRE will flip the switch.

Scope

Upgrade all IDAs that are on 7.8, see list at https://openedx.atlassian.net/wiki/spaces/AC/pages/2946203703.

All of these IDAs are part of the Open edX named release.

All tickets for this project will have ElasticSearch-7-10-upgrade label in JIRA.

SRE work will also be tracked in https://openedx.atlassian.net/browse/ISRE-1006.

Environments and path: devstack -> sandbox -> stage -> prod/edge (we only have one mongo cluster per env).

Dependencies

Elasticsearch upgrade can be done independently from other upgrade projects, e.g. MongDB, Django, etc.

Each upgraded IDA can be release independently, as soon as it is done.

Timeline

Though, we need to complete this anytime before October 15, we aim at finishing and upgrade prod by September 7.

However, since more services are used by Elasticsearch than by MongoDB, we want to get MongoDB upgrade out of the way first, then do Elasticsearch.

More detailed timeline to follow.

Project Team

Name

Squad

Role

Name

Squad

Role

@lcicchese

SRE

Tech and Project Lead

@Natalia Berdnikov

Open edX

Project Manager

@Christopher Pappas

SRE

Lead Developer

@Jeremy Ristau

TNL (IDA owner)

Team Lead, development

@Waheed Ahmed

Vanguards (IDA owner)

Team Lead, development

@Simon Chen

Cosmonauts (IDA owner)

Team Lead, development

@Zach Hancock

Cosmonauts (IDA owner)

Lead Developer

@Nimisha Asthagiri (Deactivated)

Open edX

Stakeholder (Infrastructure lead)

@Ned Batchelder

Open edX

Stakeholder (maker of Open edX named release)

Open edX Community

Open edX Community

Stakeholder (Consumers of Maple named release)

 

Next steps

@Christopher Pappas Answer questions on this page as you find fit.

DONE

@Natalia Berdnikov Send email to squads about this coming, need to be done for Maple. Kick off meeting.
@lcicchese Create epic for SRE, link here. https://openedx.atlassian.net/browse/ISRE-1006
@lcicchese Reach out to Simon about how SRE can help upgrade Analytics API ES 1.5 to 7.10.
@Adam Blackwell Open Devstack 7.10 PR (https://github.com/edx/devstack/pull/782/files )
@Adam Blackwell Notes Devstack testing PSRE ticket (https://openedx.atlassian.net/browse/PSRE-871 )
@Natalia Berdnikov Copy PSRE ticket to other service owners & put in table: https://openedx.atlassian.net/browse/PSRE-871