Date: Fri, 29 Mar 2024 00:16:40 +0000 (UTC) Message-ID: <1942278110.43.1711671400907@d6e4db8463e1> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_42_624950663.1711671400906" ------=_Part_42_624950663.1711671400906 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
This article describes the upgrade path and expectations for IDA= s moving from MySQL 5.6 to 5.7
1: This work has been completed as of 2/2, however this page has been le= ft for contextual purposes.
2: This page is public, but some of the details here pertain only to edX= deployments, and many of the links are to protected pages.
Other Upgrades Upgrade Activity= p>
Coordinate with product to determine an upgrade timeframe for devstack.<= /p>
Update the =E2=80=9CPlanned 5.7 DevStack Upgrade=E2=80=9D column below.<= /p>
Please link any created tickets to https://openedx.atlassian.net/browse/OPS-4652
Identify dependency changes
If your app uses any python database libraries, please update these.
=Upgrading Devstack
Find your app=E2=80=99s docker-compose block(s) in <= a class=3D"external-link" data-card-appearance=3D"inline" href=3D"https://g= ithub.com/edx/devstack" rel=3D"nofollow">https://github.com/edx/devstack
Update your apps container to point at mysql57
instead of <=
code>mysql and update any related make commands.=
(e.g. https://github.com/edx/devstack/blob/a76042b19aeb8=
fc8b02d93d3db4be783a54a0be1/docker-compose-marketing-site.yml#L15 )
If possible run e2e tests to see if the change causes any i= ssues.
Merge your PR and notify engineering if any manual steps are needed to r=
ecreate data locally.
(e.g. make dev.dbcopy57.<service>
, see 10/6 =E2=80=9CDi=
scovery switching to mysql 5.7=E2=80=9D email thread from Mike fo=
r examples)
If manual steps are needed and your service is an Open edX service, plea= se also post to Discourse (e.g. https://discuss.openedx.org/t/discovery-switching-to-mysql-5-7/3360)= and add make sure the Koa release planning notes= reflect your changes.
If you learned any lessons that would be valuable to other teams, please= describe or link to them here.
Upgrading Travis or CircleCI
Make sure that whatever Travis or CircleCI tests you have which use MySQ= L are using a 5.7 container. (e.g. https://github.co= m/edx/xqueue/pull/783/files)
Upgrading Sandboxes
SRE will be upgrading Sandboxes before we upgrade stage with this PR: https://github.com/edx/configuration/pull/5695= /files
However you may need to coordinate the timing of persistent sandbox depl=
oyments.
(e.g. partner registrar sandboxes should get rebuilt after prod gets upgrad=
ed, int sandboxes get upgraded ahead of time)
Be on the lookout for any performance impacts once SRE notifies you that= stage and prod are being upgraded.
Note: RDS Aurora MySQL Databases will likely require downtime, but RDS M= ySQL databases can be upgraded without downtime.
If there are downtime constraints for prod please let the SRE team know,= we will be estimating the downtime and choosing which upgrade path makes t= he most sense and will be attempting to
As of right now we only know of improvements, however, some libraries li= ke django-mysql <1.0.6 do not support MySQL 5.7 and= may need to be updated (edx-platform is on 3.0.0). We also may us= e some features of MySQL 5.6 that are no longer available in django-mysql o= r mysqlclient.
The SRE Team is tracking each stage, prod and edge database upgrade unde= r this Jira epic: h= ttps://openedx.atlassian.net/browse/PSRE-301
As each of our services have different requirements (owners, data pipeli= nes), each database upgrade may have specific steps required.
Each of the services in the diagram below (except Forums, Prospectus &am= p; Marketing), has a MySQL database that will be upgraded to Aurora 5.7, in= addition to the service, our data pipelines also connect to these database= s so special care should be taken to ensure those pipelines and the downstr= eam reports are not impacted by the upgrades.
At least the LMS, Ecommerce, Discovery, License Manager, Enterprise Cata= log, Demographics and Credentials services also have DBT generated Snowflak= e views described here, please be sure to sync with DE = about requirements before doing each cutover.
At least LMS and Ecommerce have Jenkins EMR and/or Swoop jobs, these job= s should be checked carefully before and after each respective cutover.
=The current proposed order for updating our databases is here (please ke= ep this in sync with https:= //openedx.atlassian.net/wiki/spaces/DE/pages/1961558140/Updating+Stitch+Int= egrations+for+MySQL+5.7+Upgrade#Tentative-Upgrade-Order ):
Notes (SRE) - Done on 10/23 (not on Aurora)
Discovery & Ecommerce (Engagement) (not on Aurora)
Discovery - Done on 10/29
Ecommerce - Done on 11/17 @ 10pm
Credentials and Demographics (Aperture)
Credentials - Done on 11/17 @ 3pm
Demographics - Done on 12/7 @ 10am
Registrar and Portal Designer (Masters)
Registrar Target - Done on 12/2 @ 10am
Designer Target - Done 12/9 @ 10am
Analytics API and Insights (DE)
Prod Analytics API and Analytics Data - Done 12/17 @ 10am-12pm ET
Edge Analytics - Done for 12/18 @ 10am-12:11pm
XQueue (T&L)
Move Stage xqueue schema from edxapp db to shared db Done
Prod Xqueue Target -Done on 1/5 @ 10am
License Manager, Enterprise Catalog, Blockstore, Video Encode Manager =
(Enterprise, T&L, Incident Management)
Shared Cluster - Done
Platform (Arch + TNL):
Stage edxapp - Done on 1/25
Edge edxapp - Done on 1/27
Prod csmhe & edxapp - Done on 2/2
Note: These steps may be followed twice if we do a segmented upgrade to = support TLS 1.2 first on 5.6.48 before going to 5.7 (e.g for notes)
Stage:
We will be sending out a message to #dev and @'ing @sre= -team + the technical owners listed below for each stage database upgrade, = if you would like to be on the cc list for a specific service please add yo= ur name to the column.
At the service owner or SRE team members request we will cr= oss post announcements to #warroom or other channels. (note, we may = also pause pipelines)
Once maintenance is complete & any pipelines are unpaused we will po= st in the #dev announcement thread with =E2=80=9Calso send to channel=E2=80= =9D checked to let engineers know that their services should be back up sho= uld notify us if they see anything unusual.
At the service owners request we will communicate to a specified enginee= r or channel to ask them to manually run smoke tests or trigger e2e tests.<= /p>
Prod & Edge:
We will be coordinating any required maintenance windows with the busine= ss and technical owner for each service which will be documented in the tab= le below and the corresponding tickets.
At the service owners request we can also include external contractors w= ho are impacted in this coordination (e.g. OpenCraft/LabXchange).
At the service owners request we will post a status.edx.org maintenance window, which will usually be either 1pm-2pm or 1am-2am de= pending on which engineers need to be available, on status page, (e.g. https://status.edx.org/incidents/v1n8lpl6sz13 - we = will not be posting to status page for edge notes)
We will post an announcement in #warroom and @'ing @sre= -team + @status + the service owners at the beginning of the maintenance wi= ndow.
At the service owners request we will cross post #warroom announcements = to other channels (e.g. #partner-support)
Once maintenance is complete we will perform an appropriate smoke test, = update the status page.
We will update the announcement thread with =E2=80=9Calso send to channe= l=E2=80=9D checked to let engineers know that their services should be back= up should notify us if they see anything unusual.
Read Replicas:
Read Replicas (e.g. for Platform, Ecommerce & Discovery) will be han= dled on a case by case basis.
Service |
Business and Technical Owners (see list |
Current MySQL Versions (best effort) |
In Open edX? |
5.7 DevStack Upgrade Due November 9th if in Open edX, December 15th if not. |
5.7 Travis Upgrade Due November 9th, December 15th if not. |
5.7 Sandbox Upgrade |
Planned 5.7 Stage Upgrade Due December 15 |
Planned 5.7 Prod Upgrade (+ Any Read Replicas) Due Jan 1 |
Planned 5.7 Edge Upgrade |
Maintenance Mode / Smoke Test Docs |
---|---|---|---|---|---|---|---|---|---|---|
Platform (LMS + CMS) + csm= he |
Arch - |
Stage: 5.7 |
Yes |
https://openedx.= atlassian.net/browse/BOM-2059 - Muhammad = Arif (Deactivated) Upgraded on 10/29 |
Tests run on Jenkins instead of Travis https://openedx.= atlassian.net/browse/BOM-2059 Upgraded on 10/29 |
SRE will create a test sandbox once https://openedx.atlassian.net/browse/BOM-2059 is do=
ne. |
(dedicated aurora)https://openedx.a= tlassian.net/browse/PSRE-329 is done. Upgraded on 01/26 @ 1 AM CNAME: stage-edx-edxapp.rds.edx.org |
Coordinate with Jeremy/Nimisha/Sarina/Stu/Feanil Note: There are two databases. (dedicated aurora)https://openedx.a=
tlassian.net/browse/PSRE-332 Completed on 2/2 |
Coordinate with Jeremy/Nimisha/Sarina/Stu/Feanil Completed on 1/27 |
Maintenance Mode: See Runbook. Smoke Test: edxapp + csmhe Smoke Tests Run Book= a> cc Sarina Canelake (Do Not Use) (Deactivat= ed) |
Engagement - |
Stage: 5.7 |
Yes |
https://github.com/ed=
x/devstack/pull/639 |
Travis File<=
/a> https://openedx.= atlassian.net/browse/REV-1568, now ecom team helping via https://openedx.atlassian.net/bro= wse/REV-1568 |
(dedicated aurora) |
(dedicated aurora) |
Not applicable (ecommerce not in Edge) |
Maintenance Mode: 1: Disable ASG in Asgard |
||
Aperture - |
Stage: 5.7 |
Yes |
|
N/A |
(shared aurora) |
(dedicated aurora) |
N/A |
Maintenance Mode: Disable ASG in Asgard |
||
Aperture - |
Stage: 5.7 |
No |
Done as part of https://github.com/edx/demog=
raphics/pull/80/files |
N/A |
Adam Blackwell (Deactivated) ask Matt Tuchf= arber (Deactivated) if demographics runs in sandbox. |
(dedicated aurora) |
(dedicated aurora) |
N/A |
Maintenance Mode: Disable in ArgoCD Smoke Test: Adam Blackwell (Deactivated)= a> will reach out to Matt Tuchfarber (Deactivated) |
|
SRE - |
Stage: 5.7 |
Yes |
https://github.com/ed=
x/devstack/pull/629 |
Travis F=
ile |
(shared aurora) |
(dedicated mysql) |
(dedicated mysql) |
Maintenance Mode: We can=E2=80=99t currently put just notes in maintenan=
ce, but the LMS fails gracefully for users and we can use Cloudflare. |
||
T&L - |
Stage: 5.7 |
Yes |
Upgraded as of https:/= /openedx.atlassian.net/browse/PSRE-322) UPGRADED 11/19 |
https://openedx.= atlassian.net/browse/PSRE-322 https://github.com/edx/xqueue/pull/783 UPGRADED 11/19 |
(edxapp aurora) |
(dedicated mysql) Will be done by SRE team, need coordination with Feanil Patel |
N/A |
Check with Feanil Patel Maintenance Mode: Disable ASG in Asgard Smoke Test: |
||
Registrar & Workers <= /td> |
Programs - |
Stage: 5.7 |
No |
Upgraded as of |
No Travis Docker Tests |
(shared aurora) |
(dedicated mysql) Endpoint for Stitch: |
N/A |
Maintenance Mode: Disable in Asgard Smoke Test: https://openedx.atlassian.net/wiki/x/bYCCeQ |
|
Enterprise Catalog<= /p> |
Enterprise - |
Stage: 5.7 |
Yes |
Upgraded as of |
No Travis Docker Tests |
(shared aurora) |
(shared aurora) |
N/A |
Maintenance Mode: Disable in Asgard |
|
Enterprise - |
Stage: 5.7 |
Yes, for now |
Upgraded as of |
No Travis or CircleCI |
(shared aurora) |
(shared aurora) |
N/A |
Maintenance Mode: Disable in Asgard Smoke Test: |
||
Engage - |
Stage: 5.7 |
Yes |
Upgraded as of |
Upgraded Travis as of |
(shared aurora) |
(dedicated mysql) |
N/A |
|||
Masters - |
Stage: 5.7 |
No |
Upgraded local compose file as of |
No Travis or CircleCI |
(shared aurora) |
(dedicated aurora) |
N/A |
Maintenance Mode: Disable in Asgard Smoke Tests: Designer Smoke Tests Run= Book |
||
T&L - |
Stage: 5.7 |
Yes |
https://openedx.= atlassian.net/browse/TNL-7642 https://openedx= .atlassian.net/browse/OSPR-5101 Done 11/6 |
N/A |
(shared cluster) |
(shared cluster) |
N/A |
|||
Analytics Pipeline |
Data Engineering - T:Brian Beggs |
N/A |
Yes |
N/A |
|
|||||
Data Engineering - T:Brian Beggs |
Stage: 5.7 |
Yes |
N/A (According to Brian Beggs , insight= s and analytics-api do not have a devstack.) |
Tr= avis doesn=E2=80=99t use MySQL |
Doesn=E2=80=99t have sandboxes. |
(shared aurora) = https://openedx.atlassian.net/browse/PSRE-304 No action items |
(analytics aurora) |
(analytics mysql)
|
Maintenance Mode: Brian Beggs will chec=
k with his team + Asgard |
|
Insights and/or Analytics Dashboard |
Data Engineering - T:Brian Beggs |
Stage: 5.7 |
Yes |
N/A (According to Brian Beggs , insight= s and analytics-api do not have a devstack.) |
Travis doesn=E2=80=99t use MySQL |
Doesn=E2=80=99t have sandboxes. |
(shared aurora) = https://openedx.atlassian.net/browse/PSRE-304 No action items |
(analytics aurora) |
(analytics mysql) |
Maintenance Mode: Brian Beggs will chec= k with his team + Asgard Smoke Test: Download button in Insights |
ensure retirement |
Simon Chen , Julie Davis (Deactivated) - repo ownership Jere= my Bowman (Deactivated) - for deprecation |
N/A |
No |
Done 11/13 https://ope= nedx.atlassian.net/browse/EDUCATOR-5378 - closed in favor of https://openedx.atlassian.net/brow= se/DEPR-106?searchSessionId=3D4511027c-eb42-4c57-b290-aaec5aaf03fa&sear= chObjectId=3D186639&searchContainerId=3D17023&searchContentType=3Di= ssue |
No Travis Docker or CircleCI Tests |
N/A |
N/A |
|||
Video Encode Manager |
Stage: 5.7 |
Yes |
Not in DevStack: https://openedx.atlassian.net/browse/PSRE-371 |
N/A |
N/A |
(shared cluster) |
(shared cluster) Completed |
N/A |
Kash= if Chaudhry (Unlicensed) Are you aware of how SRE can smoketest this se= rvice after we complete this database upgrade? |
|
Licence Manager |
Stage: 5.7 |
Yes |
N/A |
(shared cluster) |
(shared cluster) Completed |
N/A |
||||
ensure retirement |
Website: |
N/A |
No |
Will be retired before by then - https://openedx.atlassian.net/browse/WS-1413 |
Will be retired before then -https://openedx.atlassian.net/browse/WS-1413 |
N/A |
Needs to be removed from shared |
Needs to be removed from shared |
N/A |
N/A |
Enterprise Reporting |
Markhors: |
Stage: 5.7 |
No |
N/A |
N/A |
N/A |
N/A |
N/A |
https://openedx.= atlassian.net/browse/DOS-1619 - needs to be done by 11/9
MySQL Database 5.6 becomes end of life in Feb, 2021
Open edX Juniper will ship with support for 5.6, but Open edX Koa will s= hip with support for 5.7
Application Owners should plan to upgrade to 5.7 as part of the Koa rele= ase (December 9. Upgrades should be in place by Nov 9.)
Some apps (like edxapp, ecommerce, discovery may define MySQL URLs outsi= de of their compose files)
The following services do not need to be upgraded within this project, o= r upgrade is not applicable to them.
https://github.com/e= dx/taxonomy-connector - it is currently an IDA, but the goal is t= o turn it into a plugin for edx-platform, so we don't believe any work is n= eeded.
There are no items with the selected labels at this time.