Upgrade base OS from Ubuntu 16.04 to 20.04

This article describes the upgrade path and expectations for IDAs moving from ubuntu 16.04 LTS to Ubuntu 20.04 LTS. We will be skipping over 18.04.

This page is public, but some of the details here pertain only to edX deployments, and many of the links are to protected pages.

 

Platform, Blockstore and Designer Portal use Aurora clusters in stage or production so they will not be able to be upgraded to Ubuntu 20.04 until the databases are on MySQL 5.7. Ubuntu 20.04 requires TLS 1.2 system wide which isn’t supported for 5.6. You’ll see SSL errors for your database connections if you try.
For regular MySQL RDS, TLS 1.2 is supported for MySQL 5.6.46 and later

Instructions

  1. Upgrading your testing environment

    1. Sandboxes

      1. Update the base AMI on your sandbox build to Ubuntu_20.04

         

      2. Run the sandbox build for your application, fix bugs, repeat.

  2. Travis CI or CircleCI

    1. This might require you to update the devstack docker container your app runs in. If your app runs travis integration tests in a docker container pulled from the edxops org, you may have to do step 5 (below) first.

    2. Update the dist: field in your .travis.yml file to focal

  3. Identifying dependency changes

    1. Ansible system packages

      1. Look at the package list passed into the edx_django_service_debian_pkgs_extra parameter for your ansible role. This will be in the meta/main.yaml file, and will probably reference a variable set in defaults/main.yaml

      2. Look at tasks/main.yaml and look for calls to apt.

      3. Look up the differences between an 16.04 sandbox vs. a 20.04 sandbox (created above) by running dpkg -l | grep packagename on each machine type.

      4. on the 20.04 server, run apt-get changelog packagename to see the changelog. read through the changelog to determine if any changes need to be accounted for in your use of that dependency. Ask SRE if you need specific advice.

  4. Upgrading Devstack

    1. Find your app in https://github.com/edx/configuration/tree/master/docker/build

    2. change the line From edxops/xenial-common to From edxops/focal-common in the Dockerfile

    3. run the jenkins job that rebuilds the base image. (https://tools-edx-jenkins.edx.org/job/DockerCI/job/image-builders/)

  5. Upgrading your deployment (this may be a separate ticket)

    1. Make sure all Jenkins jobs that hit this service support the correct version of python and Ubuntu

    2. GoCD Base AMI

      1. Find your IDA’s staging environment config in https://github.com/edx/gomatic-internal/tree/master/gocd/vars/tools/environment-deployment-play

      2. Change the value for base_ami_ubuntu_version to 20.04

      3. Your pipelines should reseed after your changes are merged, if you have issues ask SRE to reseed your pipelines manually.

 Significant Changes since 16.04

  • Python 2 is no longer in the base image. It will be installed by the ansible bootstrap method, but it may not be identical to what you had on 16.04.

  • NTPd is no longer used, and instead timesyncd is used.

Timeline

Service

Owner

Current Ubuntu Version

In Open edX?

Ubuntu 20.04 DevStack Upgrade

due November 9th

Ubuntu 20.04 Travis Upgrade

due November 9th

Ubuntu 20.04 Stage Upgrade

Dependent on Aurora upgrade

Due January 15

Ubuntu 20.04 Prod Upgrade

Dependent on Aurora upgrade

Due January 15

Ubuntu 20.04 Edge Upgrade

Due January 15

Service

Owner

Current Ubuntu Version

In Open edX?

Ubuntu 20.04 DevStack Upgrade

due November 9th

Ubuntu 20.04 Travis Upgrade

due November 9th

Ubuntu 20.04 Stage Upgrade

Dependent on Aurora upgrade

Due January 15

Ubuntu 20.04 Prod Upgrade

Dependent on Aurora upgrade

Due January 15

Ubuntu 20.04 Edge Upgrade

Due January 15

Platform (LMS + CMS)

Arbi-BOM - @Jeremy Bowman

20.04

Yes

https://openedx.atlassian.net/browse/BOM-2068 - merged

No Travis Docker Tests

(dedicated aurora)

https://openedx.atlassian.net/browse/BOM-2068

(dedicated aurora)

https://openedx.atlassian.net/browse/BOM-2068

BLOCKED

https://openedx.atlassian.net/browse/BOM-2068

eCommerce & Workers

Engagement - @Shelby Quinn (Deactivated) / @Seth McCann

Dev - @Emma Green (Deactivated)

20.04

Yes

https://openedx.atlassian.net/browse/REV-1578

https://openedx.atlassian.net/browse/REV-1580

Travis File

https://openedx.atlassian.net/browse/REV-1578

https://openedx.atlassian.net/browse/REV-1580

https://openedx.atlassian.net/browse/REV-1576

https://openedx.atlassian.net/browse/REV-1579

https://openedx.atlassian.net/browse/REV-1576

https://openedx.atlassian.net/browse/REV-1579

 

Credentials

Aperture - @Matt Tuchfarber (Deactivated)

20.04

Yes

https://openedx.atlassian.net/browse/MICROBA-545

https://openedx.atlassian.net/browse/MICROBA-545

https://github.com/edx/gomatic-internal/blob/master/gocd/vars/tools/environment-deployment-play/prod-edx-credentials.yml

 

 

Demographics

Aperture -
T: @Albert (AJ) St. Aubin (Deactivated)
B: @Ryan O'Connell
eSRE:@Matt Tuchfarber (Deactivated)

20.04

No

Upgraded as of October 9

https://github.com/edx/demographics/pull/81

Upgraded as of October 9

https://github.com/edx/demographics/pull/81

Upgraded as of October 9

https://github.com/edx/demographics/pull/81

Upgraded as of October 9

https://github.com/edx/demographics/pull/81

N/A

Notes

SRE → ? @Bill DeRusha (Deactivated)

20.04

Yes

Upgraded as of October 21

https://openedx.atlassian.net/browse/PSRE-328

Upgraded as of October 21

Travis File

(shared aurora)

Upgraded as of October 21
https://openedx.atlassian.net/browse/PSRE-304

(dedicated mysql)

Upgraded as of October 21

Upgraded as of October 21

https://openedx.atlassian.net/browse/PSRE-328

Xqueue

Arbi-BOM - @Jeremy Bowman

20.04

Yes

https://openedx.atlassian.net/browse/ARCHBOM-1575

Travis File

https://openedx.atlassian.net/browse/ARCHBOM-1575

(shared aurora)

https://openedx.atlassian.net/browse/PSRE-304

https://openedx.atlassian.net/browse/ARCHBOM-1619

https://openedx.atlassian.net/browse/ARCHBOM-1619

 

Registrar & Workers

Programs - @Simon Chen

20.04

No

Upgraded as of Sep 29, 2020

https://openedx.atlassian.net/browse/MST-461

No Travis Docker Tests

(shared aurora)

Upgraded as of January 8

https://openedx.atlassian.net/browse/MST-387

https://openedx.atlassian.net/browse/MST-387

Upgraded as of January 11

BLOCKED

Enterprise Catalog

Enterprise - @Brandon Baker (Deactivated)

20.04

Yes

https://openedx.atlassian.net/browse/ENT-3318 - blocked until at least end November

No Travis Docker Tests

(shared aurora)

 

 

 

License Manager

Enterprise - @Brandon Baker (Deactivated)

20.04

Yes, for now

https://openedx.atlassian.net/browse/ENT-3319 - blocked until at least end November

No Travis or CircleCI

(shared aurora)

 

 

Discovery

Engage - @Michael Terry (Deactivated)

20.04

Yes

Upgraded as of Oct 8, 2020

https://openedx.atlassian.net/browse/DISCO-1674

Upgraded as of Oct 8, 2020

https://openedx.atlassian.net/browse/DISCO-1674

(shared aurora)

 

 

 

Portal Designer

Masters - @Simon Chen

20.04

No

Upgraded as of Sep 21, 2020

https://openedx.atlassian.net/browse/MST-388

No Travis or CircleCI

https://openedx.atlassian.net/browse/MST-388

Upgraded as of January 11

https://openedx.atlassian.net/browse/MST-388

Upgraded as of January 11

 

Blockstore

T&L - @Dave Ormsbee (Do Not Use) (Deactivated)

18.04

Yes

https://openedx.atlassian.net/browse/TNL-7643 https://openedx.atlassian.net/browse/OSPR-5101 - Braden (OpenCraft)

CircleCI file

https://openedx.atlassian.net/browse/TNL-7643

https://openedx.atlassian.net/browse/OSPR-5101 - Braden (OpenCraft)

(shared aurora)

https://openedx.atlassian.net/browse/TNL-7673

(shared aurora)

BLOCKED on https://openedx.atlassian.net/browse/PSRE-307

https://openedx.atlassian.net/browse/TNL-7673

 

Analytics Pipeline

Data Engineering - @Brian Beggs

20.04

Yes

https://openedx.atlassian.net/browse/DEPR-119

No Travis Docker Tests

Will be deprecated

 

Will be deprecated

Will be deprecated

Analytics API

Data Engineering - @Brian Beggs

20.04

Yes

N/A (According to @Brian Beggs , insights and analytics-api do not have a devstack.)

https://openedx.atlassian.net/browse/DENG-573

Needs work or investigation

https://openedx.atlassian.net/browse/DENG-552

Needs work or investigation

https://openedx.atlassian.net/browse/DENG-552

Needs work or investigation

https://openedx.atlassian.net/browse/DENG-552

Insights and/or Analytics Dashboard

Data Engineering - @Brian Beggs

20.04

Yes

N/A (According to @Brian Beggs , insights and analytics-api do not have a devstack.)

https://openedx.atlassian.net/browse/DENG-574

Needs work or investigation

https://openedx.atlassian.net/browse/DENG-553

Needs work or investigation

https://openedx.atlassian.net/browse/DENG-553

Needs work or investigation

https://openedx.atlassian.net/browse/DENG-553

Notifier
ensure retirement

@Simon Chen , @Julie Davis - repo owner

@Jeremy Bowman - deprecation

N/A

No

Done 11/13

https://openedx.atlassian.net/browse/DEPR-106?searchSessionId=4511027c-eb42-4c57-b290-aaec5aaf03fa&searchObjectId=186639&searchContainerId=17023&searchContentType=issue

No Travis Docker or CircleCI Tests

https://openedx.atlassian.net/browse/DEPR-106?searchSessionId=4511027c-eb42-4c57-b290-aaec5aaf03fa&searchObjectId=186639&searchContainerId=17023&searchContentType=issue

https://openedx.atlassian.net/browse/DEPR-106?searchSessionId=4511027c-eb42-4c57-b290-aaec5aaf03fa&searchObjectId=186639&searchContainerId=17023&searchContentType=issue

 

Taxonomy

ensure retirement

@Albemarle (Deactivated)

20.04

No

Due January 15 - https://openedx.atlassian.net/browse/WS-1413

Due January 15 - https://openedx.atlassian.net/browse/WS-1413

Not on Stage

Will be retired before December 15 - https://openedx.atlassian.net/browse/WS-1413

Not on Prod

Will be retired before December 15 - https://openedx.atlassian.net/browse/WS-1413

 

CS Comments Service

@Mat Carter

20.04

Yes

https://openedx.atlassian.net/browse/EDUCATOR-5388

https://openedx.atlassian.net/browse/EDUCATOR-5388

https://openedx.atlassian.net/browse/EDUCATOR-5389

https://openedx.atlassian.net/browse/EDUCATOR-5389

https://openedx.atlassian.net/browse/EDUCATOR-5389

https://openedx.atlassian.net/browse/DOS-1619 - needs to be done by 11/9

 

  • 16.04 becomes end of life in April, 2021.

  • Open edX Juniper will ship based on 16.04, but Open edX Koa will ship based on 20.04.

  • Application Owners should plan to upgrade to 20.04 as part of the Koa release (targeted for December 9, 2020, upgrades must be in place by Nov 9).

Out of Scope