Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 183 Next »

Overview

This document explains how to make an Open edX release. In this document, I'll use "zebrawood" for the release you are making, and "yucca" for the previous release.

Repos

An Open edX release is comprised of specific versions of many repos.  The participating repos are determined by examining the OEP-2 openedx.yaml files in each repo. In other words, every repository that has an openedx.yaml file in its root may or may not participate in the release, depending on the contents of the file. In particular, this line needs to be present:

openedx-release: {ref: master}

Tags and Branches

The release will have a branch in each participating repo.  This is the branch where releases are made.  This branch is named "open-release/RELEASENAME.master", for example, "open-release/zebrawood.master". Specific versions will be tagged on that branch, like "open-release/zebrawood.1".  The first version will be ".1", not ".0".

tag_release.py

To make the process practically feasible, a tag_release.py script exists in https://github.com/edx/repo-tools . In short, on each run it will discover which repositories are eligible for a release by listing all repos in the configured organizations (the default is edx, edx-ops, and edx-solutions) and examining their respective openedx.yaml files. If a given repository is indeed eligible, the script will execute the requested branching or tagging operation on it.

Permissions

To be able to run tag_release.py successfully, the github account in use must have write access to all the repositories that are eligible for release. For community release managers, this can be done by adding them to the community release managers team in the edX GitHub org.

Alert other teams

Other teams will want to know that you are about to make a new release branch.

T&L team/edx-platform owners

edx-platform Jenkins jobs that need to be updated for new release branches:

Have a GitHub org admin protect the new branch from accidental deletion and require the new Jenkins jobs to pass on PRs against it.

Making the release branches

A week before the release…

Run the tag_release script; if it outputs “*** edx/somerepo has openedx-release 'maybe', skipped” lines, then you should get in touch with the repo owners and ask them if they should be included in the release. If yes, then the maybe: true line should be removed from the openedx.yaml file in the repo.
(see for instance this pull request).


Pull the translation one last time

Carlos Muniz (Unlicensed) Ned Batchelder (Deactivated) Tagging you in case you would want to review, or give a direction about how strings sync can be manaully trigged

This is best done just before the branch is cut the latest the better.

- Trigger the job which pulls the translation and creates PRs (i.e. PR for fronted-app-learning: https://github.com/openedx/frontend-app-learning/pull/973 , PR for edx-platform: https://github.com/openedx/edx-platform/pull/31039 )
- If the PRs are not merged on a time, its okay (which is epxected hence the above note), leave a not about them so that we can backport the commit the after.

Why need to this: The point of doing this is that we ensure the shortest lag possible between the last time a translator review/translate string and the branch cut date. i.e. If the syncing happens on a weekly basis and we do the branch cut just a day before the scheduled weekly sync , then we would in theory lose 6 days of translation work.

Testing before creating release branches

If you want to try a dry-run of the release before making any release branches, make a branch under your personal name. Don't use the full release name in the branch name, or people will think it's a release.  Follow the instructions for "Create the release branches", but use "yourname/test-z.1" instead of "open-release/zebrawood.master", for example:

$ tag_release --doit --skip-repo "*-internal" --branch yourname/test-z.1

Create the release branches

The repo-tools repo has a script, tag_release.py, to branch and tag the proper repos.  The repos are indicated by an "openedx-release" entry in their openedx.yaml file.  Releases before Ficus would mark dependent repos (those installed by other repos, such as XBlock being installed by edx-platform).  We no longer branch or tag dependent repos.

$ tag_release --doit --skip-repo "*-internal" --branch open-release/zebrawood.master
edx/configuration: master (branch) d213681
  2016-02-17 Fred Smith: Merge pull request #2784 from edx/add-support-url
edx/edx-analytics-data-api-client: 0.6.1 (tag) d621288
  2015-05-07 Dennis Jen: Merge pull request #20 from edx/dsjen/update-setup
edx/edx-certificates: master (branch) a2ed4ed
  2015-08-28 Kelketek: Merge pull request #49 from edx/make_tmp_dir
edx/edx-platform: release (branch) 71b3080
  2016-02-17 Nimisha Asthagiri: Merge pull request #11532 from edx/rc/2016-02-16
.. and other repo information ..
Is this correct? [y/N] y
Success!

The tag_release.py script has other options to control it in various ways.

The branch will be created based on the current state of the release branches described in each repo's openedx.yaml.  In particular, for edx-platform, this will be a daily release.  Make note of which release it was, people will want to know this.

Double-check what the script did to see that it looks correct.

Get release notes started

Tell your doc writer the date the release was based on.  Also find the date the previous release was based on.  They will use these two dates to comb the edx.org release notes for information to go into the Open edX release notes.

Create a wiki page for the next release

The Open edX Release Planning page has a sub-page for each release.  Make a new wiki page for the next release (the one six months from now).

Announce the creation of the Open edX release branch

Send an email to eng-all@edx.org explaining that the open-release branch has been created from an edx.org release.  Here's an example:

Hey everyone, 

TL;DR: We are in the Lilac pre-release phase. Keep Lilac in mind as you make changes. They will need to be cherry-picked to ship as part of Lilac. Koa is still the currently supported release and also might need fixes. Lilac will become the supported release in June.

What happened: On Friday, we created the open-release/lilac.master branches in our repos.  As of now, your day-to-day work is contributing to the release after Lilac, which will be called Maple. The community will now be stabilizing and testing Lilac, and we'll be making fixes on the Lilac branches, leading up to the official Lilac release in early June.

What is Lilac?  The next Open edX community release is named Lilac.  It will be based on the code as it was last Friday.  We can include your upcoming changes if you alert us to them.

What you should do: As you do your work, please be alert for fixes that might need to be applied to Lilac. I can do the work of getting the fix into the release, but I'll need your help to know there is a fix to be applied.

There is a new wiki page for you to note things that will need to be considered when it comes time to release Maple: Maple  .  If you are uncertain whether something should be mentioned there, add it. Or, talk to me and we'll figure it out.

Have questions? If you want to know more, we have an FAQ about the Open edX community releases.  Please reach out if you have any questions.

Thanks for all your hard work!  You make Open edX software the valued open source platform it is!

Announce upcoming release to the community

Create a topic in the Announcements - Releases category on discuss.openedx.org. ((Add some boilerplate here))

If you created a pre-release branch, you should probably not write a post in “Releases” but in the build/test/release working group category. (see for instance this announcement)

Also send out an email to eng-all@edx.org to let everyone at edX know about the upcoming Open edX release, as well. This should remind people to update the wiki page for the release (for example this page for the Eucalyptus release). They should keep track of bugfixes that are in flight, so that when they get merged into master, they can also be pulled into the release candidate branch.

Please notify the DEPR working group in the #depr-slash-n-burn channel in the Open edX Slack, so that the DEPR working group can help write release notes of deprecated code in accordance with the DEPR Release Notes Process .

Make release-specific changes on the release branch

Some changes need to be made directly on the release branch, because they are intended for the release itself, and should not be on master.

Release line

In edx-platform/openedx/core/release.py, edit the RELEASE_LINE line to indicate the release:

RELEASE_LINE = "zebrawood"

Translations

We upload release-specific versions of the translated strings to Transifex. This is done by editing the edx-platform/.tx/config file on the release branch. Add two clauses like this:

[o:open-edx:p:open-edx-releases:r:release-zebrawood]
file_filter = conf/locale/<lang>/LC_MESSAGES/django.po
source_file = conf/locale/en/LC_MESSAGES/django.po
source_lang = en
type = PO

[o:open-edx:p:open-edx-releases:r:release-zebrawood-js]
file_filter = conf/locale/<lang>/LC_MESSAGES/djangojs.po
source_file = conf/locale/en/LC_MESSAGES/djangojs.po
source_lang = en
type = PO

Make sure that the file has no other "release-" resources.

⚠ The exact syntax of this file is undetermined as of 2022-06-09. Please see this PR for more information.

Inform edx-platform developers

We should update the pull request template in the master branch of edx/edx-platform to encourage developers to contribute bugfixes back to the release branch (see for instance this PR).

Documentation

There are a few places to update documentation. Some happen when the release branches are made, others when the release is finished.

When the release branches are made

Once the release is official

Make a release or release candidate

Releases and release candidates are tags marking specific versions of all the repos.  Take care with the steps to create and tag these.  It's surprising, but you have to edit files to refer to the tags before you can create the tag.

1. Double-check that all known issues have been addressed

Prior to making a release (major, minor, or even a candidate), it is imperative that all known issues be fixed. You can check at the GitHub Project board with an appropriate filter such as “label:bug milestone:lilac.1”. Over time other labels or milestones may be incorporated, but just keep in mind that issues such as an uninstallable tag are unacceptable.

Search for open pull requests against the release branch. For example, for Nutmeg, we would search: https://github.com/pulls?q=is%3Aopen+is%3Apr+base%3Aopen-release%2Fnutmeg.master+archived%3Afalse.

2. Make a test branch (not a tag!)

To test the installation, but not yet mark the release, we'll make a test branch.  The branch should use your personal prefix ("you/"), and be named for the release you are working on, but with an additional .n suffix.  Don't use the full release name, or people will find the branch and think it is a release:

Don’t create a tag for testing, because once it is pulled by other developers or processes, it becomes very hard to delete - specially on a busy repository such as edx-platform.

tag_release --doit --branch --skip-invalid --search-branch open-release/zebrawood.master --override-ref open-release/zebrawood.master you/test/z.1rc1.1

The .1 suffix is so that you can make a .2 if something needs to be fixed.

(TODO: update this with the saved repo list feature, instead of relying on --skip-invalid)

3. Update and upload translations on the test branch

For this method, you should be using tutor and should have openedx/edx-platform on the target release branch locally.

Tutor

In the case that this is a new release, it is best to use Tutor’s Nightly branch, otherwise the Master branch is recommended. Using tutor, open a shell in cms while mounting the local edx-platform repo set to the target release branch. 

tutor dev run cms bash -m path-to-edx-platform

Setup

There are some setup items that are necessary as of nutmeg [last updated 2022-09-01]:

  1. Install the development requirements

  2. Uninstall the old transifex-client (v2)

  3. Install the new transifex-client (v3)

pip install -r requirements/edx/development.txt
pip uninstall transifex-client -y
curl -o- https://raw.githubusercontent.com/transifex/cli/master/install.sh | bash

Transifex Credentials

Add your transifex credentials to ~/.transifexrc. Set up your API Token and password via the Transifex Documentation. They have the form:

https://www.transifex.com]
api_hostname = https://api.transifex.com
hostname = https://www.transifex.com
password = {PASSWORD}
rest_hostname = https://rest.api.transifex.com
token = {TOKEN}
username = api

Generate, Compile, and Upload to Transifex

Open a new bash shell within the current shell and follow these commands to generate, compile and upload Translation files. Replace {release-version} in line 5 with the name of the release. It should match the files in .tx/config.

bash
i18n_tool extract
i18n_tool dummy
i18n_tool generate
paver i18n_compilejs
tx push -s -r open-edx-releases.release-{release-version} open-edx-releases.release-{release-version}-js

Manually Renaming the Translations Files

In the case that this is the first time the resources are being uploaded to Transifex, log in to Transifex, find the files named django.po and djangojs.po, confirm that the slugs (check out the url) match the release version and rename the files to match the naming convention.

djangojs.po -> release-{release-version}-js
django.po -> release-{release-version}-js

4. Run through the native installation (until Lilac)

Do not skip this step. This is no longer necessary now that the native installation is unsupported

No matter what CI says, and even if you have reports of the same exact git hash having been successfully tested the day before, test-install the branch before tagging it. At the time of writing, this means going through all the steps defined in Koa Native Open edX platform Ubuntu 20.04 64 bit Installation . This will avoid anything from last-minute requirements installation problems, to full-blown Ubuntu package removals.

5. Test the release

Using the native installation just produced, run through the Release Test Process to make sure there are no glaring problems.

6. Create RC or release tags

Execute this step if, and only if, there were no problems up to it. It is not practical to overwrite a tag on busy git repositories. If there are any errors, a new tag will have to be created.

For each tagged repo, make tags for the release or release candidate. The tag name is different for the two:

  • Release candidate: the tag should be open-release/$RELEASE.XrcY, where X is the number of the release, and Y is the number of the RC.

  • Release: the tag should be should be open-release/$RELEASE.X, where X is the number of the release.

tag_release --doit --tag --skip-invalid --search-branch open-release/zebrawood.master --override-ref you/test/z.1rc1.1 open-release/zebrawood.1rc1		# for release candidate
# or:
tag_release --doit --tag --skip-invalid --search-branch open-release/zebrawood.master --override-ref you/test/z.1.1 open-release/zebrawood.1				# for release

You can delete the test branches you made once the release is marked:

tag_release --doit --branch --skip-invalid --reverse  you/test/z.1rc1.1

7. Update release directory page

Update the release page in https://github.com/edx/edx-developer-docs.

8. Announce to the community

Create a topic in the Announcements - Releases category on discuss.openedx.org.

Change the topic line in the #ops channel in Slack.

Things that used to be done, but are no longer:

  • For Birch, we did both an announcement email and a post to the engineering blog. 

  • You should also create a Confluence page summarizing the status of the release candidate branch. Here's the one we used for Birch.

9. (For release candidates) Collect bugfixes, create new RCs

Over the next two weeks or so, keep an eye out for bugfixes going into master, so that important bugfixes also go into the release master branch. If there are a significant number of changes (however you want to define "significant"), you should create another release candidate. That consists of updating the Confluence page for the release candidate branch, tagging the repos, possibly creating and uploading new Vagrant boxes, testing the upgrades, and sending out another email to the community informing them of the new release candidate. If there's less than a week before the planned release date, and you're putting out a new release candidate, you should push back the planned release date so that everyone has more time to test on the new release candidate. (Be sure to announce to the community and to everyone at edX that the planned release date is being pushed back.)

The final version

(TODO: what has to happen differently for the final version?)

Run the translation commands to upload the latest text to Transifex:

$ paver i18n_release_push

Check that release changes made it to master

During the stabilization of the release, you may have changes made directly on the open-release/zebrawood.master branch.  Take a look at what's there, and compare to master to be sure all the changes end up on master also.  Keep in mind that there are a few changes on the release branch that should only be on the release branch (RELEASE_LINE and translation naming, for example).

# gittreeif is a handy function from https://github.com/nedbat/dot/blob/master/.bashrc
GIT_PAGER=cat gittreeif origin/open-release/hawthorn.master log origin/open-release/zebrawood.master ^origin/master

Follow-on releases

The first released version on the open-release/zebrawood.master (for example) release branch is called open-release/zebrawood.1 .   Subsequent releases use incrementing numbers: open-release/zebrawood.2, open-release/zebrawood.3, and so on.

The procedure for creating them is exactly the same as the Make a release or release candidate described above, except that you’d name test branches as you/test/z.2rc1.1, and tags would look like open-release/zebrawood.2.

OTHER STEPS TO ADD SOMEPLACE ABOVE

  • Devstack docker images have to be built (image builders?) Run on edx internal Jenkins.

  • No labels