On-call Playbooks
These are playbooks for on-call tasks. Feel free to add more!
General on-call instructions are here: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3344957577
Contents
- 1 Contents
- 2 Note on common requests
- 3 Note on what is and isn’t on-call work
- 4 Playbooks
- 4.1 Adding and Removing Contributors from the CLA Database
- 4.2 Handling requests from unknown users
- 4.3 Onboarding 2U Employees
- 4.4 Onboarding Core Contributors
- 4.5 Onboarding for Triage Access
- 4.6 Offboarding 2U Employees
- 4.7 Offboarding Core Contributors
- 4.8 Managing teams and access
- 4.9 FC Contractor teams and access
- 4.10 Creating New Repos in the openedx org
- 4.11 Transferring repositories into openedx
- 4.12 Renaming a repository
- 4.13 Archiving a repository
- 4.14 Adding codecov to a repository
- 4.15 Adding DNS for a new CNAME record
- 4.16 Removing old DNS records
- 4.17 Helping with the community release
- 4.18 Inviting users to have write access in Confluence
- 4.19 Debugging CLA Check Failures
- 4.20 Causing the CLA report to generate a new PR off schedule
- 4.21 Adding a new Github App to the openedx Org
- 4.22 Enabling a slack channel to get posts from Discourse
- 4.23 Managing access to Axim Cloudflare
- 4.24 Managing access to Salesforce for Axim Staff
- 4.25 Managing AWS, Cloudflare, and Terraform
- 4.26 Add a Repository’s Docs to http://docs.openedx.org
- 4.27 Make Private Wiki Content Public
- 4.28 Adding Organizations to the Verifiable Credentials Registry
- 4.29 Share Secrets with 2U SRE
- 4.30 Adding required GitHub Actions CI checks to a repository
- 4.31 Audit Github Users
- 4.32 Enable renovate on a repo
- 4.33 Adding Secrets to the Openedx Github Org
- 4.34 Managing the On-Call Slack integration
- 4.35 Handling Unresponsive PR Authors
- 4.36 Using Mr Meeting to create a meeting
- 4.37 Renaming a WG
Note on common requests
Most requests will come from folks who are following the instructions on page: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3241640370/Axim+Collaborative+Engineering+Team#Axim-Request-Process . Familiarize yourself with those instructions so that you understand what is expected of the requester. Update the that page if it seems like any part of it is incorrect or causing confusion.
Note on what is and isn’t on-call work
If you get a request that seems like it’s stretching the bounds of what “on-call work” is, ask yourself:
Could this be done without Axim’s special admin rights?
If the answer is “yes”, then there’s no reason Axim exclusively has to do the work. You could:
remove the
github-requestlabel;move the issue to the most relevant repository (or
public-engineeringif there’s no other good repo);and then, based on whether or not it’s a Axim-maintained repository, either:
Keep it in on Axim Engineering project board. Let the requester know that it’s in our backlog, but we might not get to it for a while. They’re welcome to work on it in the meantime.
Remove it from the Axim Engineering board. Let the requester know that Axim won’t be working on and that they should coordinate with the repository maintainer.
Playbooks
Adding and Removing Contributors from the CLA Database
The authoritative data source for whom is covered under an individual CLA, non technical CLA, or an entity CLA is Salesforce. That data is exported once a day to GitHub where it is accessed by the CLA check bot. This section covers adding new people to an existing Entity CLA (known as an “entity contributor”), as well as adding people covered by an Individual CLA (known as an “individual contributor”).
Adding/Removing A New Entity Contributor
If an org has an Entity CLA with us, they should already have access to a Google Sheet that is integrated with Salesforce where they can add/remove their employees from the entity CLA database. If they open a GitHub issue to add users they should be redirected to their relevant spreadsheets.
The list of spreadsheets can be found here.
The list of all entities that have entity agreements can be found here.
If an entity doesn’t have a sheet there or you don’t have access to that folder. Work with Ed Z to resolve that.
Adding and removing individual contributors
Most of the process is automated, but there is associated review – and part of it is currently malfunctioning. Adding/removing individual contributors requires contract review and is currently handled by legal and Eden or Ed.
Adding and removing non technical contributors
Select the “Non-technical contributor” box in the Salesforce contact, and save the record.
Handling requests from unknown users
Sometimes, requests come from members of the openedx, or other folks we recognize.
Other times, they come from people we don’t recognize. We need to be careful about these in order to avoid our support system becoming a social engineer attack vector.
When a request comes in from someone you’re not sure about:
Look them up in Salesforce. Notice which roles they have. Notable roles include.
Core Contributor - a member of the CC program
Entity CLA Coverage Approver - someone who we trust to confirm new members of an entity. In the case of 2U, this means someone that we trust to provide names of new hires, such as a manager or trusted IT personnel.
Known managers and leaders - If the requestor is a known people manager, staff engineer, or system administrator at 2U, we can accept the request from them. If they are not marked as an Entity CLA Coverage Approver, add this role to their account in Salesforce.
If you’re not sure if they are “known” ask the team or check with Jeremy Ristau.
If they don’t have an entry, ask about them.
Do they claim to be from 2U? Ping Jeremy Ristau and ask if they’re a 2U employee.
For anyone else, just ask in
#axim-engineering.
If their identity can be confirmed, add the requester to Salesforce, as described above. Give them the Entity CLA Coverage Approver role if appropriate.
If not, reject the request.
Onboarding 2U Employees
In the past, we used to handle 2U’s onboarding specially. 2U management would open GitHub request tickets, and on-call would them add them to teams and add them to the CLA database by hand.
As of August 2025, we will handle 2U onboardings like we would onboardings for any other firm with an entity CLA. That is, 2U should update the spreadsheets described above in the “ Adding and Removing Contributors from the CLA Database” section, which should result in Ed getting a notification and Salesforce and the openedx-triage team being updated. If a 2U employee opens an onboarding ticket, it’s best to close it and refer them Jeremy Risteau and link them to the CLA spreadsheet for 2U via Slack DM.
Similarly, we now handle 2U write access using the Core Contributor program, just like we do for all other contributrors. Do not grant 2U employees write access, even if they ask for it. Instead, refer them to the documentation for becoming a Core Contributor.
Onboarding Core Contributors
Follow this document:
Onboarding for Triage Access
First, confirm the provenance of the request. If you do not recognize the requester, you can reference the Handling requests from unknown users section. You do not need to verify the requester is an Entity CLA Coverage Approver, as triage access is unrelated to CLA status. If you cannot verify the identity of the requester, then reject the request.
Invite the requested user to the
@openedx/openedx-triageteamInform the requested user that their invite has been sent to their GitHub email.
Offboarding 2U Employees
Remove them from the openedx organization. GitHub will ask if you want to retain any of their access rights; the answer should always be “No”.
Disassociate the user from the 2U entity CLA as described above.
Offboarding Core Contributors
Note that Core Contributors have their own offboarding runbook to follow:
… that runbook prompts them to email cc-program-admins@axim.org to announce their resignation, which should prompt the CC program admins to follow this runbook:
…and that runbook should prompt the CC admins to create a GitHub request, which is probably why you are reading this.
If you’re receiving a GitHub request directly from a CC and not from a CC program admin (i.e., Feanil or Sarina), then please pause and first ask the CC that they have read and followed steps in https://openedx.atlassian.net/wiki/spaces/COMM/pages/3358261582 and has emailed cc-program-admins@axim.org to announce their resignation. If you don’t get a response, then reach out to the CC program admins themselves and ask about the status of the offboarding.
Offboarding tasks for the Axim on-call engineer:
On the https://openedx.atlassian.net/wiki/spaces/COMM/pages/3156344833 wiki page, remove their row from the “current” table and add it to the “past”.
Remove elevated access:
For all: Remove them from the CC mailing list https://groups.google.com/a/axim.org/g/core-contributors/members?pli=1
For coding core contributors, remove them from the dev list:
Google Groups For coding CCs, remove access to GitHub, but leave in the Triage group.
For translators or anything related to Marketing, double check with Eden that things are set
For forum moderators, remove their moderator access but leave their account alone. Moderator badges should not be revoked.
For other roles - no more are currently defined, but when we do get new roles, if this isn’t updated figure out who the CC has been working with and have that person terminate any access.
Remove the “Core Contributor” role from their SalesForce record. If relevant, disassociate them from an entity CLA as described above. Ensure the Individual CLA box remains checked.
Contact @Fox Piacenti by direct message on Slack with the full name and email address of the departing CC and Fox will remove said CC from the bi-weekly check-ins on http://Listaflow.com .
Managing teams and access
Anyone is allowed to join the openedx-triage group if they ask. This group gives access to manage issues and pull requests, but not to merge code nor to give an authoritative (green check) review.
For other requests, follow rules in https://openedx.atlassian.net/wiki/spaces/COMM/pages/3555852316 and grant requests that seem reasonable.
FC Contractor teams and access
For FC projects, contractors may be granted write access at the discretion of Axim.
Add contractor(s) to a new group with a descriptive name prefaced by axim-* , for example, axim-fc-65-implementation
Oftentimes, contractors are working on new repos. Create a new repo as detailed below. Ensure the Axim lead(s) of the FC project are on the axim-* team created above, and use that team as (temporary) maintainers. Alternatively, list the Axim lead(s) directly.
Creating New Repos in the openedx org
In most cases if the purpose of the repo is related to the Open edX Project, you should create the repo. If it turns out that the repo should not be a part of the openedx org, it is a lot easier to move the repo out than it is to move it in. Use your best judgement and ask in Slack or at standup if you’re unsure.
Who is maintaining the repo?
Establish who the maintainer will be before creating the repo. The requester’s first contribution should be to set the maintainer in catalog-info.yml. Ask them to join the #wg-maintenance Slack channel and give an FYI that a new repository exists and that they are the maintainer of it.
Granting Access Rights
When creating a new repo, you may need to establish some initial access. Don’t add users directly--give teams access to the repo instead.
The following will always be needed:
cla-checker- write accessopenedx-triage- triage accesswg-security- read access (this is managed automatically – nothing for you to do)
Depending on the language and purpose of your repo you may also need to add various additional bot users. You can search for all relevant bot teams by searching “bot-” but an example of Python bots is:
bot-release-pypi- write accessbot-requirements-python- write access
If a repo is meant to only hold issues (that is: only a README, catalog-info, gitignore, and GH workflows; NO code, docs, or assets), then click the gear next to the repo’s description and apply the issues-only topic. Then, to ensure that all Coding CCs can flexibly work with issues on the repo, grant:
committers(GitHub group) - write access
Adding Required Checks
When making a new repo, add required checks (such as commitlint) via https://github.com/openedx/repo-tools/tree/master/edx_repo_tools/repo_checks#usage
Transferring repositories into openedx
Requesters should have submitted a GitHub Request to us based on the guidance here: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3241640370/Axim+Collaborative+Engineering+Team#Transferring-repos-into-openedx . Read that guidance if you haven’t already. Then, follow these steps to handle the request, documenting any decisions you make along the way (& their rationale) in the request issue.
First, determine whether the repository should be moved.
See our interim guidance on which repositories belong in the openedx GitHub organization. Feel free to bring the question to the team if you’re not sure.
If you believe the repository should move, ensure that we legally can move the code.
See the Managing GH Repositories (Axim-internal) page. You are encouraged to reach out to our legal counsel if you have any shred of uncertainty here
Choose a method: Transfer or Fork. We default to Transfer.
Transfer. “Transfer” is a feature that GitHub provides. Its main benefit is that links to the repository’s original location will indefinitely forward to the new location, unless the repo is forked back into the original org. Furthermore, transferring the repository implies that Axim is receiving the entire repository as a code contribution under the contributor’s existing CLA, which has legal ramifications. To execute the transfer, follow these steps:
Ask the requester to confirm the transfer using the wording in https://openedx.atlassian.net/wiki/spaces/NPGA/pages/3544940606 . Note the requester must be a principal (or higher) engineer or a manager, or have one of those people comment their approval on the request.
Ask the requester to add you to the repository with admin permissions. If you are not in the source organization (you’re probably not), they can still add you to the repository an outside collaborator. They should not need to grant you any permissions in the source organization as a whole. One you’re granted repository admin permissions, go to the repository’s Settings page, scroll all the way down to the Danger Zone, hit “Transfer”, and follow the prompts.
If the requester wasn’t comfortable granting you admin rights to the repo for some reason, then we can do it the hard way:
Get on a screenshare with the requester (they must have admin rights on the repo being transferred). Particularly, they must share their screen looking at the GitHub page of the repo.
Grant them Owner rights (from https://github.com/orgs/openedx/people , find their username. Click, then use the dropdown on the right sidebar to switch from Member to Owner)
Observe them press the transfer button
Determine which teams need access to the repo and add those teams in the next screen
Press Transfer. After a minute or two, verify that the transfer is complete
Demote them to Member (from https://github.com/orgs/openedx/people , find their username. Click, then use the dropdown on the right sidebar to switch from Owner to Member)
Fork. The benefit here is that it can be done without the involvement of anyone in the source organization. Legally, we claim fewer rights using this method; as such, we are able to use it in situations where transferring wouldn’t be appropriate. Steps:
First, check with the team and/or legal. We don’t use this method much, so if you’re doing it, you should probably check that these steps are still valid.
Fork the repository into the openedx organization.
Update any links in the repository to point to the new openedx location.
If it’s not there already, add a Apache 2 or AGPLv3 license file to the repo.
Add a note to the README stating that any contributions up to <version> are licensed by <original owner> under <original license>, and that any after <version> are licensed by Axim under <AGPL or Apache>. Example: https://github.com/openedx/django-require/#license
Tag the latest commit.
Before new changes are merged to the repo, tag the commit that was the last commit in the old org. The tag should follow a naming convention like this:
last-commit-in-<orgname>-org # Some examples would be last-commit-in-edx-org last-commit-in-opencraft-org
Ensure that the newly transferred/forked repo has a catalog-info.yaml file an a clearly named owner.
Run the repo_checks on the newly transferred repo(s)
Methods of transferring
Transfer. Based on
Transferring a repository - GitHub DocsAsk the requester to confirm the transfer using the wording in https://openedx.atlassian.net/wiki/spaces/NPGA/pages/3544940606 . Note the requester must be a principal eng or a manager, or have one of those people comment their approval on the request.
Someone must have Owner rights to both the source org (or simply the source repo) and the dest org in order to do the transfer. In order to do this transfer, follow the following steps to grant the requester (or someone with admin rights to the repo) temporary Owner rights on the openedx GitHub organization:
The main benefit here is that links to the repository’s original location will indefinitely forward to the new location, unless the repo is forked back into the original org.
Fork. You only need write access to the destination organization in order to do this.
User access (for transfers into openedx organization)
By precedent, if an organization contributes a repository into the openedx organization, we will nominate the original developers to become Core Contributors with admin or write access to the transferred repository. This is not a hard-and-fast rule, but it is often warranted, since it means that the developers have demonstrated their willingness and ability to contribute quality code to the project.
We are no longer giving edX/2U blanket write access for new repositories. See the public memo we released concerning this. There may be scenarios in which we do want to give them blanket write access (e.g., something’s coming in from the edx org); talk to the team in any of these scenarios and document the decision in the GitHub request.
Finally, you will need to grant access to a few teams and/or bots, as outlined in the “Creating New Repos in the openedx org” section.
Renaming a repository
Verify the maintainers of the repository agree with the rename request
If the repo is long-established with a lot of cross-repo references, there may be more people that need to be notified/contacted
//todo: figure out what that process looks like
Archiving a repository
Close any open PRs or Issues
Follow checklist in OEP-14
Adding codecov to a repository
See https://openedx.atlassian.net/wiki/spaces/COMM/pages/3438280709
Adding DNS for a new CNAME record
See https://github.com/openedx/terraform-internal/pull/11
Removing old DNS records
See https://github.com/openedx/terraform-internal/pull/12
Helping with the community release
The Build-Test-Release Working Group’s Release Manager will reach out to on-call from time to time with release-related requests. This is necessary because the Release Manager is not an openedx GitHub organization owner, yet they still need to create tags in, push branches to, and merge PRs to various openedx repositories. We can smooth this out by temporarily granting the Release Manager permission and/or merging PRs on their behalf.
The release process, and the ways on-call will need to help, are documented here: https://openedx.atlassian.net/wiki/spaces/COMM/pages/19662426/Process+to+Create+an+Open+edX+Release#Grant-the-release-manager-permissions
Inviting users to have write access in Confluence
To help fight spam we’ve restricted access to self-create accounts to some email domains, which can be found here: https://admin.atlassian.com/o/3989578f-a3e1-472d-a535-be32ae4fdd81/user-access-settings
Requests to add users should come through the support form to on-call. The user(s) can be invited here, presuming they don’t seem to be spammers: https://admin.atlassian.com/o/3989578f-a3e1-472d-a535-be32ae4fdd81/users?status=ACTIVE
New users can be added to the confluence-users group unless other access is requested.
Debugging CLA Check Failures
Check to see if the github username is in the salesforce-export.csv file.
Yes → Re-run the check by updating the PR Title.
No → Go to https://tcril.my.salesforce.com
Search for the github username and see if a contact exists.
Yes → Check to see if the the
Open edX Indinidual CLAorContributor Covered Under Entitycheckboxes are checked.Note: You should not check the boxes yourself unless you are absolutely certain that there is a contract associated with the individual or entity in salesforce.
One or both boxes are checked → The CLA has been processed but the data hasn’t been exported to github yet. It should be within the next 24hrs. You can provide this information publicly.
Neither box is checked → The CLA has not yet been processed by our legal team but should be within the next few days. You can provide this information publicly.
No → Is there a backlog of unmerged PRs to the salesforce-export.csv?
Yes → Investigate, and potentially re-run the tests.
No → There may be an issue in Zapier, escalate to Ed or Feanil
Causing the CLA report to generate a new PR off schedule
Sometimes there’s a data issue with the file that is used for our CLA check and you don’t want to wait for the next afternoon run for the fix to get merged into check file in GitHub. If that’s the case, you can run the report from Salesforce manually and push it to Zapier via an email hook.
Log in to Salesforce and navigate to the reports tab.
In the left-hand menu, select “All Reports.”
Find and open “Contributor Report.”
Click the caret on the right-hand side and select “Export.”
Select “Details Only,” then .csv, and the Unicode (UTF-8) encoding.
Click “Export”
Email the report to z70p9d2z@robot.zapier.com as an attachement.
Assuming that all is well the PR should be opened here shortly and auto-merged.
Adding a new Github App to the openedx Org
Collect the following things:
App terms of service.
Annual App cost if any.
Who would pay for it? Us? A community member?
Is the app something that more than one community member could benefit from, even if only one is interested right now? e.g., 2U wants an alerting system for a repo, nobody else cares but if someone DID care, they could participate in the app?
Links to App entry in the GitHub Market Place
Which repos will the app be enabled on?
All repos, specific repos, only public repos, etc.
What permissions does the app seek for the org and repos it has access to?
This may be difficult to figure out without trying to install the app on a test organization. The info is not clearly stated on the app marketplace page for most apps.
Send this info to
vendors at Axim orgto do a vendor review.Legal Approves → Install the app and enable it on the relevant repositories.
Enabling a slack channel to get posts from Discourse
Add the channel to the discourse Plugin
In Slack, go to the Discourse App
Click on the name “Discourse Chat Integration” to get to the settings modal.
Use the “Add this app to a channel” button to add the app to a slack channel.
As a discourse admin, go to: https://discuss.openedx.org/admin/plugins/chat-integration/slack
Add any rules for the relevant channel
Discourse’s docs for this:
Discourse Chat Integration
Managing access to Axim Cloudflare
@Edward Zarecor handles this, just forward the request to him.
Managing access to Salesforce for Axim Staff
You may not have the bit required to do this. Both Google Workspace and Salesforce admin rights are required. If you have a request, but not the access, please let @Edward Zarecor know.
In Salesforce click the gear icon in the top right of the page and select setup.
From the right-hand menu select Administration > Users > Users
Click the “new user” link.
Add the users first name, last name, and canonical email, [first initial][last name]@axim.org, other required fields will be updated automatically
For the Role, User License, and Profile fields select: Axim Internal User, Salesforce, Standard User, or possibly System Administrator upon request.
Set user users “federation id” to be their canonical email address as above.
Save the record.
In Google Workspaces admin, https://admin.google.com, add the users to the “Salesforce Users Group.”
Managing AWS, Cloudflare, and Terraform
We manage our AWS account via Terraform at https://github.com/openedx/terraform-internal/ . At the moment, both AWS and Cloudflare access are required to plan & run that Terraform.
Some of us at Axim have AWS & Cloudlare access, others don’t. If you’re on-call and need to run Terraform, reach out to the team chat, and either we will get you set up to run Terrarform, or run it for you. @Feanil Patel is the most knowledgeable here; if unsure, ping him.
How to give yourself AWS access:
Make a terraform-internal PR. Here’s a template.
Have your manager give a thumbs-up on the PR to confirm that the access is warranted.
Add a Repository’s Docs to http://docs.openedx.org
We want to publish the documentation for all repos under the docs.openedx.org domain. We do this using readthedocs.org. Each repo has its on documentation project but they are all made subrojects of the docs.openedx.org project. There is a openedx-rtd-maintainer user that has acess to the openedx org and can setup publishing via RTD.
Go to readthedocs.org and login as the
openedx-rtd-maintanerGo to the Dashboard
Click Import a Project.
Click
openedxto filter to justopenedxrepositories.