On-call Playbooks

On-call Playbooks

These are playbooks for on-call tasks. Feel free to add more!

Contents

Note on common requests

Most requests will come from folks who are following the instructions on page: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3241640370/Axim+Collaborative+Engineering+Team#Axim-Request-Process . Familiarize yourself with those instructions so that you understand what is expected of the requester. Update the that page if it seems like any part of it is incorrect or causing confusion.

Note on what is and isn’t on-call work

If you get a request that seems like it’s stretching the bounds of what “on-call work” is, ask yourself:

Could this be done without Axim’s special admin rights?

If the answer is “yes”, then there’s no reason Axim exclusively has to do the work. You could:

  1. remove the github-request label;

  2. move the issue to the most relevant repository (or public-engineering if there’s no other good repo);

  3. and then, based on whether or not it’s a Axim-maintained repository, either:

    • Keep it in on Axim Engineering project board. Let the requester know that it’s in our backlog, but we might not get to it for a while. They’re welcome to work on it in the meantime.

    • Remove it from the Axim Engineering board. Let the requester know that Axim won’t be working on and that they should coordinate with the repository maintainer.

Playbooks

Adding and Removing Contributors from the CLA Database

The authoritative data source for whom is covered under an individual CLA, non technical CLA, or an entity CLA is Salesforce. That data is exported once a day to GitHub where it is accessed by the CLA check bot. This section covers adding new people to an existing Entity CLA (known as an “entity contributor”), as well as adding people covered by an Individual CLA (known as an “individual contributor”).

Adding/Removing A New Entity Contributor

If an org has an Entity CLA with us, they should already have access to a Google Sheet that is integrated with Salesforce where they can add/remove their employees from the entity CLA database. If they open a GitHub issue to add users they should be redirected to their relevant spreadsheets.

The list of spreadsheets can be found here.

The list of all entities that have entity agreements can be found here.

If an entity doesn’t have a sheet there or you don’t have access to that folder. Work with Ed Z to resolve that.

Currently, you should only be adding entity contributors from 2U into Salesforce - and only if the requester on the Axim Engineering on-call ticket has the CLA Approver role in Salesforce.

For new contributors from other entities, they follow a different process - an approver from their company emails oscm@axim.org, and one of the members of that team processes the request into Salesforce. If you have a request from a non-2U individual to add them to their company’s entity CLA, please direct them to their company’s lead or HR department to have a designated company representative send the request to the OSCM list.

The most common change is to either add or remove a contributor from a companies entity CLA.

To do this, first log on to Salesforce.

In the top search bar, search for the appropriate company, in Salesforce terminology this will be an “Account.”

Select the result from the Accounts section of the meta-search.

From the Account page select the Related tab.

If you are adding a new entity contributor, click on New in the Contact section.

Enter the following details:

  1. Set the Account Name to the relevant institution (2U/edx, OpenCraft, etc.)

  2. Enter the contributor's First Name and Last Name

  3. Enter the contributor's GitHub Username (case sensitive - it should match what you see on their profile at https://github.com/<username>)

  4. Enter their Email address

  5. Locate the Contributor Covered Under Entity and check it

  6. Locate the Role field and add the Entity Contributor role to the user.

Save the record.

If you are removing an entity contributor

Find the contact record that associates the contributor with the account, say, 2U. You can do this be navigating from the account or by search for the contact by GitHub username.

Do not delete the contact, instead make the following changes.

  1. Click on the pencil by the Account Name field on the contact record and click X to remove the association. A household account will automatically be related to the contact on save.

  2. Deselect the Contributor Covered Under Entity checkbox.

  3. Save the record.

  4. Add a link to the GitHub issue in the Chatter panel on the right-hand side and any notes that are relevant for the future.

If you encounter Contact Owner related errors, contact @Edward Zarecor

Adding and removing individual contributors

Most of the process is automated, but there is associated review – and part of it is currently malfunctioning. Adding/removing individual contributors requires contract review and is currently handled by legal and Eden or Ed.

Adding and removing non technical contributors

Select the “Non-technical contributor” box in the Salesforce contact, and save the record.

Handling requests from unknown users

Sometimes, requests come from members of the openedx, or other folks we recognize.

Other times, they come from people we don’t recognize. We need to be careful about these in order to avoid our support system becoming a social engineer attack vector.

When a request comes in from someone you’re not sure about:

  • Look them up in Salesforce. Notice which roles they have. Notable roles include.

    • Core Contributor - a member of the CC program

    • Entity CLA Coverage Approver - someone who we trust to confirm new members of an entity. In the case of 2U, this means someone that we trust to provide names of new hires, such as a manager or trusted IT personnel.

    • Known managers and leaders - If the requestor is a known people manager, staff engineer, or system administrator at 2U, we can accept the request from them. If they are not marked as an Entity CLA Coverage Approver, add this role to their account in Salesforce.

      • If you’re not sure if they are “known” ask the team or check with Jeremy Ristau.

  • If they don’t have an entry, ask about them.

    • Do they claim to be from 2U? Ping Jeremy Ristau and ask if they’re a 2U employee.

    • For anyone else, just ask in #axim-engineering.

  • If their identity can be confirmed, add the requester to Salesforce, as described above. Give them the Entity CLA Coverage Approver role if appropriate.

  • If not, reject the request.

Onboarding 2U Employees

In the past, we used to handle 2U’s onboarding specially. 2U management would open GitHub request tickets, and on-call would them add them to teams and add them to the CLA database by hand.

As of August 2025, we will handle 2U onboardings like we would onboardings for any other firm with an entity CLA. That is, 2U should update the spreadsheets described above in the “ Adding and Removing Contributors from the CLA Database” section, which should result in Ed getting a notification and Salesforce and the openedx-triage team being updated. If a 2U employee opens an onboarding ticket, it’s best to close it and refer them Jeremy Risteau and link them to the CLA spreadsheet for 2U via Slack DM.

Similarly, we now handle 2U write access using the Core Contributor program, just like we do for all other contributrors. Do not grant 2U employees write access, even if they ask for it. Instead, refer them to the documentation for becoming a Core Contributor.

Onboarding Core Contributors

Follow this document:

Onboarding for Triage Access

  • First, confirm the provenance of the request. If you do not recognize the requester, you can reference the Handling requests from unknown users section. You do not need to verify the requester is an Entity CLA Coverage Approver, as triage access is unrelated to CLA status. If you cannot verify the identity of the requester, then reject the request.

  • Invite the requested user to the @openedx/openedx-triage team

  • Inform the requested user that their invite has been sent to their GitHub email.

Offboarding 2U Employees

Remove them from the openedx organization. GitHub will ask if you want to retain any of their access rights; the answer should always be “No”.

Disassociate the user from the 2U entity CLA as described above.

Offboarding Core Contributors

Note that Core Contributors have their own offboarding runbook to follow:

… that runbook prompts them to email cc-program-admins@axim.org to announce their resignation, which should prompt the CC program admins to follow this runbook:

…and that runbook should prompt the CC admins to create a GitHub request, which is probably why you are reading this.

If you’re receiving a GitHub request directly from a CC and not from a CC program admin (i.e., Feanil or Sarina), then please pause and first ask the CC that they have read and followed steps in https://openedx.atlassian.net/wiki/spaces/COMM/pages/3358261582 and has emailed cc-program-admins@axim.org to announce their resignation. If you don’t get a response, then reach out to the CC program admins themselves and ask about the status of the offboarding.

Offboarding tasks for the Axim on-call engineer:

  1. On the https://openedx.atlassian.net/wiki/spaces/COMM/pages/3156344833 wiki page, remove their row from the “current” table and add it to the “past”.

  2. Remove elevated access:

    1. For all: Remove them from the CC mailing list https://groups.google.com/a/axim.org/g/core-contributors/members?pli=1

    2. For coding core contributors, remove them from the dev list: Google Groups

    3. For coding CCs, remove access to GitHub, but leave in the Triage group.

    4. For translators or anything related to Marketing, double check with Eden that things are set

    5. For forum moderators, remove their moderator access but leave their account alone. Moderator badges should not be revoked.

    6. For other roles - no more are currently defined, but when we do get new roles, if this isn’t updated figure out who the CC has been working with and have that person terminate any access.

  3. Remove the “Core Contributor” role from their SalesForce record. If relevant, disassociate them from an entity CLA as described above. Ensure the Individual CLA box remains checked.

  4. Contact @Fox Piacenti by direct message on Slack with the full name and email address of the departing CC and Fox will remove said CC from the bi-weekly check-ins on http://Listaflow.com .

Managing teams and access

Anyone is allowed to join the openedx-triage group if they ask. This group gives access to manage issues and pull requests, but not to merge code nor to give an authoritative (green check) review.

For other requests, follow rules in https://openedx.atlassian.net/wiki/spaces/COMM/pages/3555852316 and grant requests that seem reasonable.

FC Contractor teams and access

For FC projects, contractors may be granted write access at the discretion of Axim.

Add contractor(s) to a new group with a descriptive name prefaced by axim-* , for example, axim-fc-65-implementation

Oftentimes, contractors are working on new repos. Create a new repo as detailed below. Ensure the Axim lead(s) of the FC project are on the axim-* team created above, and use that team as (temporary) maintainers. Alternatively, list the Axim lead(s) directly.

Creating New Repos in the openedx org

In most cases if the purpose of the repo is related to the Open edX Project, you should create the repo. If it turns out that the repo should not be a part of the openedx org, it is a lot easier to move the repo out than it is to move it in. Use your best judgement and ask in Slack or at standup if you’re unsure.

Who is maintaining the repo?

Establish who the maintainer will be before creating the repo. The requester’s first contribution should be to set the maintainer in catalog-info.yml. Ask them to join the #wg-maintenance Slack channel and give an FYI that a new repository exists and that they are the maintainer of it.

Granting Access Rights

When creating a new repo, you may need to establish some initial access. Don’t add users directly--give teams access to the repo instead.

The following will always be needed:

  • cla-checker - write access

  • openedx-triage - triage access

  • wg-security - read access (this is managed automatically – nothing for you to do)

Depending on the language and purpose of your repo you may also need to add various additional bot users. You can search for all relevant bot teams by searching “bot-” but an example of Python bots is:

  • bot-release-pypi - write access

  • bot-requirements-python - write access

If a repo is meant to only hold issues (that is: only a README, catalog-info, gitignore, and GH workflows; NO code, docs, or assets), then click the gear next to the repo’s description and apply the issues-only topic. Then, to ensure that all Coding CCs can flexibly work with issues on the repo, grant:

Adding Required Checks

When making a new repo, add required checks (such as commitlint) via https://github.com/openedx/repo-tools/tree/master/edx_repo_tools/repo_checks#usage

Transferring repositories into openedx

Requesters should have submitted a GitHub Request to us based on the guidance here: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3241640370/Axim+Collaborative+Engineering+Team#Transferring-repos-into-openedx . Read that guidance if you haven’t already. Then, follow these steps to handle the request, documenting any decisions you make along the way (& their rationale) in the request issue.

  1. First, determine whether the repository should be moved.

    • See our interim guidance on which repositories belong in the openedx GitHub organization. Feel free to bring the question to the team if you’re not sure.

  2. If you believe the repository should move, ensure that we legally can move the code.

  3. Choose a method: Transfer or Fork. We default to Transfer.

    1. Transfer. “Transfer” is a feature that GitHub provides. Its main benefit is that links to the repository’s original location will indefinitely forward to the new location, unless the repo is forked back into the original org. Furthermore, transferring the repository implies that Axim is receiving the entire repository as a code contribution under the contributor’s existing CLA, which has legal ramifications. To execute the transfer, follow these steps:

      1. Ask the requester to confirm the transfer using the wording in https://openedx.atlassian.net/wiki/spaces/NPGA/pages/3544940606 . Note the requester must be a principal (or higher) engineer or a manager, or have one of those people comment their approval on the request.

      2. Ask the requester to add you to the repository with admin permissions. If you are not in the source organization (you’re probably not), they can still add you to the repository an outside collaborator. They should not need to grant you any permissions in the source organization as a whole. One you’re granted repository admin permissions, go to the repository’s Settings page, scroll all the way down to the Danger Zone, hit “Transfer”, and follow the prompts.

      3. If the requester wasn’t comfortable granting you admin rights to the repo for some reason, then we can do it the hard way:

        1. Get on a screenshare with the requester (they must have admin rights on the repo being transferred). Particularly, they must share their screen looking at the GitHub page of the repo.

        2. Grant them Owner rights (from https://github.com/orgs/openedx/people , find their username. Click, then use the dropdown on the right sidebar to switch from Member to Owner)

        3. Observe them press the transfer button

        4. Determine which teams need access to the repo and add those teams in the next screen

        5. Press Transfer. After a minute or two, verify that the transfer is complete

        6. Demote them to Member (from https://github.com/orgs/openedx/people , find their username. Click, then use the dropdown on the right sidebar to switch from Owner to Member)

    2. Fork. The benefit here is that it can be done without the involvement of anyone in the source organization. Legally, we claim fewer rights using this method; as such, we are able to use it in situations where transferring wouldn’t be appropriate. Steps:

      1. First, check with the team and/or legal. We don’t use this method much, so if you’re doing it, you should probably check that these steps are still valid.

      2. Fork the repository into the openedx organization.

      3. Update any links in the repository to point to the new openedx location.

      4. If it’s not there already, add a Apache 2 or AGPLv3 license file to the repo.

      5. Add a note to the README stating that any contributions up to <version> are licensed by <original owner> under <original license>, and that any after <version> are licensed by Axim under <AGPL or Apache>. Example: https://github.com/openedx/django-require/#license

  4. Tag the latest commit.

    1. Before new changes are merged to the repo, tag the commit that was the last commit in the old org. The tag should follow a naming convention like this:

      1. last-commit-in-<orgname>-org # Some examples would be last-commit-in-edx-org last-commit-in-opencraft-org

         

  5. Ensure that the newly transferred/forked repo has a catalog-info.yaml file an a clearly named owner.

  6. Run the repo_checks on the newly transferred repo(s)

Methods of transferring

  • Transfer. Based on Transferring a repository - GitHub Docs

    1. Ask the requester to confirm the transfer using the wording in https://openedx.atlassian.net/wiki/spaces/NPGA/pages/3544940606 . Note the requester must be a principal eng or a manager, or have one of those people comment their approval on the request.

    2. Someone must have Owner rights to both the source org (or simply the source repo) and the dest org in order to do the transfer. In order to do this transfer, follow the following steps to grant the requester (or someone with admin rights to the repo) temporary Owner rights on the openedx GitHub organization:

    3. The main benefit here is that links to the repository’s original location will indefinitely forward to the new location, unless the repo is forked back into the original org.

  • Fork. You only need write access to the destination organization in order to do this.

User access (for transfers into openedx organization)

By precedent, if an organization contributes a repository into the openedx organization, we will nominate the original developers to become Core Contributors with admin or write access to the transferred repository. This is not a hard-and-fast rule, but it is often warranted, since it means that the developers have demonstrated their willingness and ability to contribute quality code to the project.

We are no longer giving edX/2U blanket write access for new repositories. See the public memo we released concerning this. There may be scenarios in which we do want to give them blanket write access (e.g., something’s coming in from the edx org); talk to the team in any of these scenarios and document the decision in the GitHub request.

Finally, you will need to grant access to a few teams and/or bots, as outlined in the “Creating New Repos in the openedx org” section.

Renaming a repository

  • Verify the maintainers of the repository agree with the rename request

  • If the repo is long-established with a lot of cross-repo references, there may be more people that need to be notified/contacted

    • //todo: figure out what that process looks like

Archiving a repository

  • Close any open PRs or Issues

  • Follow checklist in OEP-14

Adding codecov to a repository

See https://openedx.atlassian.net/wiki/spaces/COMM/pages/3438280709

Adding DNS for a new CNAME record

See https://github.com/openedx/terraform-internal/pull/11

Removing old DNS records

See https://github.com/openedx/terraform-internal/pull/12

Helping with the community release

The Build-Test-Release Working Group’s Release Manager will reach out to on-call from time to time with release-related requests. This is necessary because the Release Manager is not an openedx GitHub organization owner, yet they still need to create tags in, push branches to, and merge PRs to various openedx repositories. We can smooth this out by temporarily granting the Release Manager permission and/or merging PRs on their behalf.

The release process, and the ways on-call will need to help, are documented here: https://openedx.atlassian.net/wiki/spaces/COMM/pages/19662426/Process+to+Create+an+Open+edX+Release#Grant-the-release-manager-permissions

Inviting users to have write access in Confluence

To help fight spam we’ve restricted access to self-create accounts to some email domains, which can be found here: https://admin.atlassian.com/o/3989578f-a3e1-472d-a535-be32ae4fdd81/user-access-settings

Requests to add users should come through the support form to on-call. The user(s) can be invited here, presuming they don’t seem to be spammers: https://admin.atlassian.com/o/3989578f-a3e1-472d-a535-be32ae4fdd81/users?status=ACTIVE

New users can be added to the confluence-users group unless other access is requested.

Debugging CLA Check Failures

  1. Check to see if the github username is in the salesforce-export.csv file.

    1. Yes → Re-run the check by updating the PR Title.

    2. No → Go to https://tcril.my.salesforce.com

      1. Search for the github username and see if a contact exists.

        1. Yes → Check to see if the the Open edX Indinidual CLA or Contributor Covered Under Entity checkboxes are checked.

          Note: You should not check the boxes yourself unless you are absolutely certain that there is a contract associated with the individual or entity in salesforce.

          1. One or both boxes are checked → The CLA has been processed but the data hasn’t been exported to github yet. It should be within the next 24hrs. You can provide this information publicly.

          2. Neither box is checked → The CLA has not yet been processed by our legal team but should be within the next few days. You can provide this information publicly.

        2. No → Is there a backlog of unmerged PRs to the salesforce-export.csv?

          1. Yes → Investigate, and potentially re-run the tests.

          2. No → There may be an issue in Zapier, escalate to Ed or Feanil

Causing the CLA report to generate a new PR off schedule

Sometimes there’s a data issue with the file that is used for our CLA check and you don’t want to wait for the next afternoon run for the fix to get merged into check file in GitHub. If that’s the case, you can run the report from Salesforce manually and push it to Zapier via an email hook.

  1. Log in to Salesforce and navigate to the reports tab.

  2. In the left-hand menu, select “All Reports.”

  3. Find and open “Contributor Report.”

  4. Click the caret on the right-hand side and select “Export.”

  5. Select “Details Only,” then .csv, and the Unicode (UTF-8) encoding.

  6. Click “Export”

  7. Email the report to z70p9d2z@robot.zapier.com as an attachement.

  8. Assuming that all is well the PR should be opened here shortly and auto-merged.

Adding a new Github App to the openedx Org

  1. Collect the following things:

    1. App terms of service.

    2. Annual App cost if any.

      1. Who would pay for it? Us? A community member?

    3. Is the app something that more than one community member could benefit from, even if only one is interested right now? e.g., 2U wants an alerting system for a repo, nobody else cares but if someone DID care, they could participate in the app?

    4. Links to App entry in the GitHub Market Place

    5. Which repos will the app be enabled on?

      1. All repos, specific repos, only public repos, etc.

    6. What permissions does the app seek for the org and repos it has access to?

      1. This may be difficult to figure out without trying to install the app on a test organization. The info is not clearly stated on the app marketplace page for most apps.

  2. Send this info to vendors at Axim org to do a vendor review.

    1. Legal Approves → Install the app and enable it on the relevant repositories.

Enabling a slack channel to get posts from Discourse

  1. Add the channel to the discourse Plugin

    1. In Slack, go to the Discourse App

      1.  

    2. Click on the name “Discourse Chat Integration” to get to the settings modal.

      1.  

    3. Use the “Add this app to a channel” button to add the app to a slack channel.

  2. As a discourse admin, go to: https://discuss.openedx.org/admin/plugins/chat-integration/slack

    1. Add any rules for the relevant channel

      1. Discourse’s docs for this: Discourse Chat Integration

Managing access to Axim Cloudflare

@Edward Zarecor handles this, just forward the request to him.

Managing access to Salesforce for Axim Staff

You may not have the bit required to do this. Both Google Workspace and Salesforce admin rights are required. If you have a request, but not the access, please let @Edward Zarecor know.

  1. In Salesforce click the gear icon in the top right of the page and select setup.

  2. From the right-hand menu select Administration > Users > Users

  3. Click the “new user” link.

  4. Add the users first name, last name, and canonical email, [first initial][last name]@axim.org, other required fields will be updated automatically

  5. For the Role, User License, and Profile fields select: Axim Internal User, Salesforce, Standard User, or possibly System Administrator upon request.

  6. Set user users “federation id” to be their canonical email address as above.

  7. Save the record.

  8. In Google Workspaces admin, https://admin.google.com, add the users to the “Salesforce Users Group.”

Managing AWS, Cloudflare, and Terraform

We manage our AWS account via Terraform at https://github.com/openedx/terraform-internal/ . At the moment, both AWS and Cloudflare access are required to plan & run that Terraform.

Some of us at Axim have AWS & Cloudlare access, others don’t. If you’re on-call and need to run Terraform, reach out to the team chat, and either we will get you set up to run Terrarform, or run it for you. @Feanil Patel is the most knowledgeable here; if unsure, ping him.

How to give yourself AWS access:

  • Make a terraform-internal PR. Here’s a template.

  • Have your manager give a thumbs-up on the PR to confirm that the access is warranted.

Add a Repository’s Docs to http://docs.openedx.org

We want to publish the documentation for all repos under the docs.openedx.org domain. We do this using readthedocs.org. Each repo has its on documentation project but they are all made subrojects of the docs.openedx.org project. There is a openedx-rtd-maintainer user that has acess to the openedx org and can setup publishing via RTD.

  1. Go to readthedocs.org and login as the openedx-rtd-maintaner

  2. Go to the Dashboard

  3. Click Import a Project.

     

  4. Click openedxto filter to just openedx repositories.