Contents

Note on common requests

Most requests will come from folks who are following the instructions on page: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3241640370/tCRIL+Engineering+Team#tCRIL-Request-Process . Familiarize yourself with those instructions so that you understand what is expected of the requester. Update the that page if it seems like any part of it is incorrect or causing confusion.

Note on what is and isn’t on-call work

If you get a request that seems like it’s stretching the bounds of what “on-call work” is, ask yourself:

Could this be done without Axim’s special admin rights?

If the answer is “yes”, then there’s no reason Axim exclusively has to do the work. You could:

1. remove the github-request label;

2. move the issue to the most relevant repository (or public-engineering if there’s no other good repo);

3. and then, based on whether or not it’s a Axim-maintained repository, either:

   • Keep it in on Axim Engineering project board. Let the requester know that it’s in our backlog, but we might not get to it for a while. They’re welcome to work on it in the meantime.

   • Remove it from the Axim Engineering board. Let the requester know that Axim won’t be working on and that they should coordinate with the repository maintainer.

Playbooks

Adding and Removing Contributors from the CLA Database

The authoritative data source for whom is covered under an individual CLA, non technical CLA, or an entity CLA is Salesforce. That data is exported once a day to GitHub where it is accessed by the CLA check bot. This section covers adding new people to an existing Entity CLA (known as an “entity contributor”), as well as adding people covered by an Individual CLA (known as an “individual contributor”).

Adding/Removing A New Entity Contributor

The most common change is to either add or remove a contributor from a companies entity CLA.

To do this, first log on to Salesforce.

In the top search bar, search for the appropriate company, in Salesforce terminology this will be an “Account.”

Open

Select the result from the Accounts section of the meta-search.

From the Account page select the Related tab.

If you are adding a new entity contributor, click on New in the Contact section.

Enter the following details:

1. Set the Account Name to the relevant Institution(2U/edx, OpenCraft, etc.)

2. The contributor's first and last name

3. The contributor's GitHub username (case sensitive - it should match what you see on their profile at https://github.com/<username>)

4. Ideally, their email address

5. Locate the field Contributor Covered Under Entity and select it

6. Locate the Role field and add the Entity Contributor role to the user.

Save the record.

If you are removing an entity contributor

Find the contact record that associates the contributor with the account, say, 2U. You can do this be navigating from the account or by search for the contact by GitHub username.

Do not delete the contact, instead make the following changes.

1. Click on the pencil by the Account Name field on the contact record and click X to remove the association. A household account will automatically be related to the contact on save.

2. Deselect the Contributor Covered Under Entity checkbox.

3. Save the record.

4. Add a link to the GitHub issue in the Chatter panel on the right-hand side and any notes that are relevant for the future.

Adding and removing individual contributors

Most of the process is automated, but there is associated review – and part of it is currently malfunctioning. Adding/removing individual contributors requires contract review and is currently handled by legal and Eden or Ed.

Adding and removing non technical contributors

Select the “Non-technical contributor” box in the Salesforce contact, and save the record.

Open

Handling requests from unknown users

Sometimes, requests come from members of the openedx, or other folks we recognize.

Other times, they come from people we don’t recognize. We need to be careful about these in order to avoid our support system becoming a social engineer attack vector.

When a request comes in from someone you’re not sure about:

• Look them up in Salesforce. Notice which roles they have. Notable roles include.

  • Core Contributor - a member of the CC program

  • Entity CLA Coverage Approver - someone who we trust to confirm new members of an entity. In the case of 2U, this means someone that we trust to provide names of new hires, such as a manager or trusted IT personnel.

  • Known managers and leaders - If the requestor is a known people manager or staff engineer at 2U, we can accept the request from them. If they are not marked as an Entity CLA Coverage Approver, add this role to their account in Salesforce.

    • If you’re not sure if they are “known” ask the team or check with @georgebabey

• If they don’t have an entry, ask about them.

  • Do they claim to be from 2U? Ping @georgebabey and ask if they’re a 2U employee.

  • For anyone else, just ask in #axim-engineering.

• If their identity can be confirmed, add the requester to Salesforce, as described above. Give them the Entity CLA Coverage Approver role if appropriate.

• If not, reject the request.

Onboarding 2U Employees

• First, confirm the provenance of the request, as described in Handling requests from unknown users. If you can’t, then reject the request.

• Add the user to the openedx GitHub organization. Inform the user that their invite has been sent to their GitHub email.

• As part of the invitation, add the user to the groups:

  • @openedx/openedx-triage

  • @openedx/2u-edx-legacy (was called @openedx/push-pull-all up until ~July 2023)

  • Other sensible groups that the user requsets. For guidance see: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3771793429 (that’s a public memo, feel free to share). Ask the team if in doubt.

• Add the user to the CLA database.

• Close the issue once the invitation is sent and the user has been added to the CLA database.

Onboarding Core Contributors

Right now, we follow this document:

You can copy-paste these runbooks directly into a GitHub ticket and get the lil checkboxes.

You should have access to the Core Contributor course on the training site. Please add the new CC to the course.

The DocuSign step is covered in the course, but if there’s questions you can ask one of the Program Administrators (cc-program-admins@axim.org) for assistance.

You can check course completion status on the course student admin page. Enter the email address used to enroll in the “View a specific learner's grades and progress” textbox.

Offboarding 2U Employees

Remove them from the openedx organization. GitHub will ask if you want to retain any of their access rights; the answer should always be “No”.

Disassociate the user from the 2U entity CLA as described above.

Offboarding Core Contributors

Note that Core Contributors have their own offboarding runbook to follow:

That runbook prompts them to make a Axim request to have their own access revoked. That’s probably what brought you (the on-call engineer) here.

Remove them from the openedx organization. GitHub will ask if you want to retain any of their access rights; the answer should always be “No” unless there are some special circumstances that we’ve discussed as a team and documented.

Remove the Core Contributor role from their record in Salesforce. If relevant, disassociate them from an entity CLA as described above.

Managing teams and access

See

Grant requests that follow those rules and seem reasonable.

Creating New Repos in the openedx org

In most cases if the purpose of the repo is related to the Open edX Project, you should create the repo. If it turns out that the repo should not be a part of the openedx org, it is a lot easier to move the repo out than it is to move it in. Use your best judgement and ask in Slack or at standup if you’re unsure.

Establish who the maintainer will be before creating the repo. The requester’s first contribution should be to set the maintainer in catalog-info.yml. Ask them to join the #wg-maintenance Slack channel and give an FYI that a new repository exists and that they are the maintainer of it.

When creating a new repo, you may need to establish some initial access. Don’t add users directly--give teams access to the repo instead.

The following will always be needed:

• cla-checker - write access

• openedx-triage - triage access

• wg-security - read access (this is managed automatically – nothing for you to do)

Depending on the language and purpose of your repo you may also need to add various additional bot users. You can search for all relevant bot teams by searching “bot-” but an example of Python bots is:

• bot-release-pypi - write access

• bot-requirements-python - write access

Finally, if a repo is meant to only hold issues (that is: only a README, catalog-info, gitignore, and GH workflows; NO code, docs, or assets), then click the gear next to the repo’s description and apply the issues-only topic. Then, to ensure that all Coding CCs can flexibly work with issues on the repo, grant:

• core-contributor-program-committers - write access

Transferring repositories into openedx

Requesters should have submitted a GitHub Request to us based on the guidance here: . Read that guidance if you haven’t already. Then, follow these steps to handle the request, documenting any decisions you make along the way (& their rationale) in the request issue.

1. First, determine whether the repository should be moved.

   • See our interim guidance on which repositories belong in the openedx GitHub organization. Feel free to bring the question to the team if you’re not sure.

2. If you believe the repository should move, ensure that we legally can move the code.

   • See the Managing GH Repositories (Axim-internal) page. You are encouraged to reach out to our legal counsel if you have any shred of uncertainty here

3. Choose a method: Transfer or Fork. We default to Transfer.

   1. Transfer. “Transfer” is a feature that GitHub provides. Its main benefit is that links to the repository’s original location will indefinitely forward to the new location, unless the repo is forked back into the original org. Furthermore, transferring the repository implies that Axim is receiving the entire repository as a code contribution under the contributor’s existing CLA, which has legal ramifications. To execute the transfer, follow these steps:

      1. Ask the requester to confirm the transfer using the wording in . Note the requester must be a principal (or higher) engineer or a manager, or have one of those people comment their approval on the request.

      2. Ask the requester to add you to the repository with admin permissions. If you are not in the source organization (you’re probably not), they can still add you to the repository an outside collaborator. They should not need to grant you any permissions in the source organization as a whole. One you’re granted repository admin permissions, go to the repository’s Settings page, scroll all the way down to the Danger Zone, hit “Transfer”, and follow the prompts.

      3. If the requester wasn’t comfortable granting you admin rights to the repo for some reason, then we can do it the hard way:

         1. Get on a screenshare with the requester (they must have admin rights on the repo being transferred). Particularly, they must share their screen looking at the GitHub page of the repo.

         2. Grant them Owner rights (from , find their username. Click, then use the dropdown on the right sidebar to switch from Member to Owner)

         3. Observe them press the transfer button

         4. Determine which teams need access to the repo and add those teams in the next screen

         5. Press Transfer. After a minute or two, verify that the transfer is complete

         6. Demote them to Member (from , find their username. Click, then use the dropdown on the right sidebar to switch from Owner to Member)

   2. Fork. The benefit here is that it can be done without the involvement of anyone in the source organization. Legally, we claim fewer rights using this method; as such, we are able to use it in situations where transferring wouldn’t be appropriate. Steps:

      1. First, check with the team and/or legal. We don’t use this method much, so if you’re doing it, you should probably check that these steps are still valid.

      2. Fork the repository into the openedx organization.

      3. Update any links in the repository to point to the new openedx location.

      4. If it’s not there already, add a Apache 2 or AGPLv3 license file to the repo.

      5. Add a note to the README stating that any contributions up to <version> are licensed by <original owner> under <original license>, and that any after <version> are licensed by Axim under <AGPL or Apache>. Example:

4. Tag the latest commit.

   1. Before new changes are merged to the repo, tag the commit that was the last commit in the old org. The tag should follow a naming convention like this:

      1. last-commit-in-<orgname>-org # Some examples would be last-commit-in-edx-org last-commit-in-opencraft-org

Methods of transferring

• Transfer. Based on

  1. Ask the requester to confirm the transfer using the wording in . Note the requester must be a principal eng or a manager, or have one of those people comment their approval on the request.

  2. Someone must have Owner rights to both the source org (or simply the source repo) and the dest org in order to do the transfer. In order to do this transfer, follow the following steps to grant the requester (or someone with admin rights to the repo) temporary Owner rights on the openedx GitHub organization:

  3. The main benefit here is that links to the repository’s original location will indefinitely forward to the new location, unless the repo is forked back into the original org.

• Fork. You only need write access to the destination organization in order to do this.

User access (for transfers into openedx organization)

By precedent, if an organization contributes a repository into the openedx organization, we will nominate the original developers to become Core Contributors with admin or write access to the transferred repository. This is not a hard-and-fast rule, but it is often warranted, since it means that the developers have demonstrated their willingness and ability to contribute quality code to the project.

We have not been giving edX/2U blanket write access (via the push-pull-all group) for new repositories. See the public memo we released concerning this. There may be scenarios in which we do want to give them blanket write access (e.g., something’s coming in from the edx org); talk to the team in any of these scenarios and document the decision in the GitHub request.

Finally, you will need to grant access to a few teams and/or bots, as outlined in the “Creating New Repos in the openedx org” section.

Renaming a repository

• Verify the maintainers of the repository agree with the rename request

• If the repo is long-established with a lot of cross-repo references, there may be more people that need to be notified/contacted

  • //todo: figure out what that process looks like

Archiving a repository

• First, make sure that the archival request is following the requirements of. OEP-14: Archiving GitHub Repositories.

• If it is, archive the repo:

  • repo settings

  • scroll down to the DANGER ZONE

  • click “Archive this repository”

• Then move the repo to openedx-unsupported org:

  • repo settings

  • scroll down to DANGER ZONE

  • transfer ownership

  • (You will also need write access in the openedx-unsupported org to do this. Ask someone from the team if you’re missing them.)
    
    

Adding codecov to a repository

See

Adding DNS for a new CNAME record

See

Removing old DNS records

See

Helping with the community release

The Build-Test-Release Working Group’s Release Manager will reach out to on-call from time to time with release-related requests. This is necessary because the Release Manager is not an openedx GitHub organization owner, yet they still need to create tags in, push branches to, and merge PRs to various openedx repositories. We can smooth this out by temporarily granting the Release Manager permission and/or merging PRs on their behalf.

The release process, and the ways on-call will need to help, are documented here:

Inviting users to have write access in Confluence

To help fight spam we’ve restricted access to self-create accounts to some email domains, which can be found here: https://admin.atlassian.com/o/3989578f-a3e1-472d-a535-be32ae4fdd81/user-access-settings

Requests to add users should come through the support form to on-call. The user(s) can be invited here, presuming they don’t seem to be spammers: https://admin.atlassian.com/o/3989578f-a3e1-472d-a535-be32ae4fdd81/users?status=ACTIVE

New users can be added to the confluence-users group unless other access is requested.

Debugging CLA Check Failures

1. Check to see if the github username is in the salesforce-export.csv file.

   1. Yes → Re-run the check by updating the PR Title.

   2. No → Go to https://tcril.my.salesforce.com

      1. Search for the github username and see if a contact exists.

         1. Yes → Check to see if the the Open edX Indinidual CLA or Contributor Covered Under Entity checkboxes are checked.

            Note: You should not check the boxes yourself unless you are absolutely certain that there is a contract associated with the individual or entity in salesforce.

            1. One or both boxes are checked → The CLA has been processed but the data hasn’t been exported to github yet. It should be within the next 24hrs. You can provide this information publicly.

            2. Neither box is checked → The CLA has not yet been processed by our legal team but should be within the next few days. You can provide this information publicly.

         2. No → Is there a backlog of unmerged PRs to the salesforce-export.csv?

            1. Yes → Investigate, and potentially re-run the tests.

            2. No → There may be an issue in Zapier, escalate to Ed or Feanil

Causing the CLA report to generate a new PR off schedule

Sometimes there’s a data issue with the file that is used for our CLA check and you don’t want to wait for the next afternoon run for the fix to get merged into check file in GitHub. If that’s the case, you can run the report from Salesforce manually and push it to Zapier via an email hook.

1. Log in to Salesforce and navigate to the reports tab.

2. In the left-hand menu, select “All Reports.”

3. Find and open “Contributor Report.”

4. Click the caret on the right-hand side and select “Export.”

5. Select “Details Only,” then .csv, and the Unicode (UTF-8) encoding.

6. Click “Export”

7. Email the report to z70p9d2z@robot.zapier.com as an attachement.

8. Assuming that all is well the PR should be opened here shortly and auto-merged.

Adding a new Github App to the openedx Org

1. Collect the following things:

   1. App terms of service.

   2. Annual App cost if any.

      1. Who would pay for it? Us? A community member?

   3. Is the app something that more than one community member could benefit from, even if only one is interested right now? e.g., 2U wants an alerting system for a repo, nobody else cares but if someone DID care, they could participate in the app?

   4. Links to App entry in the GitHub Market Place

   5. Which repos will the app be enabled on?

      1. All repos, specific repos, only public repos, etc.

   6. What permissions does the app seek for the org and repos it has access to?

      1. This may be difficult to figure out without trying to install the app on a test organization. The info is not clearly stated on the app marketplace page for most apps.

2. Send this info to vendors at Axim org to do a vendor review.

   1. Legal Approves → Install the app and enable it on the relevant repositories.

Enabling a slack channel to get posts from Discourse

1. Add the channel to the discourse Plugin

   1. In Slack, go to the Discourse App

      1.  

   2. Click on the name “Discourse Chat Integration” to get to the settings modal.

      1.  

   3. Use the “Add this app to a channel” button to add the app to a slack channel.

2. As a discourse admin, go to: https://discuss.openedx.org/admin/plugins/chat-integration/slack

   1. Add any rules for the relevant channel

      1. Discourse’s docs for this:

Managing access to Axim Cloudflare

@Edward Zarecor handles this, just forward the request to him.

Managing access to Salesforce for Axim Staff

You may not have the bit required to do this. Both Google Workspace and Salesforce admin rights are required. If you have a request, but not the access, please let @Edward Zarecor know.

1. In Salesforce click the gear icon in the top right of the page and select setup.

2. From the right-hand menu select Administration > Users > Users

3. Click the “new user” link.

4. Add the users first name, last name, and canonical email, [first initial][last name]@axim.org, other required fields will be updated automatically

5. For the Role, User License, and Profile fields select: Axim Internal User, Salesforce, Standard User, or possibly System Administrator upon request.

6. Set user users “federation id” to be their canonical email address as above.

7. Save the record.

8. In Google Workspaces admin, https://admin.google.com, add the users to the “Salesforce Users Group.”

Managing access to Axim AWS

We manage our AWS account via Terraform at https://github.com/openedx/terraform-internal/ . At the moment, both AWS and Cloudflare access are required to plan & run that Terraform. If you need AWS access, follow these steps (on-call engineers: no action is required from you).

How to give yourself AWS access:

• Make a terraform-internal PR. Here’s a template.

• Have your manager give a thumbs-up on the PR to confirm that the access is warranted.

• Ping @Feanil Patel to plan, run, & merge the PR.

Add a Repository’s Docs to http://docs.openedx.org

We want to publish the documentation for all repos under the docs.openedx.org domain. We do this using readthedocs.org. Each repo has its on documentation project but they are all made subrojects of the docs.openedx.org project. There is a openedx-rtd-maintainer user that has acess to the openedx org and can setup publishing via RTD.

1. Go to readthedocs.org and login as the openedx-rtd-maintaner

2. Go to the Dashboard

3. Click Import a Project.

    

4. Click openedxto filter to just openedx repositories.

    

5. Find the relevant repository and click the “Plus” button. (If you can’t find the project, you may need to add the repo to openedx-rtd-bot GitHub team with write access to the repo (needed to put status on PRs about docs PR builds)

    

6. Click through the addition wizard.

7. Go to Admin → Advancd Settings and check the Build pull requests for this project checkbox and hit Save at the bottom.

8. Go to the Admin → Integrations → GitHub incoming webhook and copy the webhook URL.

    

9. Go to the repository’s Settings → Webhooks page and click Add webhook

    

10. Paste the RTD URL from above into the Payload URL Field.

11. Under Which events would you like to trigger this webhook? Select Let me select individual events and choose the following.

    1. Branch or tag creation

    2. Branch or tag deletion

    3. Pull requests

    4. Pushes

12. Click Add Webhook

13. Go to https://readthedocs.org/dashboard/docsopenedxorg/subprojects/

14. Add the new project as a Subproject of the docs.openedx.org project.

15. Inform the requester of the URL for the repo docs under docs.openedx.org. Something like: https://docs.openedx.org/projects/xblock-utils/en/latest/

Make Private Wiki Content Public

If pages are private on the wiki, we need 2U review before we can make them or parts of them public.

1. Create a ticket on the 2U Open Source Process WG Ticket like this one.

2. Work with the 2U assigned reviewer.

   1. Get them access to the content by adding them to the space/pages.

   2. Work with them to cleanup the content.

   3. Delete the history of the pages before making the agreed upon pages public.

Adding Organizations to the Verifiable Credentials Registry

For organization that want to integrate with the LC Wallet application, they will need to be added to a registry that is managed by the Digital Credentials Consortium team. They have asked that Axim manage those requests initially rather than having organizations contact them directly. We are not sure how much interest there will be in this and will adapt depending on what happens.

For now, ping @Edward Zarecor on these if you see such a request.

Share Secrets with 2U SRE

There are ocassions where we need to share credentials with 2U SRE for some of the features in the Open edX community that they support. Some examples include, SMTP creds for github action notifications from the requirements bot and action runners to speed up edx-platform test runs.

Context: We have a folder in keeper that we have shared with 2U SRE (axim-eng-and-2u-sre-shared-secrets-for-openedx-github-org).

1. Make sure the 2U SRE person you’re working has access to the axim-eng-and-2u-sre-shared-secrets-for-openedx-github-org keeper folder.

   1. If they don’t have access

      1. Confirm their identity with the manager of 2U SRE and grant their 2U e-mail address access to the folder.

2. Check to see if a record already exists for the secret they want to share.

   1. If it does

      1. Ask the 2U SRE person to update it with the new value.

   2. If it does not

      1. Create a new record(so we can maintain its permissions)

      2. Ask the 2U SRE person to add the secret to the record.

Adding required GitHub Actions CI checks to a repository

GitHub actions workflows in any given repository do not need to pass to merge PRs by default. To set a GitHub actions workflow job as required, you should:

1. Ensure the job has been run in the past week.

   1. This is necessary because workflows that have not been run will not appear in the UI element for adding them as required.

   2. If the requested job has not been run in the past week, you can re-run the CI on any open PR (or dig through the history in the “Actions” tab to re-run the job)

2. Click on the “Settings” tab for the repo

3. Click on “Branches” within settings to see “Branch protection rules”

4. Find the branch you want to add the requirement to and click “Edit”

   1. You will most likely want to edit the default branch

5. Under “Require status checks to pass before merging” you should see a search box with placeholder text that says “Search for status checks in the last week for this repository”

   1. Use this search box to find the job you want to add as a required check.

6. Save your changes.

7. Verify the check is required by looking at any open PR (you should see “Required” next the the check).

Audit Github Users

Because we are adding users from other organizations to the openedx github org and sometimes data can get out of sync, we should regularly perform an audit of who has GitHub access and whether they continue to be in the CLA list.

To help perform the audit, we’ve written a new audit_users script in the repo tools repo.

1. In a python environment run, pip install git+ssh://git@github.com/openedx/repo-tools.git

2. Export a github token into your environment that has read:org and repo access.

   export GITHBU_TOKEN=ghp_....

    

3. Run the script: audit_users (no arguments needed). You should get an output of users that don’t have associated entries in our CLA database.

4. You’ll need to investigate why they have access and either remove them from GitHub or add/update their records in salesforce.

Enable renovate on a repo

Go to the Open edX org Renovate GitHub app installation page and add the requested repo under “Only select repositories” in “Repository access”