Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Info

These are playbooks for on-call tasks. Feel free to add more!

...

Most requests will come from folks who are following the instructions on page: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3241640370/tCRILAxim+Collaborative+Engineering+Team#tCRILTeam#Axim-Request-Process . Familiarize yourself with those instructions so that you understand what is expected of the requester. Update the that page if it seems like any part of it is incorrect or causing confusion.

...

Adding/Removing A New Entity Contributor

The most common change is to either add or remove a contributor from a companies entity CLA.

To do this, first log on to Salesforce.

In the top search bar, search for the appropriate company, in Salesforce terminology this will be an “Account.”

...

Select the result from the Accounts section of the meta-search.

From the Account page select the Related tab.

If you are adding a new entity contributor, click on New in the Contact section.

Enter the following details:

  1. Set the Account Name to the relevant Institution(2U/edx, OpenCraft, etc.)

  2. The contributor's first and last name

  3. The contributor's GitHub username (case sensitive - it should match what you see on their profile at https://github.com/<username>)

  4. Ideally, their email address

  5. Locate the field Contributor Covered Under Entity and select it

  6. Locate the Role field and add the Entity Contributor role to the user.

Save the record.

...

Find the contact record that associates the contributor with the account, say, 2U. You can do this be navigating from the account or by search for the contact by GitHub username.

Do not delete the contact, instead make the following changes.

  1. Click on the pencil by the Account Name field on the contact record and click X to remove the association. A household account will automatically be related to the contact on save.

  2. Deselect the Contributor Covered Under Entity checkbox.

  3. Save the record.

  4. Add a link to the GitHub issue in the Chatter panel on the right-hand side and any notes that are relevant for the future.

Note

If you encounter Contact Owner related errors, contact Edward Zarecor

Adding and removing individual contributors

Most of the process is automated, but there is associated review – and part of it is currently malfunctioning. Adding/removing individual contributors requires contract review and is currently handled by legal and Eden or Ed.

Adding and removing non technical contributors

Select the “Non-technical contributor” box in the Salesforce contact, and save the record.

...

👽 Handling requests from unknown users

Sometimes, requests come from members of the openedx, or other folks we recognize.

Other times, they come from people we don’t recognize. We need to be careful about these in order to avoid our support system becoming a social engineer attack vector.

When a request comes in from someone you’re not sure about:

  • Look them up in Salesforce. Notice which roles they have. Notable roles include.

    • Core Contributor - a member of the CC program

    • Entity CLA Coverage Approver - someone who we trust to confirm new members of an entity. In the case of 2U, this means someone that we trust to provide names of new hires, such as a manager or trusted IT personnel.

    • Known managers and leaders - If the requestor is a known people manager or staff engineer at 2U, we can accept the request from them. If they are not marked as an Entity CLA Coverage Approver, add this role to their account in Salesforce.

      • If you’re not sure if they are “known” ask the team or check with @georgebabey

  • If they don’t have an entry, ask about them.

    • Do they claim to be from 2U? Ping @georgebabey and ask if they’re a 2U employee.

    • For anyone else, just ask in #axim-engineering.

  • If their identity can be confirmed, add the requester to Salesforce, as described above. Give them the Entity CLA Coverage Approver role if appropriate.

  • If not, reject the request.

🎓 Onboarding 2U Employees

...

Add the user to the openedx GitHub organization. Inform the user that their invite has been sent to their GitHub email.

...

As part of the invitation, add the user to the groups:

  • @openedx/openedx-triage

  • @openedx/2u-edx-legacy (was called @openedx/push-pull-all up until ~July 2023)

    • We’re no longer adding users to the 2u-edx-legacy group.

  • Other sensible groups that the user requsets. For guidance see: Granting write access to repos in the openedx org (that’s a public memo, feel free to share). Ask the team if in doubt.

...

Add the user to the CLA database.

...

Close the issue once the invitation is sent and the user has been added to the CLA database.

🌐 Onboarding Core Contributors

Right now, we follow this document:

You can copy-paste these runbooks directly into a GitHub ticket and get the lil checkboxes.

You should have access to the Core Contributor course on the training site. Please add the new CC to the course.

The DocuSign step is covered in the course, but if there’s questions you can ask one of the Program Administrators (cc-program-admins@axim.org) for assistance.

...

Info

Currently, you should only be adding entity contributors from 2U into Salesforce - and only if the requester on the Axim Engineering on-call ticket has the CLA Approver role in Salesforce.

For new contributors from other entities, they follow a different process - an approver from their company emails oscm@axim.org, and one of the members of that team processes the request into Salesforce. If you have a request from a non-2U individual to add them to their company’s entity CLA, please direct them to their company’s lead or HR department to have a designated company representative send the request to the OSCM list.

The most common change is to either add or remove a contributor from a companies entity CLA.

To do this, first log on to Salesforce.

In the top search bar, search for the appropriate company, in Salesforce terminology this will be an “Account.”

...

Select the result from the Accounts section of the meta-search.

From the Account page select the Related tab.

If you are adding a new entity contributor, click on New in the Contact section.

Enter the following details:

  1. Set the Account Name to the relevant Institution(2U/edx, OpenCraft, etc.)

  2. The contributor's first and last name

  3. The contributor's GitHub username (case sensitive - it should match what you see on their profile at https://github.com/<username>)

  4. Ideally, their email address

  5. Locate the field Contributor Covered Under Entity and select it

  6. Locate the Role field and add the Entity Contributor role to the user.

Save the record.

Anchor
remove-entity-contributor
remove-entity-contributor
If you are removing an entity contributor

Find the contact record that associates the contributor with the account, say, 2U. You can do this be navigating from the account or by search for the contact by GitHub username.

Do not delete the contact, instead make the following changes.

  1. Click on the pencil by the Account Name field on the contact record and click X to remove the association. A household account will automatically be related to the contact on save.

  2. Deselect the Contributor Covered Under Entity checkbox.

  3. Save the record.

  4. Add a link to the GitHub issue in the Chatter panel on the right-hand side and any notes that are relevant for the future.

Note

If you encounter Contact Owner related errors, contact Edward Zarecor

Adding and removing individual contributors

Most of the process is automated, but there is associated review – and part of it is currently malfunctioning. Adding/removing individual contributors requires contract review and is currently handled by legal and Eden or Ed.

Adding and removing non technical contributors

Select the “Non-technical contributor” box in the Salesforce contact, and save the record.

...

👽 Handling requests from unknown users

Sometimes, requests come from members of the openedx, or other folks we recognize.

Other times, they come from people we don’t recognize. We need to be careful about these in order to avoid our support system becoming a social engineer attack vector.

When a request comes in from someone you’re not sure about:

  • Look them up in Salesforce. Notice which roles they have. Notable roles include.

    • Core Contributor - a member of the CC program

    • Entity CLA Coverage Approver - someone who we trust to confirm new members of an entity. In the case of 2U, this means someone that we trust to provide names of new hires, such as a manager or trusted IT personnel.

    • Known managers and leaders - If the requestor is a known people manager or staff engineer at 2U, we can accept the request from them. If they are not marked as an Entity CLA Coverage Approver, add this role to their account in Salesforce.

      • If you’re not sure if they are “known” ask the team or check with @georgebabey

  • If they don’t have an entry, ask about them.

    • Do they claim to be from 2U? Ping @georgebabey and ask if they’re a 2U employee.

    • For anyone else, just ask in #axim-engineering.

  • If their identity can be confirmed, add the requester to Salesforce, as described above. Give them the Entity CLA Coverage Approver role if appropriate.

  • If not, reject the request.

🎓 Onboarding 2U Employees

  • First, confirm the provenance of the request, as described in Handling requests from unknown users. If you can’t, then reject the request.

  • Add the user to the openedx GitHub organization. Inform the user that their invite has been sent to their GitHub email.

  • As part of the invitation, add the user to the groups:

    • @openedx/openedx-triage

    • @openedx/2u-edx-legacy (was called @openedx/push-pull-all up until ~July 2023)

      • We’re no longer adding users to the 2u-edx-legacy group.

    • Other sensible groups that the user requsets. For guidance see: Granting write access to repos in the openedx org (that’s a public memo, feel free to share). Ask the team if in doubt.

  • Add the user to the CLA database.

  • Close the issue once the invitation is sent and the user has been added to the CLA database.

🌐 Onboarding Core Contributors

Right now, we follow this document:

You can copy-paste these runbooks directly into a GitHub ticket and get the lil checkboxes.

You should have access to the Core Contributor course on the training site. Please add the new CC to the course.

The DocuSign step is covered in the course, but if there’s questions you can ask one of the Program Administrators (cc-program-admins@axim.org) for assistance.

You can check course completion status on the course student admin page. Enter the email address used to enroll in the “View a specific learner's grades and progress” textbox.

...

Grant requests that follow those rules and seem reasonable.

🌅 Creating New Repos in the openedx org

...

👩‍🔧 FC Contractor teams and access

For FC projects, contractors may be granted write access at the discretion of Axim.

Add contractor(s) to a new group with a descriptive name prefaced by axim-* , for example, axim-fc-65-implementation

Oftentimes, contractors are working on new repos. Create a new repo as detailed below. Ensure the Axim lead(s) of the FC project are on the axim-* team created above, and use that team as (temporary) maintainers. Alternatively, list the Axim lead(s) directly.

🌅 Creating New Repos in the openedx org

In most cases if the purpose of the repo is related to the Open edX Project, you should create the repo. If it turns out that the repo should not be a part of the openedx org, it is a lot easier to move the repo out than it is to move it in. Use your best judgement and ask in Slack or at standup if you’re unsure.Establish

👩‍🔧 Who is maintaining the repo?

Establish who the maintainer will be before creating the repo. The requester’s first contribution should be to set the maintainer in catalog-info.yml. Ask them to join the #wg-maintenance Slack channel and give an FYI that a new repository exists and that they are the maintainer of it.

👩‍💻 Granting Access Rights

When creating a new repo, you may need to establish some initial access. Don’t add users directly--give teams access to the repo instead.

...

  • bot-release-pypi - write access

  • bot-requirements-python - write access

Finally, if If a repo is meant to only hold issues (that is: only a README, catalog-info, gitignore, and GH workflows; NO code, docs, or assets), then click the gear next to the repo’s description and apply the issues-only topic. Then, to ensure that all Coding CCs can flexibly work with issues on the repo, grant:

  • core-contributor-program-committers - write accesscommitters (GitHub group) - write access

✅ Adding Required Checks

When making a new repo, add required checks (such as commitlint) via https://github.com/openedx/repo-tools/tree/master/edx_repo_tools/repo_checks#usage

🏗 Transferring repositories into openedx

...

  1. First, determine whether the repository should be moved.

    • See our interim guidance on which repositories belong in the openedx GitHub organization. Feel free to bring the question to the team if you’re not sure.

  2. If you believe the repository should move, ensure that we legally can move the code.

  3. Choose a method: Transfer or Fork. We default to Transfer.

    1. Transfer. “Transfer” is a feature that GitHub provides. Its main benefit is that links to the repository’s original location will indefinitely forward to the new location, unless the repo is forked back into the original org. Furthermore, transferring the repository implies that Axim is receiving the entire repository as a code contribution under the contributor’s existing CLA, which has legal ramifications. To execute the transfer, follow these steps:

      1. Ask the requester to confirm the transfer using the wording in /wiki/spaces/NPGA/pages/3544940606 . Note the requester must be a principal (or higher) engineer or a manager, or have one of those people comment their approval on the request.

      2. Ask the requester to add you to the repository with admin permissions. If you are not in the source organization (you’re probably not), they can still add you to the repository an outside collaborator. They should not need to grant you any permissions in the source organization as a whole. One you’re granted repository admin permissions, go to the repository’s Settings page, scroll all the way down to the Danger Zone, hit “Transfer”, and follow the prompts.

      3. If the requester wasn’t comfortable granting you admin rights to the repo for some reason, then we can do it the hard way:

        1. Get on a screenshare with the requester (they must have admin rights on the repo being transferred). Particularly, they must share their screen looking at the GitHub page of the repo.

        2. Grant them Owner rights (from https://github.com/orgs/openedx/people , find their username. Click, then use the dropdown on the right sidebar to switch from Member to Owner)

        3. Observe them press the transfer button

        4. Determine which teams need access to the repo and add those teams in the next screen

        5. Press Transfer. After a minute or two, verify that the transfer is complete

        6. Demote them to Member (from https://github.com/orgs/openedx/people , find their username. Click, then use the dropdown on the right sidebar to switch from Owner to Member)

    2. Fork. The benefit here is that it can be done without the involvement of anyone in the source organization. Legally, we claim fewer rights using this method; as such, we are able to use it in situations where transferring wouldn’t be appropriate. Steps:

      1. First, check with the team and/or legal. We don’t use this method much, so if you’re doing it, you should probably check that these steps are still valid.

      2. Fork the repository into the openedx organization.

      3. Update any links in the repository to point to the new openedx location.

      4. If it’s not there already, add a Apache 2 or AGPLv3 license file to the repo.

      5. Add a note to the README stating that any contributions up to <version> are licensed by <original owner> under <original license>, and that any after <version> are licensed by Axim under <AGPL or Apache>. Example: https://github.com/openedx/django-require/#license

  4. Tag the latest commit.

    1. Before new changes are merged to the repo, tag the commit that was the last commit in the old org. The tag should follow a naming convention like this:

      1. Code Block
        last-commit-in-<orgname>-org
        # Some examples would be
        last-commit-in-edx-org
        last-commit-in-opencraft-org

...

  1. In Salesforce click the gear icon in the top right of the page and select setup.

  2. From the right-hand menu select Administration > Users > Users

  3. Click the “new user” link.

  4. Add the users first name, last name, and canonical email, [first initial][last name]@axim.org, other required fields will be updated automatically

  5. For the Role, User License, and Profile fields select: Axim Internal User, Salesforce, Standard User, or possibly System Administrator upon request.

  6. Set user users “federation id” to be their canonical email address as above.

  7. Save the record.

  8. In Google Workspaces admin, https://admin.google.com, add the users to the “Salesforce Users Group.”

🎛️ Managing

...

AWS, Cloudflare, and Terraform

We manage our AWS account via Terraform at https://github.com/openedx/terraform-internal/ . At the moment, both AWS and Cloudflare access are required to plan & run that Terraform. If you need AWS access, follow these steps (on-call engineers: no action is required from you).How to give yourself AWS

Some of us at Axim have AWS & Cloudlare access, others don’t. If you’re on-call and need to run Terraform, reach out to the team chat, and either we will get you set up to run Terrarform, or run it for you. Feanil Patel is the most knowledgeable here; if unsure, ping him.

How to give yourself AWS access:

  • Make a terraform-internal PR. Here’s a template.

  • Have your manager give a thumbs-up on the PR to confirm that the access is warranted.Ping Feanil Patel to plan, run, & merge the PR.

📖 Add a Repository’s Docs to http://docs.openedx.org

...

  1. Ensure the job has been run in the past week.

    1. This is necessary because workflows that have not been run will not appear in the UI element for adding them as required.

    2. If the requested job has not been run in the past week, you can re-run the CI on any open PR (or dig through the history in the “Actions” tab to re-run the job)

  2. Click on the “Settings” tab for the repo

  3. Click on “Branches” within settings to see “Branch protection rules”

  4. Find the branch you want to add the requirement to and click “Edit”

    1. You will most likely want to edit the default branch

  5. Under “Require status checks to pass before merging” you should see a search box with placeholder text that says “Search for status checks in the last week for this repository”

    1. Use this search box to find the job you want to add as a required check.

  6. Save your changes.

  7. Verify the check is required by looking at any open PR (you should see “Required” next the the check).

🔍 Audit Github Users

Because we are adding users from other organizations to the openedx github org and sometimes data can get out of sync, we should regularly perform an audit of who has GitHub access and whether they continue to be in the CLA list.

To help perform the audit, we’ve written a new audit_users script in the repo tools repo.

...

In a python environment run, pip install git+ssh://git@github.com/openedx/repo-tools.git

Export a github token into your environment that has read:org and repo access.

Code Block
export GITHBU_TOKEN=ghp_....

...

Run the script: audit_users (no arguments needed). You should get an output of users that don’t have associated entries in our CLA database.

...

You’ll need to investigate why they have access and either remove them from GitHub or add/update their records in salesforce.

🎨 Enable renovate on a repo

...

  1. .

  2. Verify the check is required by looking at any open PR (you should see “Required” next the the check).

🔍 Audit Github Users

Because we are adding users from other organizations to the openedx github org and sometimes data can get out of sync, we should regularly perform an audit of who has GitHub access and whether they continue to be in the CLA list.

To help perform the audit, we’ve written a new audit_users script in the repo tools repo.

  1. In a python environment run, pip install git+ssh://git@github.com/openedx/repo-tools.git

  2. Export a github token (classic, not fine-grained) into your environment that has read:org and repo access.

    Code Block
    export GITHUB_TOKEN=ghp_....

  3. Run the script: audit_users (no arguments needed). You should get an output of users that don’t have associated entries in our CLA database.

  4. You’ll need to investigate why they have access and either remove them from GitHub or add/update their records in salesforce.

    1. Bot Users: You can ignore bot users

    2. Triage-only Access: You can ignore users that only have Triage access (in the Triage group, and any other group that grants Triage or less access, such as the Product Manager group which exists only for @-mentions to work)

    3. Any sort of write access:

      1. Search the axim-engineering repo for their on or offboarding ticket. Perhaps they were on/offboarded incorrectly - fix if so

      2. Otherwise, remove their write access while investigating why they have it. Are they in a 2U group? Reach out to George. Another group? Check in Standup, and/or in GitHub logs to figure out how they got the access and ask that person.

🎨 Enable renovate on a repo

Go to the Open edX org Renovate GitHub app installation page and add the requested repo under “Only select repositories” in “Repository access”

🔏 Adding Secrets to the Openedx Github Org

For each org secret, we want to document how to rotate the secret as well so as a part of the process of adding it, you should also update /wiki/spaces/~feanil/pages/3251372059 with an entry for the new secret you’re adding. If it got accidentally exposed, what steps would we need to take to re-create it?

  1. Go to /wiki/spaces/~feanil/pages/3251372059 and add an entry for how to rotate the new secret.

  2. Go to https://github.com/organizations/openedx/settings/secrets/actions

  3. Add the new secret.

🐱 Managing the On-Call Slack integration

(This is defined as a webhook that posts a msg payload to the #axim-engineering chat room, and could be added to other GitHub repos to post to our axim-engineering chatroom)

🙊 Handling Unresponsive PR Authors

In the case that a PR author hasn’t signed the CLA but has a really trivial fix, you should first try a few times to get them to sign a CLA. If they are unresponsive, you can close the PR and remake it youself. BE SURE to follow the guidance from legal:

For trivial PRs with unresponsive authors, as a rule of thumb if the change is:

  • less than 10 lines of code,

  • mostly deletions, or

  • the only way to do something, or it is just updated facts

it's okay to close it out and still make the change.

In this case, I can see that we tried a couple times to get the CLA, and that's good practice, too.