Improving OSPR Management

Proposal by @Kshitij Sobti and @Tim Krones on behalf of OpenCraft

Topic

Indicate which topic your proposal falls under:

Enhance Core Contributor Onboarding

Improve Collaboration, Communication & Reporting

Improve Fulfilling Commitments and Planning Processes

Improve Review Processes

Overview

Our proposal addresses these two challenges in particular:

Finding reviewers: Core Contributors tend to struggle to find reviewers.
Review pace: Core Contributors would like the current review process to be faster without compromising quality.

We’re suggesting to approach these challenges from three different angles:

Making it easier for PR authors (including but not limited to CCs) to manage their own PRs.
Making it easier for CCs to help with PR reviews.
Automating repetitive parts of the OSPR management process to increase throughput.

Solution

Facilitating Self-Management of OSPRs

Simplifying the OSPR welcome message

As discussed recently on the Open edX forum, the repetitive value of the long OSPR welcome message is quite low, especially if someone is already familiar with the OSPR process. What’s more, important information about who maintains the parent repository is buried at the end of the message.

By collapsing individual sections of the message it should be possible to simplify it quite a bit, and key information could be highlighted better by putting it at the front of the message. [example]

Simplifying documentation about the kinds of OSPRs that need product review

When following the link from the OSPR welcome message to view the list of criteria making product review mandatory for OSPRs, it is not immediately clear where the list is. It lives in a collapsed section on top of the target page. It would be helpful to be able to see the list right away, and for it to include a few example PRs that do and don’t require product review.

Facilitating CC Reviews for OSPRs

Automating assignment of OSPRs

When it comes to picking up OSPRs for review, a bit of bystander effect can always set in, if everyone thinks it’s someone else’s responsibility. We could have every PR automatically be assigned to the maintainer(s) of the parent repo. They could then self-request a review or, if necessary, request a review from someone else, which would help spread the responsibility for reviewing OSPRs to other CCs. As a first step we could consider trialing this approach with a willing maintainer and see how it goes.

Improving documentation on how to find OSPRs to review

We have special views on the Contributions board that list OSPRs in need of a reviewer (all OSPRs, edx-platform only). It might not be clear to CCs – especially if they’ve been onboarded to the program only recently – that finding OSPRs to review can be as simple as going to those views, picking one or more PRs that align with their area(s) of expertise, and assigning themselves as reviewers. In order to make CCs aware of this option, we’d suggest that the onboarding course for CCs and/or related onboarding documentation be updated to include pointers about how to find OSPRs to review.

Reducing Toil for OSPR Managers

Making CC info for individual repositories easily discoverable

There have been some recent discussions on the Open edX forum and on GitHub about this topic.

One way to approach this would be to create some simple tools to get the current set of CCs for a repository, and list/order them based on a few simple factors, e.g.:

parent organization
number of PRs they’re currently reviewing

With this information it would not only be possible to quickly identify the entire pool of potential CC reviewers for a given PR, but also to determine which of them might be in the best position to pick up the review for that PR right now.

Another way would be to try and integrate CC info directly into the Contributions board (which is the main tool OSPR managers use for triage).

We could also consider leveraging the CODEOWNERS file. Since that file allows us to map parts of the code in a repository to who “owns” that bit of code, utilizing it might be especially helpful when it comes to determining who would be the best reviewer for a specific PR that has been opened against a big repository like edx-platform.

Automating manual OSPR management tasks

The OSPR management process involves a number of tasks that OSPR managers currently perform manually, and that could be automated to free up time for processing more PRs in the time that OSPR managers have available for triage. These tasks include:

Managing labels

We suggest automating the following tasks from this category:

Setting the core contributor label upon PR creation.
- This will require getting a global list of CCs – even if someone is not a coding CC for one or more repos, they could still open a PR – e.g. from the Core Contributors to the Open edX Project wiki page, and checking whether the PR author is on that list.
Removing obsolete labels when a PR gets merged:
- blocked by other work
- changes requested
- inactive
- needs maintainer attention
- needs more product information
- needs rescoping
- needs reviewer assigned
- needs test run
- waiting for eng review
- waiting on author
Removing obsolete labels when a PR gets closed:
- needs maintainer attention
- needs reviewer assigned
- needs test run
- waiting for eng review
- waiting on author

Managing metadata for the Contributions board

We suggest automating the following tasks from this category:

Retrieving the creation date of an OSPR from GitHub to populate the Date opened field. Ideally, this would also be done for platform roadmap tickets.
Retrieving maintainer info from catalog-info.yaml, just like we are already doing for the OSPR welcome message, and using it to populate the Repo Owner / Owning Team field.
- For repos maintained by specific individuals, we suggest to include both their full name (if available) and their GitHub handle, e.g. Brian Mesick (@bmtcril).
- As a side note, “Repo Owner / Owning Team” terminology is outdated at this point. “Maintainer(s)” or “Maintaining Team(s)” would be more appropriate, as it would better reflect current practices. We should consider renaming the field.

Additionally, we could consider introducing a Date merged/closed field and automatically populate it with data from GitHub as well.

This might help us identify repositories whose time-to-merge is particularly high. (Though checking for the number of long-running PRs and/or looking at data from the Maintainer Pull Request Reporting dashboard could give us a sense of that as well.)

Managing board statuses

Here, a possible improvement would be to automatically change the status of an OSPR from Needs Triage to Waiting on Author on the Contributions board if it is marked as draft upon creation.

Impact

We suggest using time-to-merge as the main factor to look at for measuring the impact of the changes proposed above. There are multiple options for tracking / evaluating time-to-merge.

Using metadata from the Contributions board: Averaging the difference between Date opened and Date merged/closed, either globally or for specific repositories.
Using data from the Maintainer Pull Request Reporting dashboard.
Using https://chaoss.community/kb/metric-change-request-closure-ratio/ data from GrimoireLab.

Irrespective of the tracking option we decide on, any review/evaluation of time-to-merge should take into account that OSPRs labeled as “FC” (Funded Contributions) typically get handled much faster than non-FC OSPRs because there are teams actively working on them in a set timeframe. So to avoid altering the data/results, we’d need to look at FC and non-FC pull requests separately.

Timeline

We expect that documentation-only changes from this proposal would not take more than 1-2 months to implement.
Implementing technical changes from this proposal would likely take a bit longer than that (~6 months).