Blended Development Runbook and Best Practices

Created by Nimisha Asthagiri
Last updated: Jan 04, 2021 by Sarina Canelake

Why Blend?

  1. Increase edX dev capacity - to create a valuable platform for edX.org and Open edX - to achieve sustainability in 3-years.

  2. Re-energize the Open edX community with

    1. funded projects that are aligned with the edX.org product roadmap and

    2. direct access to edX senior technical/product folks so future contributions are aligned with edX needs.

How to Blend?

Some early good signs

  • Overall, we are seeing high-quality code from our Blended developers.

  • When Blended developers are reviewing each other’s code, it helps us expedite our own reviews.

  • edX developers are having closer relationships with the community.

  • We are seeing edX able to move faster with various parallel efforts.

  • Communication in PRs has been relatively responsive, with relatively tight feedback loops.

Runbook on managing blended development

(The following list of Problems were curated by the TNL team in their retro on July 1st, 2020)

Communication

Problem to avoid

Mitigation (see Best Practices below)

C1. Reverse engineering design & implementation through time-consuming code reviews.

2B: PR Reviews → Self-review-n-doc PR
2C: PR Reviews → Core-committer merges

C2. Catching up on discovery/context already completed by the Blended provider.

2B: PR Reviews → Self-review-n-doc PR

C3. Intermediating without facilitating or adding value.
(Blended Developer PR → edX Tech lead → Blended Tech lead → Blended Developer ..)

1C: Project Considerations → Engagement model
2C: PR Reviews → Core-committer merges

C4. Low signal to noise ratio → easy to lose track of small-but-important items.
(PRs, tickets, etc)

2F: PR Reviews → PR-ready bat-signal

C5. Blended developers lack the context of under-specified tasks.

1B3: Project Considerations → When to Blend → Tech Review
5: Check-ins

C6. As edX tech lead, difficulty answering product-oriented questions due to lack of context on the product being delivered.

2D: PR Reviews → edX pair-review

Quality

Q1. Under-confidence or over-confidence of the architecture/design by the Blended team.

2E: PR Reviews → Cross-team visibility

Q2. Lack of a deep understanding of the implementation since you are not implementing it.

2B: PR Reviews → Self-review-n-doc PR

Q3. Blended developers lack the context of coding standards and arch decisions/directions.

4: edX Documentation

Q4. Inconsistent quality from Blended developers, especially from non-tech leads.

2A: PR Reviews → Intra-blended review

WIP/Interrupts

W1. Unable to focus on the team’s own deep work.

1C: Project Considerations → Engagement model
2C: PR Reviews → Core-committer merges
3C: Squad Balance → Blended-ratio

W2. Unable to focus on the Blended team’s work.

1C: Project Considerations → Engagement model
2C: PR Reviews → Core-committer merges
3C: Squad Balance → Blended-ratio

W3. Projects actually take longer to complete.

1A: Project Considerations → Whether to Blend
3C: Squad Balance → Blended-ratio

W4. Context-switching between unrelated projects impacts effectiveness & efficiency.

3C: Squad Balance → Blended-ratio
3D: Squad Balance → Single-focus

W5. Vetting future blended projects requires further context-switching and time-consuming.

1B2: Project Considerations → When to Blend → Tech Discovery

W6. Reviewing delays when teams include junior developers across time-zones.

1D2: Project Considerations → Who to Blend → Number of devs
2E: PR Reviews → Cross-team visibility

Agile

A1. The blended project is delivered in a waterfall style without iterations and user feedback.

5B: Check-ins → Blended Demo

A2. UX input is incorporated toward the end of the project due to the lack of UX design time.

1B1: Project Considerations → When to Blend → UX Ready

A3. Squad engineers work independently without any squad-specific focus or goals.

3C: Squad Balance → Blended-ratio

Best Practices

  1. Project Considerations

    1. Whether to Blend

      1. No to Proprietary. Do not Blend projects that heavily require developer access to edX.org proprietary deployment, data, or content.

      2. Yes to Independence. Projects that are mostly independent with a few integration points with the core platform are ideal for Blended development.

    2. When to Blend

      1. UX Ready. Delay the start of a Blended project until after UX design and wireframes are complete (for the first iteration) to prevent thrashing and delays.

      2. Tech Discovery. Delay discovery of a Blended project until the edX technical lead of the project is available to be part of the discovery phase.

      3. Tech Review. Delay the start of the Blended project until the squad has validated that the brief/tasks have sufficient detail.

      4. Core/Runways Ready. Delay implementation of a Blended project until edX implements needed core APIs and infrastructure.

    3. Engagement model

      1. Design-only review. edX reviews only the design decisions primarily. Consider this model similar to purchasing an external library/tool.

      2. Test-driven review. Create tests first or use an external test harness to validate the implementation, without needing close review. Consider this model for certifiable standards.

      3. Development + Review. edX engineers are actively involved in development, sharing the workload with the providers.

    4. Who to Blend

      1. Provider capability. Choose providers in the preferred provider program according to their demonstrated experiences and skills.

      2. Number of devs - on both edX side and provider side.

  2. PR Reviews

    1. Intra-blended review. Require the Blended team to review each other’s PRs within the team before submission.

    2. Self-review-n-doc PR. Require the Blended contributor to self-review.

      1. Adding comments on the PR with “notes for reviewers”.

      2. Adding appropriate ADRs to document decisions and design intent along with the PR.

    3. Core-committer merges. Ask Core Committers in the Blended organization to review and merge PRs.

      1. ADRs would still be reviewed by all.

      2. See https://openedx.atlassian.net/wiki/spaces/COMM/pages/1729200289/OSPR+Reviews+by+Core+Committers.

      3. Have changes be built as an edx-platform extension in a separate repo. That is, add a new repo - not a new app in edx-platform.

    4. edX pair-review. Pull in teammates for pair review, especially if they have more context on something (with the understanding that they’re not necessarily signing up to be a tech lead of the project!). To avoid interrupting their flow, turn it into a task.

    5. Cross-team visibility. Visibility into the Blended team's SLA expectations, skill-levels, resourcing, etc.

      1. Provide background/acknowledgment of non-Core-Contributors' skill level (eg: “This is my first React project”, etc. to provide context)

      2. How many developers.

    6. PR-ready bat-signal. Follow the Blended PR Workflow. When the PR is ready for a pass, attend to it as quickly as possible. Otherwise, ignore all other intermediary reviews/communication on PRs.

  3. Squad Balance

    1. Dual-track. Squad rotates through 2 different tracks: interrupt-driven (Blended, Support, On-call, etc) versus deep-focus (squad implementation).

    2. Dual-leads. Technical lead for non-blended projects are not also technical leads for blended projects–someone should own coordination for things like the courseware MFE rollout.

    3. Blended-ratio. The squad has a maximum number of parallel blended projects based on capacity and non-blended workload.

    4. Single-focus. The squad is singularly focused on a single Blended project (with reviews and co-implementation with the Blended team), with only a few additional Blended projects on the side.

  4. edX Documentation

    1. ADRs, OEPs, Handbook. Invest in writing ADRs, OEPs, Developer Handbook entries, etc to provide deeper context to Blended teams.

  5. Check-ins

    1. Joint Retro. Globally optimize pain-points on both sides by having a project-wide retro regularly with the Blended team.

    2. Blended Demo. Design/a11y/implementation check-ins at various points in the project.