This document summarizes learnings from product management that we would like to apply in the developer-facing Arch and SRE teams. It also includes a standardized set of practices that we propose to externalize our work.

References


Outcomes over Output

note

An outcome is a measurable change in user behavior.

An outcome is a measurable change in user behavior.

The goal of any user-facing feature/deliverable is to change the user’s behavior in a measurable and positive way. What matters is whether the deliverables are impacting user behavior, which in turn impacts product indicators, which in turn impacts business indicators.

User behavior <A> changes from <x> to <y> over timeframe <t>.

Outcome Hierarchy

A hierarchy of outcomes allows a team to connect their team’s outcomes to an overall business impact: business outcome, product outcomes, traction metrics.

Example

A. Business outcome: reduce time-to-market, via developer efficiency

The average amount of time for a developer to start a task to merge the changes into version control is reduced from 10h to 7h (by 30%). (Developer value stream prior to SRE’s Accelerate metrics.)


B. Product outcome: reduce developer context-switching while working on a developer task

The average number of times that a developer context-switches to another task on a daily basis is reduced from 60 to 42 (by 30%).


C. Traction metrics: feature usage and performance

The 95% duration that a developer waits on the most frequently used command in their local development is reduced from 60mns to 30mns (by 50%).

When developers have a question on local development, they discover the answer within 2mns instead of 20mns (10x faster).

Developers run (3x more) local development commands in a week.

Developers post 3x less frequently in the devstack-questions slack channel.

Product outcome

Product outcome:


Anti-Patterns

(blue star) Anti-pattern: Feature Factory

If you ship a feature, and it doesn’t have the desired outcome, did you succeed?

Many software teams struggle to deliver value to users. More often than not, it’s because:

  1. product and engineering teams stopped talking to users

  2. they don’t measure whether what they delivered (features) had the desired impact

    1. they don’t measure at all

    2. they measure the wrong thing

They are stuck in a feature factory, just “building stuff” and churning out features — and hoping that someone will use it.

Shipping a feature does not equal success. Changing users’ behavior in measurable and positive ways is what leads to success.

(blue star) Anti-pattern: Project-mindset

To foster an outcomes-based flow of value, teams need to move away from thinking about delivering projects, to instead having long-term user-centric:

  1. ownership of a product or

  2. support of a platform as a service.

(blue star) Anti-pattern: Push Adoption of unneeded solutions

“If I had an hour to solve a problem, I’d spend 55mns thinking about the problem and 5mns thinking about solutions.” - Einstein

By emphasizing the prioritization of problems instead of the prioritization of technical solutions, teams can focus energy on continual learning of the user’s problem space. This shift in mindset from delivering solutions to solving problems enables the team to discover users' needs, desires, and delights, which can also evolve over time.

This is different from a solutions-first approach, where a solution is chosen before fully understanding the problem space, and then pushed to be adopted by users.

(blue star) Anti-pattern: Metrics that don’t connect to business outcomes

The following measures, by themselves, tell us little about the business impact of the work. They need to be connected to the higher-level business outcome.

A more business-centric way to build and manage a product is by measuring our efforts in terms of outcomes, not output. When we measure outcomes, we measure the impact the new features had on the activity of the users AND the success of the company.


Continuous Learning via User-centricity & User-first

Opportunity trees and Product Katas (see below) are tools to enable continuous learning and refining of the user-problem space. We continue to discover opportunities through regular user touchpoints.

To ensure user-centricity, opportunities are phrased from the user’s point of view. Tie the opportunity to a specific moment in time.

note

Opportunities = user needs, user pain points, user desires, or user questions, at a moment in time.

Opportunities = user needs, user pain points, user desires, or user questions, at a moment in time.

Opportunity Tree Example


Process & Externalizing Work

Rituals

Org-wide

We will externalize our work as part of the following organization-standard rituals:

  1. Quarterly Strategy Reviews (folder)

  2. Consumer Reviews (overview, folder)

  3. Engineering Operating Reviews (overview, folder)

  4. Squad Demos (/wiki/spaces/AT/pages/1742471458, sre)

With Director(s)

In addition to the above org-wide rituals, work with your engineering director to establish a cadence of regular review of the squad’s process and artifacts. For example, the director may request a weekly touchpoint to review the squad’s WIP and artifacts from steps 3-5 below.

Process & Artifacts

This section lists standard ways for our teams to explore the problem space, prioritize, externalize, and iterate.

While the following steps are listed linearly, they are intended to be developed and updated continuously with deeper customer insights through each iteration.

(blue star) 1. MAP: User Journey

Starting with the user journey gives a high-level understanding of our user’s needs and a user-centric perspective of the high-level touchpoints with the products and services that we provide.

Cadence

Participants

Create once

Have an ideation and affinity-mapping (to group into high-level categories) exercise with members of the squad. If possible, invite stakeholders to the session.

(blue star) Remind attendees to ideate on a “moment of time” in the user’s journey and to keep it technology-agnostic and not solution-oriented.

Review and update every 6 months

With members of the squad and stakeholders.

Arch example (link)

SRE example (link)

note

Idea-1: Products, Tools, and Services offered by the team can be mapped to the part of the user journey each of them addresses.

Idea-2: Later (step 5 below), the team’s roadmap can be externalized by mapping upcoming work to each part of this user journey.

Idea-1: Products, Tools, and Services offered by the team can be mapped to the part of the user journey each of them addresses.

Idea-2: Later (step 5 below), the team’s roadmap can be externalized by mapping upcoming work to each part of this user journey.

(blue star) 2. DISCOVER: User Insights

Develop an understanding of the problem space by gaining user insights:

Cadence

Participants

Initial data gathering

A few members of the squad can timebox an exercise to gather and analyze data from existing resources. There will always be some initial data that can form a basis for hypotheses.

(blue star) Remind attendees to focus on the problem space and not ideate on solutions, just yet (that’s step 3).

Every sprint

See step 5 below on continuously learning with every sprint.

(lightbulb) 3. IDEATE: Opportunity Tree

Here are some benefits to using an opportunity tree to explore the problem space:

note

Idea: The top-level depth of the opportunity tree can be taken from the developer journey described above. This allows the team to ideate against the desired outcome across each part of the developer journey.

Idea: The top-level depth of the opportunity tree can be taken from the developer journey described above. This allows the team to ideate against the desired outcome across each part of the developer journey.

Cadence

Participants

Any time a problem needs to be explored:

  • Start of a new epic

  • Start of a new quarter

  • Start of a new fiscal year

  • Emerging new user insights

Members of the squad, along with stakeholders (if possible).

(blue star) Remind attendees to first focus on going wide with opportunities before zeroing-in on solutions.

(blue star) Remind attendees to phrase opportunities from the user perspective, to ensure user-centricity: user needs, user pain points, user desires, or user questions, at a moment in time.

Arch example (link)

SRE example (link)

(blue star) 4. PRIORITIZE: HPS & RICE & CD3

In order to prioritize amongst the various opportunities ideated, we use a structured template to further define and calculate the value and cost of each opportunity under consideration. The template has the following components:

HPS: Hypothesized Problem Statements (reference)

We believe that <this capability>
Will result in <this outcome>
We know we have succeeded when we see <this measurable signal>

RICE (reference)

  • Reach: how many developers will this impact?

  • Impact: how many developer hours are lost each week?

  • Confidence: how confident are you in your estimates?

  • Effort: how many “dev-weeks” will the team take to address this problem?

CD3: Cost-of-delay over Duration (reference)

Incorporate the values from the RICE model into the CD3 equation, as follows:

Weekly Impact = (Weekly frequency of issue) * (# of hours lost each time)

Cost-of-delay (hours lost weekly) = (Reach of developers) * (Weekly Impact)

CD3 = Cost-of-delay / Effort

TODO: Create a reusable template based on Arch and SRE’s sheets.

Cadence

Participants

Any time a new epic or reasonably sized story is considered for prioritizing in the squad’s backlog.

Members of the squad can do this together as part of their planning and/or grooming process.

(blue star) Remind the squad that anything can be measured. Without validation, you would be prioritizing output and not an outcome.

note

When appropriate, publish the proposed prioritized opportunity in a Consumer Review Approach Memo.

When appropriate, publish the proposed prioritized opportunity in a Consumer Review Approach Memo.

(blue star) 5. CONTINUOUSLY LEARN: Kata Table

“The process is about progressively solving problems, identifying the next step, and expanding understanding. Once there is sufficient understanding to anchor knowledge and take the next step, do so. Step and repeat. This framework will deepen your user knowledge and expose a host of opportunities and solutions to impact the business objective.” [reference]

Cadence

Participants

The squad’s assumed product manager and engineering manager can determine who and how this is maintained.

Example

See how this team went from 85% of users not activating to 38% of users not activating, via iterations of learning. This is well externalized via this “Product Kata” table.

(blue star) 6. SET EXPECTATIONS: Roadmap

Provide a concise way to externalize the team’s roadmap. Rather than focusing on the date of delivery, include the following:

Cadence

Participants

Every quarter

The squad’s assumed product manager can take the lead of creating and maintaining the roadmap visualization. Assuming the squad participates in the steps above, the roadmap would not be a surprise for them since it would be a result of continuously learning (step 5), ideating (step 3), and understanding user insights (step 2).

Examples

(blue star) 7. DESIGN, LAUNCH & ROLLOUT: OEPS, ADRs, Announcements

Use Open edX Proposals and Architecture Decision Records to document and publish your technical design decisions.

Announce your deliverables using Arch’s /wiki/spaces/AT/pages/2331836527 as a framework.