Introduction

A measurement is not an absolute thing; it only relates one entity to another. As our team made efforts to improve our productivity and efficiency, we realized we needed a frame of reference by which to measure that improvement. This post is about our effort to establish a baseline measurement of the work we do. We’ll share both how we established that baseline and what we learned by doing it.

If you’d like to jump past the story of our journey, you can skip ahead to our takeaways. Here's a shortlist:

• You don't just want to measure against your OKRs; you also want to measure the energy you expend to hit those OKRs.
• The team is the in the best position to define the metrics by which the amount of work they do is measured.
• The Escalation team triages bugs at a greater rate than they are resolved.

Overview

The Escalation team is always looking to work better and more efficiently. Over the past months, we’ve worked on various efforts toward that end. While we were able to see improvements on smaller scales, we wanted to know if our efforts would affect our overall productivity.

However, we felt that our metrics we were capturing—mostly based around how many bugs we fixed and how quickly we addressed them—didn’t capture all the work we were actually doing. They may have measured some of the value our team delivered, but didn't give a sense of how much effort it took to deliver that value. In a sense, we were flying blind; we didn’t have a good way of quantitatively measuring what we were spending our energy on, making it difficult to measure changes in the team’s productivity and efficiency.

While the team recognized the value in getting this quantitative picture of our team’s work, we were also a little nervous to put a number on what we did. You make yourself feel vulnerable when you open your work to quantitative scrutiny, and it was important for us to name that and be conscious of it as we moved forward with this project.

This project was entirely conceived, executed, and maintained by the remote team. We present it now, as a blog post, to tell our story and to gather your reactions.

Identifying and Reviewing the Kinds of Work We Do

Some tickets take a lot of effort; some take very little. Before we created this performance review, we didn’t have a way to distinguish between the complexity of various tickets. After some discussion, we decided to use story points on bugs (a practice we weren’t following before).

Once we decided we would measure our efforts with story points, we had to enumerate the kinds of work we actually did. We adopted the guideline that our performance report should be anything that the Escalation team spends significant time doing.

After some reflection, we identified three basic tasks as part of our daily routine:

1. Triaging new issues

2. Fixing triaged issues

3. Closing out issues that are not bugs (usually after triaging them)

To track issues we triaged, we assign those tickets an “escalation-triaged” label. We give those tickets we fix or close without a fix an “Escalation” label. For now, we give tickets we triage or close without a fix one story point; tickets we do fix are assigned story points based on their complexity.

Leveraging those labels, we generate a wiki page on the team’s velocity and breakdown of our work. For now, we don’t report this review out; we’re using it an internal-facing tool.

What We’ve Learned from Doing This:

From coming up with and adopting this practice

• The team is in the best position to define their performance metrics, since they have the clearest sense of what they spend their time and effort working on.

• Our team works on more than just fixing bugs, but we weren’t capturing that other work.

• In order for a team to be high-performing, they need an index on the amount and kind of work they do.

• There’s a difference between measuring outcomes that you desire—speedy times to first response, number of CAT-2s fixed—and measuring the work your team is currently doing. It’s not enough to move the needle on metrics you care about; it’s also important to understand the cost of obtaining those metrics.

From the data we’ve gathered

• The amount of work we can do is correlated to the number of people on the team

• How much effort goes into our three categories of work

• What our team’s capacity for work is

• We spend a lot of time triaging bugs that aren’t fixed; in fact, we triage bugs at a greater rate than they are addressed.That means that we either need to:

  • Spend less time on triage, since a lot of this work doesn’t contribute to fixed bugs, meaning it doesn’t contribute value to end users

  • Fix more smaller bugs as part of the triage process, to cut down on the waste of having tickets sit in the backlog or go through a costly handoff between teams or through product

Future Plan:

• We will make this process automated after manual performance calculation for at least 10 sprints.

• We will create a JIRA dashboard that we'll use in our retros

• We will iterate on our measurements to make sure we continue to measure all the work we do as a team

Attribution

Original Author: Atiya Ishaq

Edited by: Adam Palay (Deactivated)