Devstack Metrics

Owned by Robert Raposa
Last updated: Jun 21, 2021 by Jinder Singh (Deactivated)
3 min read

The Arch-BOM team is working on instrumenting certain devstack commands so that we can measure improvements. The metrics will not contain any personally identifiable information and will only be sent with consent.

Metric Specification

See Segment’s analytics for Python documentation.

  • event:

  • user_id

    • Property Value: Anonymous ID per device, generated on first run

    • Business Value: Differentiate different people's workflows (don't conflate patterns of usage from one user to the next. for example, frontend vs. backend devs)

    • Note: This is part of Segment’s API for the track call.

  • Event properties

    • command

      • Property Value: What make target was run (e.g. dev.pull)

      • Business Value: Which targets are used most often? (And connection to other metrics.)

      • Notes: Avoiding full command line -- more chance of privacy issues there

    • command_type

      • Property Value: What type of command was run

      • Business Value: Distinguishing between different types of commands we might collect

      • Notes: Always just make for now; at some point we may collect docker commands or make targets run from inside Docker, at which point we’ll use a different value here.

    • duration_ms

      • Property Value: Time in milliseconds that the command took to succeed or fail.

      • Business Value: Which commands are taking up the most of the people's time?

    • exit_status

      • Property Value: The exit status of the command.

        • make gives 0 for normal child process exit and 2 for non-zero child exit – we can’t get the actual value of e.g. the docker exit status.

        • If make is interrupted by a signal, indicated here by negative value of signal (e.g. -2 if killed by Ctrl-C)

      • Business Value: Determining which commands are failing (or being killed) might tell us which commands are leading to debugging sessions

    • git_checked_out_master

      • Property Value: Boolean, whether the master branch is currently checked out

      • Business Value: Allows us to filter out events made on other branches, since we’re not sure how much time people spend on branches.

      • Notes: This is mostly derivable from git_commit_time plus a copy of the repo’s public reflog; collecting it this way is only a convenience since New Relic can’t integrate that public data readily.

    • git_commit_time

      • Property Value: The timestamp on the commit that was checked out in the edx/devstack repository at the time the command was run

      • Business Value: Did dev behavior change as a result of devstack changes (since people do not pull changes immediately)?

      • Notes: This would also allow us to tell which specific commit they have checked out, but not in an immediate way. (The opposite direction is true as well.) The timestamp is more important than the commit hash since it will tell us whether their version is newer/older than some code change. This property will also let us see whether people are updating devstack, but this is not the primary business value.

    • is_test

      • Property Value: 'false' for real data, or anything else for test data.

      • Business Value: To enable testing without messing with our actual metrics data.

    • start_time

      • Property Value: Time command was started, ISO-8601 date-time (e.g. 2021-05-17T19:44:09.105052+00:00)

      • Business Value: Did dev behavior change over time? (Evaluating hypotheses when making changes.

      • Notes: With as much precision as is available without extra work—allows reconstructing event sequences.

  • Questions:

    • Do we want separate events for command_started and command_completed, or just a single event that captures the final status (error, completed, etc.)?

  • Notes:

    • Adding multiple events makes Snowflake configuration more complicated. We may want a single event to make this integration simple in the future.

Future possibilities

Attributes we may want to consider collecting:

These are ideas that we have intentionally deferred until we find we actually need this data.

  • Hardware/OS (performance, OS-specific errors)

  • Git: Whether working tree is dirty (uncommitted changes)

Additional Resources