The Arch-BOM team is working on instrumenting certain devstack commands so that we can measure improvements. The metrics will not contain any personally identifiable information and will only be sent with consent.
See Segment’s analytics for Python documentation.
event:
Property Value: devstack.command.run
See DE best practices doc (if you can)
Note: This is part of Segment’s API for the track call.
user_id
Property Value: Anonymous ID per device, generated on first run
Business Value: Differentiate different people's workflows (don't conflate patterns of usage from one user to the next. for example, frontend vs. backend devs)
Note: This is part of Segment’s API for the track call.
Event properties
command
Property Value: What make target was run (e.g. dev.pull)
Business Value: Which targets are used most often? (And connection to other metrics.)
Notes: Avoiding full command line -- more chance of privacy issues there
command_type
Property Value: What type of command was run
Business Value: Distinguishing between different types of commands we might collect
Notes: Always just make for now; at some point we may collect docker commands or make targets run from inside Docker, at which point we’ll use a different value here.
duration_ms
Property Value: Time in milliseconds that the command took to succeed or fail.
Business Value: Which commands are taking up the most of the people's time?
exit_status
Property Value: The exit status of the command.
make gives 0 for normal child process exit and 2 for non-zero child exit – we can’t get the actual value of e.g. the docker exit status.
If make is interrupted by a signal, indicated here by negative value of signal (e.g. -2 if killed by Ctrl-C)
Business Value: Determining which commands are failing (or being killed) might tell us which commands are leading to debugging sessions
git_checked_out_master
Property Value: Boolean, whether the master branch is currently checked out
Business Value: Allows us to filter out events made on other branches, since we’re not sure how much time people spend on branches.
Notes: This is mostly derivable from git_commit_time plus a copy of the repo’s public reflog; collecting it this way is only a convenience since New Relic can’t integrate that public data readily.
git_commit_time
Property Value: The timestamp on the commit that was checked out in the edx/devstack repository at the time the command was run
Business Value: Did dev behavior change as a result of devstack changes (since people do not pull changes immediately)?
Notes: This would also allow us to tell which specific commit they have checked out, but not in an immediate way. (The opposite direction is true as well.) The timestamp is more important than the commit hash since it will tell us whether their version is newer/older than some code change. This property will also let us see whether people are updating devstack, but this is not the primary business value.
is_test
Property Value: 'false' for real data, or anything else for test data.
Business Value: To enable testing without messing with our actual metrics data.
start_time
Property Value: Time command was started, ISO-8601 date-time (e.g. 2021-05-17T19:44:09.105052+00:00)
Business Value: Did dev behavior change over time? (Evaluating hypotheses when making changes.
Notes: With as much precision as is available without extra work—allows reconstructing event sequences.
Questions:
Do we want separate events for command_started and command_completed, or just a single event that captures the final status (error, completed, etc.)?
Notes:
Adding multiple events makes Snowflake configuration more complicated. We may want a single event to make this integration simple in the future.
Attributes we may want to consider collecting:
These are ideas that we have intentionally deferred until we find we actually need this data.
These are ideas that we have intentionally deferred until we find we actually need this data.
Hardware/OS (performance, OS-specific errors)
Git: Whether working tree is dirty (uncommitted changes)
Discovery ticket https://openedx.atlassian.net/browse/ARCHBOM-1694#icft=ARCHBOM-1694