Devstack Metrics Data Retention

Context

The team was given a legal requirement to keep metrics data for 2 years (or less). This decision and the configuration were implemented as part of https://openedx.atlassian.net/browse/ARCHBOM-1777.

Decision

The current data stores and retention settings for the devstack-metrics Segment events are as follows:

• Segment’s internal(ish) S3 bucket (enables replaying of events).

  • This will be configured to 1 year retention (the maximum available less than forever).

  • See private DESUPPORT implementation ticket: https://openedx.atlassian.net/servicedesk/customer/portal/16/DESUPPORT-910 (ask Robert Raposa or DE for access)

• New Relic DevstackMetrics event type.

  • This data expires with our current New Relic data retention of 31 days.

Consequences

• Any new Segment destinations for devstack metrics will need to be configured for 2 years or less.

• If we want 2-year data retention, we’d need to configure a new Segment destination to our own S3 bucket, and configure it with a 2-year data retention policy.

  • Note: only a maximum of 1-year of Segment data could be replayed to this destination. This would need to happen sooner than June 2022 to avoid data loss.