Devstack Metrics Data Retention

Owned by Robert Raposa
Last updated: Jun 03, 2021
1 min read

Context

The team was given a legal requirement to keep metrics data for 2 years (or less). This decision and the configuration were implemented as part of https://openedx.atlassian.net/browse/ARCHBOM-1777.

Decision

The current data stores and retention settings for the devstack-metrics Segment events are as follows:

  • Segment’s internal(ish) S3 bucket (enables replaying of events).

    • This will be configured to 1 year retention (the maximum available less than forever).

    •  

    • This was implemented in private DESUPPORT implementation ticket: https://openedx.atlassian.net/servicedesk/customer/portal/16/DESUPPORT-910 (ask @Robert Raposa or DE for access)

  • New Relic DevstackMetrics event type.

    • This data expires with our current New Relic data retention of 31 days.

Consequences

  • Any new Segment destinations for devstack metrics will need to be configured for 2 years or less.

  • If we want 2-year data retention, we’d need to configure a new Segment destination to our own S3 bucket, and configure it with a 2-year data retention policy.

    • Note: only a maximum of 1-year of Segment data could be replayed to this destination. This would need to happen sooner than June 2022 to avoid data loss.