Create an experiment to verify our hypothesis that contention over updating sequence user state during parallel loads of the courseware page is causing the CSM write latency spikes we're seeing.
Using a branch, implement sequence state storage in memcached. We're just doing an experiment regarding latency, so we don't have to actually wire it through the XBlock field data APIs. We just need to have Sequences get and set position through memcached access.
Using a sandbox, compare the write latency of this new branch vs. master under situations that we suspect cause contention (e.g. edx-dl).
If this experiment confirms that this would significantly improve the situation in prod, map out a few options. We might try to make CSM updates more granular, or modify the XBlock field API to allow for an optional ephemeral flag (backed by memcached), or rethink how sequences store position in general, etc. Whatever we come up with, we have to make sure that we run it by:
T&L, to make sure this would be acceptable from a product point of view.
Analytics, to make sure this would be acceptable from a research point of view (it hopefully is, since events give a much better picture of sequence navigation anyhow).
The community in general, that might have features built on top of sequence position state (maybe just make it a feature flag?)