Agenda
Shall we set our new time to 2pm Tea Time for the indefinite future?
Poll: 5 yes, 1 no
Action item: Change meeting time for now - we can change back to lunch time once we’re back in the office, if we choose.
Data Redundancy
Incremental changes we can make
Asynch management commands - how can we reduce current pains with this design pattern?
How do we monitor performance issues?
We are currently shielded from production issues with celery and Jenkins workers (by DevOps, S&E, etc).
Not currently easy to determine SLO and impact of failure of these asynch tasks.
High-medium-low queues - currently relates to concurrency and not priority
Idea: have eventual versus real-time queues
Lacking dev-friendly operational management of our management commands
Onboarding and startup costs for all the inter-related service dependencies.
Idea: just-in-celery solution can simplify this - with dependent services being celery - and the service updates its own data
Need to become better at only fetching subset of data (e.g., based on data-range) rather than the entire data.
Note: this does add more complexity than if we had an event-bus.
Idea: different clusters for different priority requests
Are there any follow-up questions from the workshop?
Nim said that we would no longer want to recommend the #2 solution - do we agree with this?
Rationale: higher transactional complexity and inconsistent response times when there are cache misses.
In some cases, we are caching 1 request’s response at a time - rather than caching all the data that we would need from the service.
Memcached would have lower latency than the database.
Q: So put Memcached between a service and its Database, but not between a service and another service?
There may be certain cases where this may be a viable option.
However, we still probably want to not recommend this option as the 1st choice - and in the exceptional case we can escalate to this Arch audience so we have a discussion of the trade-offs.
Idea: can use a current problem (such as e-commerce transactions) as a case study for future Arch Study Groups.
...