Arch Tea Time: 2020-03-26 (Data Redundancy followup)

Owned by Nimisha Asthagiri (Deactivated)
Last updated: May 14, 2020
2 min read

Agenda

  • Shall we set our new time to 2pm Tea Time for the indefinite future?

    • Poll: 5 yes, 1 no

    • Action item: Change meeting time for now - we can change back to lunch time once we’re back in the office, if we choose.

  • Data Redundancy

    • Incremental changes we can make

      • Asynch management commands - how can we reduce current pains with this design pattern?

        • How do we monitor performance issues?

        • We are currently shielded from production issues with celery and Jenkins workers (by DevOps, S&E, etc).

        • Not currently easy to determine SLO and impact of failure of these asynch tasks.

        • High-medium-low queues - currently relates to concurrency and not priority

          • Idea: have eventual versus real-time queues

        • Lacking dev-friendly operational management of our management commands

          • Onboarding and startup costs for all the inter-related service dependencies.

          • Idea: just-in-celery solution can simplify this - with dependent services being celery - and the service updates its own data

        • Need to become better at only fetching subset of data (e.g., based on data-range) rather than the entire data.

          • Note: this does add more complexity than if we had an event-bus.

      • Idea: different clusters for different priority requests

    • Are there any follow-up questions from the workshop?

      • Nim said that we would no longer want to recommend the #2 solution - do we agree with this?

        • Rationale: higher transactional complexity and inconsistent response times when there are cache misses.

          • In some cases, we are caching 1 request’s response at a time - rather than caching all the data that we would need from the service.

        • Memcached would have lower latency than the database.

        • Q: So put Memcached between a service and its Database, but not between a service and another service?

          • There may be certain cases where this may be a viable option.

          • However, we still probably want to not recommend this option as the 1st choice - and in the exceptional case we can escalate to this Arch audience so we have a discussion of the trade-offs.

    • Idea: can use a current problem (such as e-commerce transactions) as a case study for future Arch Study Groups.

    •