Kafka Uncertainties

We have a working implementation, but are still unclear on some failure modes and what we want as error handling strategies:

• Producers:

  • Compression is off by default. We could turn it on, but it might not be worth it if we’re using linger.ms=0 and therefore probably have batches of 1 most of the time.

• Consumer error handling:

  • Do we want to use a backoff (delay before next poll) when we encounter certain kinds of errors?

    • In event-bus-kafka 1.6.0 we did add EVENT_BUS_KAFKA_CONSUMER_POLL_FAILURE_SLEEP to prevent polling in a tight loop when the broker is failing

  • Are there errors where we would want to push to a dead letter queue (DLQ)? What then?

  • Are there errors where we would want to just not mark the offset as consumed, and just exit the consumer (restart)? This would require somehow knowing that the consumer is entirely broken.

  • What happens if there are multiple receivers and only one errors out? Would we want to retry just that one later?

    • event-bus-kafka 1.8.0 at least logs failing receivers

• Consumer transactionality:

  • Should we commit offsets only on-commit of the DB transaction? (Can we assume that all consumers will be on Django with a DB, though? Don't have general knowledge of side effects of signals.)

• Recovery:

  • If we have a bug that causes us to drop/mis-process events, how can we re-run events?

    • Probably involves resetting consumer offsets. event-bus-kafka does include offsets in its error logs.

• Unexplored settings:

  • partition.assignment.strategy – may want to use CooperativeStickyAssignor since it vaguely sounds like it might give the best cache locality

• Topics:

  • We should enable log compaction at some point. delete.retention.ms seems low (1 day) and we may want to increase it, but we’re not even sending deletes yet.

…and anything from below that is still relevant:

Older notes

Discovery Questions and Notes

Requirements for Enabling Squads

• Infrastructure ready

• Onboarding Documentation ready

  • How to create a new event

    • Requirements for a new event

      • (Initially) Must have the feature using it behind a feature flag.

      • Must have a schema

  • How to consume an event

  • How can I learn what events exist?

  • How to not break the abstraction layer (if we have one)

Schemas

• Do we start with FULL_TRANSITIVE as a compatibility level?

  • Do we need to determine what changes we can’t make, if any, if transitive?

  • Or does this just slow performance?

Producing Events

• If we use an outbox:

  • Do we need to have back-pressure and fail when the outbox is too large?

  • Do we need to deserialize/serialize again when pulling from outbox?

  • Is outbox needed? Does it solve ensuring we get into outbox as part of db transaction in original request?

  • May we just have an emergency outbox for when the bus is down but just use synchronous send by default?

Abstraction Layer

• We’ll need to both keep this in mind, and looking into this more as we know more about what we wish to abstract.

General

• When updating OEP, also see https://openedx.atlassian.net/wiki/spaces/AT/pages/3133407254/Event+Bus+POC#Findings

• Is it a problem that we are using the term Event Bus and Message Bus interchangeably?

  • Our Architecture Manifesto (WIP) more generally documents Asynchronous over Synchronous.

  • Our manifesto was loosely based on the Reactive Manifesto, which highlights Message Driven (in contrast to Event Driven).

    • Answer: No. Event Bus and Message Bus can be used interchangeably because the term Bus implies the pub/sub messaging pattern. Message Driven vs Event Driven may have different meanings, but it is unclear if that is universal or just according to the Reactive Manifesto.

• Do we need to get more clear on use cases, or do we wish to find a one-size fits all technology?

  • Answer: We will be starting with pub/sub.

• Documenting event definitions and event implementations (sending).

  • See annotations introduced in openedx-events for event definitions.

  • DE is collecting ownership information for events defined/emitted.

• When and how do we handle back-filling data for a new event stream with legacy data?

• When might we address questions around reliable data synchronization?

  • Where would this fall in terms of discovery work?

  • For example:

    • Ensuring event matches data committed to database.

    • Ensuring events are not lost.

  • Ultimately, will we want how-tos.

• Ownership and rollout questions regarding this infrastructure work.

  • How to enable early wild-west-like learning with a small subset, and open more widely as the path gets better paved.

  • Reminder to get feedback early and often from anyone, as possible.

• When is the right time for a Consumer Review? Is this inform only?

Potential Materials

• Building Event-Driven Microservices Book

• https://learning.oreilly.com/playlists/caa16a2b-2ce8-4211-b268-852876b0bf70/

• Kafka documentation: https://kafka.apache.org/documentation/ (very thorough overall)

• librdkafka FAQ: https://github.com/edenhill/librdkafka/wiki/FAQ (implementation we’re using, indirectly)

• Types of Events- At edX

• Signal Events: https://github.com/eduNEXT/openedx-events

• https://open-edx-proposals.readthedocs.io/en/latest/oep-0041-arch-async-server-event-messaging.html

Other edX Initiatives

• https://openedx.atlassian.net/wiki/pages/createpage.action?spaceKey=RS&title=Message%20Bus%20Discovery%3A%20Apache%20Kafka (Revenue Team)

Additional Notes

• Message Bus Logistics Google Doc

• Kafka Discovery

  • See 3 Python Libraries for Kafka Compared

    • In addition to choosing a Python library, we need to determine if messages are sent synchronously.

  • Kafka Connect or Kafka Streams

    • https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/

    • Note: Connectors are not managed by Amazon MSK

  • Kafka Hosting

    • Configuring to not lose data

      • https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html

      • https://towardsdatascience.com/10-configs-to-make-your-kafka-producer-more-resilient-ec6903c63e3f

    • Required discovery

      • Runbooks

      • Observability

      • Backups/disaster recovery

    • Future:

      • Zookeeper removal: https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum

      • Addition of Tiered Storage

  • Schemas

    • AWS Glue Schema Registry

    • https://www.confluent.io/blog/17-ways-to-mess-up-self-managed-schema-registry

• Event-Driven Architecture

  • Medium: Benefits and challenges of Event-Driven architecture

  • https://www.infoworld.com/article/3269207/busting-event-driven-myths.html

    • “It is also important to have strong monitoring and observability solutions in place. You need to know which service sent which events, and who is subscribed to these events. Having good visibility into the flow of events will let you understand the system and troubleshoot it with more confidence and less guessing.”

  • https://www.redhat.com/architect/defining-events-EDA

Miscellaneous discovery for message bus:

• Leading book club

• Pulsar vs Kafka

• Naming and name-spacing

• Event observability

• Types of events?

  • Log Events

  • New Relic Events

  • Tracking Events

  • Data Analysis

  • Segment Events

  • Realtime Events (Partners)

    • xAPI and Caliper

    • Missing unique ids?

  • Intraprocess Events

    • (Backend) Django Signals

    • What changes for inter/intra?

  • (Frontend) Javascript Events