Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 18 Current »

This is a place for notes, resource links, etc. that may be fixed and moved as part of more permanent documentation in the future.

Kafka Uncertainties

We have a working implementation, but are still unclear on some failure modes and what we want as error handling strategies:

  • Consumer error handling:

    • Do we want to use a backoff (delay before next poll) when we encounter certain kinds of errors?

    • Are there errors where we would want to push to a dead letter queue (DLQ)? What then?

    • Are there errors where we would want to just not mark the offset as consumed, and just exit the consumer (restart)? This would require somehow knowing that the consumer is entirely broken.

    • What happens if there are multiple receivers and only one errors out? Would we want to retry just that one later?

  • Consumer transactionality:

    • Should we commit offsets only on-commit of the DB transaction? (Can we assume that all consumers will be on Django with a DB, though? Don't have general knowledge of side effects of signals.)

  • Can we easily rely on re-running events after a fix? What would that take?

Older notes

These are notes from July 2022 and earlier, and many of the notes and questions are outdated.

Discovery Questions and Notes

Requirements for Enabling Squads

  • Infrastructure ready

  • Onboarding Documentation ready

    • How to create a new event

      • Requirements for a new event

        • (Initially) Must have the feature using it behind a feature flag.

        • Must have a schema

    • How to consume an event

    • How can I learn what events exist?

    • How to not break the abstraction layer (if we have one)

Schemas

  • Do we start with FULL_TRANSITIVE as a compatibility level?

    • Do we need to determine what changes we can’t make, if any, if transitive?

    • Or does this just slow performance?

Producing Events

  • If we use an outbox:

    • Do we need to have back-pressure and fail when the outbox is too large?

    • Do we need to deserialize/serialize again when pulling from outbox?

    • Is outbox needed? Does it solve ensuring we get into outbox as part of db transaction in original request?

    • May we just have an emergency outbox for when the bus is down but just use synchronous send by default?

Abstraction Layer

  • We’ll need to both keep this in mind, and looking into this more as we know more about what we wish to abstract.

General

  • When updating OEP, also see https://openedx.atlassian.net/wiki/spaces/AT/pages/3133407254/Event+Bus+POC#Findings

  • Is it a problem that we are using the term Event Bus and Message Bus interchangeably?

    • Our Architecture Manifesto (WIP) more generally documents Asynchronous over Synchronous.

    • Our manifesto was loosely based on the Reactive Manifesto, which highlights Message Driven (in contrast to Event Driven).

      • Answer: No. Event Bus and Message Bus can be used interchangeably because the term Bus implies the pub/sub messaging pattern. Message Driven vs Event Driven may have different meanings, but it is unclear if that is universal or just according to the Reactive Manifesto.

  • Do we need to get more clear on use cases, or do we wish to find a one-size fits all technology?

    • Answer: We will be starting with pub/sub.

  • Documenting event definitions and event implementations (sending).

  • When and how do we handle back-filling data for a new event stream with legacy data?

  • When might we address questions around reliable data synchronization?

    • Where would this fall in terms of discovery work?

    • For example:

      • Ensuring event matches data committed to database.

      • Ensuring events are not lost.

    • Ultimately, will we want how-tos.

  • Ownership and rollout questions regarding this infrastructure work.

    • How to enable early wild-west-like learning with a small subset, and open more widely as the path gets better paved.

    • Reminder to get feedback early and often from anyone, as possible.

  • When is the right time for a Consumer Review? Is this inform only?

Potential Materials

Other edX Initiatives

Additional Notes

Miscellaneous discovery for message bus:

  • Leading book club

  • Pulsar vs Kafka

  • Naming and name-spacing

  • Event observability

  • Types of events?

    • Log Events

    • New Relic Events

    • Tracking Events

    • Data Analysis

    • Segment Events

    • Realtime Events (Partners)

      • xAPI and Caliper

      • Missing unique ids?

    • Intraprocess Events

      • (Backend) Django Signals

      • What changes for inter/intra?

    • (Frontend) Javascript Events

  • No labels