Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 17 Next »

This is a place for notes, resource links, etc. that may be fixed and moved as part of more permanent documentation in the future.

Discovery Questions and Notes

Requirements for Enabling Squads

  • Infrastructure ready

  • Onboarding Documentation ready

    • How to create a new event

      • Requirements for a new event

        • (Initially) Must have the feature using it behind a feature flag.

        • Must have a schema

    • How to consume an event

    • How can I learn what events exist?

    • How to not break the abstraction layer (if we have one)

Schemas

  • Do we start with FULL_TRANSITIVE as a compatability level?

    • Do we need to determine what changes we can’t make, if any, if transitive?

    • Or does this just slow performance?

Producing Events

  • When and how do we handle back-filling data for a new event stream with legacy data?

  • If we use an outbox:

    • Do we need to have back-pressure and fail when the outbox is too large?

    • Do we need to deserialize/serialize again when pulling from outbox?

    • Is outbox needed? Does it solve ensuring we get into outbox as part of db transaction in original request?

    • May we just have an emergency outbox for when the bus is down but just use synchronous send by default?

Abstraction Layer

  • We’ll need to both keep this in mind, and looking into this more as we know more about what we wish to abstract.

General

  • When updating OEP, also see https://openedx.atlassian.net/wiki/spaces/AT/pages/3133407254/Event+Bus+POC#Findings

  • Is it a problem that we are using the term Event Bus and Message Bus interchangeably?

    • Our Architecture Manifesto (WIP) more generally documents Asynchronous over Synchronous.

    • Our manifesto was loosely based on the Reactive Manifesto, which highlights Message Driven (in contrast to Event Driven).

      • Answer: No. Event Bus and Message Bus can be used interchangeably because the term Bus implies the pub/sub messaging pattern. Message Driven vs Event Driven may have different meanings, but it is unclear if that is universal or just according to the Reactive Manifesto.

  • Do we need to get more clear on use cases, or do we wish to find a one-size fits all technology?

    • Answer: We will be starting with pub/sub.

  • Documenting event definitions and event implementations (sending).

  • How will back-filling of events work?

  • When might we address questions around reliable data synchronization?

    • Where would this fall in terms of discovery work?

    • For example:

      • Ensuring event matches data committed to database.

      • Ensuring events are not lost.

    • Ultimately, will we want how-tos.

  • Ownership and rollout questions regarding this infrastructure work.

    • How to enable early wild-west-like learning with a small subset, and open more widely as the path gets better paved.

    • Reminder to get feedback early and often from anyone, as possible.

  • When is the right time for a Consumer Review? Is this inform only?

Potential Materials

Other edX Initiatives

Additional Notes

Miscellaneous discovery for message bus:

  • Leading book club

  • Pulsar vs Kafka

  • Naming and name-spacing

  • Event observability

  • Types of events?

    • Log Events

    • New Relic Events

    • Tracking Events

    • Data Analysis

    • Segment Events

    • Realtime Events (Partners)

      • xAPI and Caliper

      • Missing unique ids?

    • Intraprocess Events

      • (Backend) Django Signals

      • What changes for inter/intra?

    • (Frontend) Javascript Events

  • No labels