| Info |
|---|
This is a place for notes, resource links, etc. that may be fixed and moved as part of more permanent documentation in the future. |
Tasking Notes
| Note |
|---|
These are personal notes for Robert Raposa to eventually get into JIRA. |
Event Bus Tasks
Schema:
CloudEvents vs Avro
attrs
in openedx-events
Make the real version of this code, likely the transformation functions need to also move into the bridge class.
Needs tests written to validate the encode/decode.
Currently does not handle non-base types(opaque keys) but the final version should be able to handle this.
Maybe error out on other class types we’re not prepared to serialize/desserialize.
Should hard-code skipping PII objects(There is a personal data attrs object) for now.
Ensure devstack work is merged to master.
Instructions, review
Looking into Grades events
Grade codes
Design with Aperture
Certificate Change - ideal
Toggles
Starting up Kafka Client based off Kafka setting in place.
This is what provides option for new events to start.
Implementing producer and event
Don’t need yet: Event should have its own toggle to make optional.
Design events for credentials
An unfiltered event, where credentials does its own filtering, rather than leaving in edx-platform
simplest error handling to start
Verify that credentials action is idempotent and could handle event bus + old functionality at the same time.
Kafka Prototype updated(Spike, not expecting to merge this)
TODO: Links to edx-platform PRs
Match Pulsar
No outbox
Maybe get details from Feanil.
Create a management command that will consume a test topic and process the event(Print it)
Producer should produce with a well defined avro schema.
Fine to be a test event/topic
Consumer should be able to de-serialize to a well defined Python Object.
Discovery: abstraction layer (Feanil)
Implementing consumer
TODO:
infrastructure work
consumer management command
kubernetes/helm work for a consumer process that runs forever
"Hello World", sleeping and logging
Stage [deferred]
streamnative:
set up stage/prod clusters
defer peering until PII and other data
Discovery Questions and Notes
...
When updating OEP, also see https://openedx.atlassian.net/wiki/spaces/AT/pages/3133407254/Event+Bus+POC#Findings
Is it a problem that we are using the term Event Bus and Message Bus interchangeably?
Our Architecture Manifesto (WIP) more generally documents Asynchronous over Synchronous.
Our manifesto was loosely based on the Reactive Manifesto, which highlights Message Driven (in contrast to Event Driven).
Answer: No. Event Bus and Message Bus can be used interchangeably because the term Bus implies the pub/sub messaging pattern. Message Driven vs Event Driven may have different meanings, but it is unclear if that is universal or just according to the Reactive Manifesto.
Do we need to get more clear on use cases, or do we wish to find a one-size fits all technology?
Answer: We will be starting with pub/sub.
Documenting event definitions and event implementations (sending).
See annotations introduced in openedx-events for event definitions.
DE is collecting ownership information for events defined/emitted.
How will back-filling of events work?
When might we address questions around reliable data synchronization?
Where would this fall in terms of discovery work?
For example:
Ensuring event matches data committed to database.
Ensuring events are not lost.
Ultimately, will we want how-tos.
Ownership and rollout questions regarding this infrastructure work.
How to enable early wild-west-like learning with a small subset, and open more widely as the path gets better paved.
Reminder to get feedback early and often from anyone, as possible.
When is the right time for a Consumer Review? Is this inform only?
Potential Materials
https://learning.oreilly.com/playlists/caa16a2b-2ce8-4211-b268-852876b0bf70/
Types of Events- At edX
Signal Events: https://github.com/eduNEXT/openedx-events
https://open-edx-proposals.readthedocs.io/en/latest/oep-0041-arch-async-server-event-messaging.html
...