Event Bus POC
Status
Working on Milestone 1: Basics working in devstack and Stage
Purpose of POC
In-depth trial of a technology choice, including development and hosting.
Ensure primary use case is solved. See draft OEP-52 for details.
Note: Likely use case is the certificate and grades use case documented in https://openedx.atlassian.net/wiki/pages/createpage.action?spaceKey=AT&title=Event%20Bus%20Use%20Cases.
Devstack
In progress work
PRs:
See commit comments for more instructions.
Uses confluent platform and confluent python library.
Streamnative Vendor Review:
Next steps
Management command for consumer.
Management command for outbox-to-Producer
Add schema to messages for Pulsar.
Can we make grade change events simpler to test? Course author manual grade change - does this fire the event?
This is an event with a customer chomping at the bit.
Confluent (Kafka) Vendor Review?
Cloud vs Platform (in our AWS)
[TBD] Error handling? - After we figure out which liberation pattern we’re using (outbox, in-code).
[TBD] Update OEP?
[TBD] What would need to be in an abstraction layer?
[DEFER] Schema evolution
[DEFER] Move consumer to management commands, so they can be started in separate containers.
Implement producer code for grades event in edx-platform.
Implement management command consumer code in credentials.
Understand flow of running for engineers.
Findings
Kafka Hosting
Standing up Kafka in Amazon MSK was difficult.
Terraform topics didn’t seem to create.
Arch-BOM needs fully-managed infrastructure. If not SRE, then we need a fully-managed third-party.
Trying Pulsar synchronous-send
Pulsar Schema Cons:
Python client AvroSchema + Record class bugs require workarounds.
Compatibility level set for namespace.
Loosening requires affecting all topics, or migrating to a new topic.
General Question: Do entity topics need to be transitive?
No UI available? What does Streamnative offer?
Pulsar schema evolution errors are displayed as Unknown Errors which is not as useful as we’d like.
Pulsar command-line schema management may have a number of other bugs, around newlines, etc.