Building Event-Driven Microservices Ch 3-4, 2021-09-29
- Robert Raposa
- Feanil Patel (Deactivated)
Discussions of the book Building Event-Driven Microservices
Chapters 3-4 Outline
Discussion Notes
Author strongly recommends schema management. How do we feel about schema management for events?
It’s always gonna have a schema, it's a matter of how much you manage it.
Formal schema management is a useful tool for doing this thoughtfully at scale.
Where does schema management live?
Schema registry, holds schema and can be used to evaluate if new schema is compatible with existing schema.
Compatibility mode
Start with full and go from there.
Hard to imagine not having full compatibility if we want to have a large number of consumers.
Counterpoint: We could deprecate old versions of schema and communicate deadlines between producer team and consumer teams.
This suggests another line of communication between various teams.
Design Section
Many of the principals seemed to conflict.
One event definition per stream.
How do we handle CRUD? Separate streams so we have one for each action. That seems very heavy and we’d have to worry about ordering at the consumer.
The idea is to not overload your entity topics.
Let a thousand streams bloom but each stream should be one entity.
Where ordering matters, we may want to push away from one event per stream, to be able to reason about when events happened.
There’s pure click stream, entity streams, and there are things in between.
Eg. I need to take an action that I need to take when an enrollment occurs.
Eg. User used to pass and used to fail.
Some things may need be short cut by IDs so event sizes don’t blow up. The referenced IDs would be the key in entity events
When mapping from tables to entities, how do you deal with ids and foreign keys?
Might have to be critical of what the domain concept is that you want to convey. This might be at odds with how it’s laid out in a relational database.
Be mindful of entities growing too big.
But also be careful about pushing a bunch of ids in a message and then seeing a bunch of call-backs to fetch the data of those IDs
The idea of redundancy is not really built into the messaging systems.
Keeping track of what the context was of a change.
Eg. Indicate what the enrollment mode was before the change and what it is after this event.
Do we want to keep entity events and subset of entity events?
Sometimes you might not care about the underlying event but a meta concept on top of it.
Grade change vs pass a course.
Author is pushing Kafka
Expectation is that it’s treated more akin to the SQL database behind an app.
Commit to the data store as a core data store that can make high reliability guarantees.
We don’t say what to do if the database falls out of sync.
Data Liberation
Saying I’m going to entity stream everything, I don’t like encapsulation.
By exposing all your internal RDBMS schema, change management can be very complex and schema management becomes more difficult.
Things happen and you need to react to it is a core part of the business, the book provides many strategies but it’s up to you to build good events that give you enough context to take the correct business actions.