Building Event-Driven Microservices Ch 1-2, 2021-09-15

Overview

This meeting reviewed chapters 1 and 2 through with a presentation and some minor discussion, mostly concentrated on kick-off material. Future meetings will be run more like a book club, without a presentation.

Meeting Notes

Reading:
Chapters 1 and 2 of Building Event-Driven Microservices by Adam Bellemare

Slides: https://docs.google.com/presentation/d/1fFwZt6swGhHrmybAtJ1QxdRLHhxO9BfmRPCSzr8wuqc/edit?usp=sharing  

What are you hoping to get out of this session? Did you do the reading?

• Dave O: Generally interested in the direction we’re going with the message bus and learning more about best practices that people have been learning about or have learned from past experience. Didn’t do reading.

• Diane K: want to listen in and learn about message bus approach, have only read the OEP

• + Andy S: ditto learning, did do the reading, have some strong basically unfounded opinions to possibly share mostly about testing

• Jeremy B: Learn more about where we’re going with this, especially as I’m managing one of the key teams working on it.  Haven’t done the reading yet, but plan to.

• Ben W: here to listen and learn.  Didn’t even realize there was reading :-/

• Jinder: This seems to be a major architectural decision for the openedx ecosystem and I did not do the reading

• Adam B: I hope to learn and skimmed the why event driven chapter as we were switching docs

• David J: Hoping to ramp up on our goals and approach with the event bus to increase my understanding.  Did not do the reading; didn’t know I should!

• Brian M. Hoping to learn more about the current thoughts on message bus architecture. Did not do the reading.

• Chris Deery: I have worked in Event bus systems in the past, and I’m curious about the constraints and choices you are considering. I did not do the reading.

• Kyle M: just here to listen and learn. Didn’t do the reading.

• Adam S: mostly here to listen and learn, and understand how/when we might be able to use event-based communication across microservices; didn’t do the reading (unaware there was a reading to do).

• Long L: Here to listen and learn as well, have not done the reading.

• Régis B.: I’m interested in learning about edX’ specific vision on event-driven programming. Yes, I did the reading.

• Nim: Event Bus (and Containerization) are strategic building blocks for us to get to a decoupled architecture, enabling the Decentralized and Asynchronous aspirations of our Arch Manifesto.  This type of Data decoupling will help us more efficiently break up our monolith and implement new Bounded Contexts in our system. I did the reading.

Discussion Topics

• [quest] Adam Blackwell: How can we leverage other opensource code and write as little custom code as possible ourselves?  (Alternate: How can I learn more about what’s out there)

Notes

Open

• Distributed Monolith?

  • Is there a danger?

    • Nim: “Modular Monolith” is another term used in the industry. That is intended for a large codebase within the same repo, but with decoupled logical boundaries.  This is a viable alternative to a microservices architecture.

• What is a distributed monolith?

  • From the book: “Services may be composed such that they act as a distributed monolith, with many intertwining calls being made between them. This situation often arises when a team is decomposing a monolith and decides to use synchronous point-to-point calls to mimic the existing boundaries within their monolith. Point-to-point services make it easy to blur the lines between the bounded contexts, as the function calls to remote systems can slot in line-for-line with existing monolith code.”

  • Feanil: More complexity more adventure

  • Kyle: It takes the "independent" out of "independently deployed application"

• Periodic Bulk Updates

  • Question: Do Open edX operators need to run periodic jobs to have a working Open edX instance?

    • Feanil: Generally these are optional.  The often spoke of example is discovery course metadata

  • Question: How do other large operators run bulk updates if not from Jenkins?

    • Answer: There are several methods including https://github.com/edx/configuration/pull/4963

    • NimAt a past Arch Study, we enumerated 3 design patterns for bulk updating data across services. Slides 18-21: https://docs.google.com/presentation/d/1wlp9N6m-L6TyDZG5w1OMcQQLt5b8W7D1Vnke4qe58-k/edit#slide=id.g71d0fcec38_1_147

• Chapter 1 Check In

  • Hopes and Fears and Curiosities

    • [Adam B] Afraid we will build something that is hard to maintain and will make it harder to deploy new Open edX instances

    • [Adam B] Fear: I’m afraid it will be hard to know when to use an event bus vs a celery task/worker vs synchronous APIs

    • [Adam B] Hope: To have a way for services to publish to an event bus and for other services to subscribe to those events.

    • [Regis] Hope that this becomes a way for us to easily extend the platform

    • [Regis] Fear that it becomes difficult for Open edX operators to support

    • [Jeremy] Fear that it makes reasoning about the system harder, especially when refactoring (changing or removing an event, etc.)

    • [Jeremy] Hope that it allows more rapid development, with less time spent figuring out how to move data around

    • [David] I worry about maintaining clear, semantically meaningful event APIs.  What’s the right event to use for a given use case?  What are the right use cases for a given event?

    • [Nim] Hope: Data decoupling, with the Event Streams becoming source-of-truth will allow us to break the monolith.

    • [Nim] Fear: Staged rollout of a multi-month effort on this (similar to MFE framework rollout) will uncover unknown unknowns over time.

    • [Nim] Curiosities: Event flow architecture - connecting all eventing technologies, with simplicity and maintenance in mind. Eventing technologies that cross-modules, cross-services, and cross-organizations.

    • [Matt T] Hope: 

    • [Matt T] Fear: Battles between consumer needs

    • [Brian M] Increased Devstack Complexity / maintaining event structure in test data across services

    • Fear: Boosterism

  • Key Discussion Topics:

    • Data:

      • Owning who publishes data may put the cost of good data hygiene up front.

      • If a team need a new event published, they might end up waiting for another team or accidentally owning things

    • Observability:

      • Synchronous calls are often easier to troubleshoot

      • It will be important to know how to figure out what happened in a new event driven world.

  • Chapter 2:

    • Schema

      • Question: Do we talk about schemas in https://open-edx-proposals.readthedocs.io/en/latest/oep-0026-arch-realtime-events.html , is that OEP a good place to learn about schema considerations?

    • Partitioning

      • Book talks about partitioning in specific tools and how some things are part of a paid product offering

    • Source of Truth

      • Devil in the details