Visualization Brainstorm - Product Core and Tech Core
Context
tCRIL has been brainstorming about what “core” means, both from a technical perspective and a product/user perspective. The new Product WG’s charter tasks the WG with defining a Core Product Offering. The graphics below provide a visual representation of possible convergence and divergence between said Core Product Offering and a technical version of the platform core.
The end goal is to decide how the Core Product Offering should relate to:
a theoretical “kernel” (a.k.a. “technical core”)
theoretical outer rings of the product
for example: an “Extended Product Offering” made up of “Official Extensions”
what’s included & what’s enabled in named community releases
which components we support and at what level
the location and ownership of code repositories (not specifically mentioned in these graphics, but relevant)
Ask
Take a look at the graphics below. Let us know your reactions, thoughts, concerns, questions, and any suggested changes. If there are places where more explanation or examples would help, let us know that too!
To keep the discussion where everyone can easily see it, please leave comments on this wiki page instead of the source Lucidchart.
Based on feedback, we’ll iterate on these ideas and eventually bring a more formal proposal to the community for review, probably as an OEP.
Graphics
Source
Requires a Lucidchart login. Note that it’s split into 3 tabs (at the bottom-left corner of the screen).
Screenshots
Comments
@Kyle McCormick I like your diagram. It makes sense to me, and I appreciate that you didn’t enumerate which extension falls in which ring - as you say on the first page, this will need to be informed by product needs.
@Jenna Makowski you write, “we are identifying the basic list of features required… I wonder if we can anticipate a next step of mapping that list against your tech core and extended product boundaries?” That seems completely logical and indeed feels like the best next step, as to fully define the boundaries I think we do need to have product definitions. I think the exercises you’re doing are wonderful. And I really like the idea of being able to swap features in and out around the core product offering to get a good “base” installation for a particular purpose (eg on-campus).
@Jenna Makowski
The Core Product Offering, ie what you get in a basic install (and by extension, this informs inclusion/enabling in named releases, etc) should respond to the primary use case for Open edX. As a starting point, we chose to focus on the use case that Open edX was effectively designed for: delivering online courses asynchronously, and at scale [and recognizing that “scale” is a characteristic that needs some wrangling.]
We anticipate having a first cut of this basic list to share by the next Product WG meeting on Aug 16. Here’s the working document - still very much still a WIP, please take for now with a grain of salt (in my mind, it’s sort of a Platform Map 2.0, but hopefully simplified and oriented around need--->feature).
Nice, I’ll take a closer look next week!
Etc. I think the end goal would be to transform the current dotted line around the Core Product Offering into a solid one and shift it such that it maps against the tech core in reality - does that seem right?
Actually, I think we should embrace there being a gap between the Core Product Offering and the Technical Core (which I’ve recently think of as the “Kernel” to establish them as different concepts). That gap between the CPO and the Kernel is made up of extensions that are installed and enabled by default. From the end user’s point of view, they shouldn’t know that those features are implemented as extensions at all. From a developer’s point of view, though, these extensions are build outside the complexity of the Kernel, which helps us maintain both the extension and the Kernel. Ideally, us engineers will strive to shrink the Kernel to be as small as possible, making the platform maximally maintainable & modular, regardless of the size of the CPO.
I do agree, though, that want to avoid having things in the Kernel that are not in the CPO. That just creates wasted effort & complexity on the technical side. In other words, the Kernel should be a strict subset of the CPO.
Actually, I think we should embrace there being a gap between the Core Product Offering and the Technical Core (which I’ve recently think of as the “Kernel” to establish them as different concepts). That gap between the CPO and the Kernel is made up of extensions that are installed and enabled by default. From the end user’s point of view, they shouldn’t know that those features are implemented as extensions at all. From a developer’s point of view, though, these extensions are build outside the complexity of the Kernel, which helps us maintain both the extension and the Kernel. Ideally, us engineers will strive to shrink the Kernel to be as small as possible, making the platform maximally maintainable & modular, regardless of the size of the CPO.
This makes sense to me. Depending on what type of persona someone is, like a course author for example, they’d have no reason to know whether the feature that comes with their product core install is enabled/configured as an extension or baked into the kernel. It wouldn’t matter to them as long as the feature functioned as expected.
Exactly.
That being said, when something is implemented via an extension framework, there is often some extra challenge in making that experience feel as seamless as it would if the feature were baked in directly. For example, in the early days of the Learning MFE, there was some time where the theming of XBlock components within the unit didn’t match the themeing of the surrounding courseware navigation UI, so it was apparent to the perceptive user that the contents of the unit had been “plugged in” to the page. We were able to fix it, but the possibility of those kind of wrinkles are just one of the technical constraints we need to think to think about when deciding to implement an extension vs. a baked-in feature. Of course, the more we invest time in designing & improving the extension frameworks themselves, the more we can ensure that they don’t result in awkward user interfaces.
[enthusiastic analysis and/or arguing semantics] I dig this. Recognizing that you’re intentionally keeping the “Core Rings” diagram agnostic to actual specific capabilities, I wonder if there’s anything that’s not related to “extensions” that belongs in that ideal technical core. i.e., is it worth calling out that some subset of data models and APIs belongs there? Put another way, are there some non-extension framework parts of the core that are fundamental to all the baked-in features? If the ideal core is something we want to work toward in reality, there needs to be something for it to extend!
I wonder if there’s anything that’s not related to “extensions” that belongs in that ideal technical core. i.e., is it worth calling out that some subset of data models and APIs belongs there?
Great question @David Joy (Deactivated) . When I imagine the ideal kernel (“kernel” == “tech core”), I think “what would the kernel look like if we had absolutely unlimited time & expertise to throw at it and no external constraints?” And in that world, I’m actually not sure if we’d bake any features in except ones that were extensible. Consider a few features that we might imagine to belong in the ideal kernel:
AuthN: I’d argue that this one’s already an extension framework. You can auth using LMS login, or you can auth via variety of SSO / external IdP options. With some refactoring, this could be a proper implementation-agnostic extension framework.
Account/Profile: You might think of a user’s account & profile as “the union of all settings, preferences, and data from all different Open edX features that they have used.” Your neighborhood Open edX instance installs some custom “achievements” plugin? You’d automatically get a new entry on your settings and profile pages. That’s an extension framework!
Support Tools: As plugins are made for Open edX, it’d be great if they could also plug in a view for the support tools interface. Extension framework.
Enrollment: This is one of those things that we actively wish wasn’t a baked-in feature. We’ve talked about how nice it’d be if LMS just a general idea of “learning context access”, with enrollment just becoming the a Course-Run-specific way (as opposed to LXC-Pathway-specific way, or whatever) of being granted access. Yup, it’s a Tide ad extension framework.
Of course, each of those extension frameworks above would need associated frontend views. The AuthN framework will still be tied together by a login/registration page, the Account/Profile frameworks would each have a centralized page, etc. You might consider those frontend views to be “features baked into the kernel.”
But, if you imagined that each of those pages were built on top of a generic REST API exposed by the underlying extension framework, allowing the community to build custom frontends on top those REST APIs… then those default frontend views are not in the kernel, because they’re not strictly necessary for the platform to run!
Hey all, looks like there’s been some substantive commentary in the comments here already. Can someone make an synthesizing pass over them and try to update the page itself?
So that’s all to say I haven’t been able to read all the previous comments but I have a couple questions:
Where does Tutor live in this diagram? I would like for it to be a container. My understanding is that keeping things inside Tutor is one way to ease extracting things from the monolith without adding deployment complexity for instance operators. So does Tutor contain… the actual kernel?
Where does the event bus live? If we make event bus a pre-req for Credentials (which is an official extension) it would need to be at that level or deeper. What if it’s a prereq for a bundled extension?
Where does Tutor live in this diagram? I would like for it to be a container. My understanding is that keeping things inside Tutor is one way to ease extracting things from the monolith without adding deployment complexity for instance operators.
Correctly placing Tutor on this diagram is tough. It’s important, though, so I’m glad you asked.
Shorter answer:
I am torn between placing it:
in the Core Product Offering, but outside the Kernel; or
outside the Open edX universe entirely.
Longer answer:
Tutor is a “distribution” of Open edX. It packages up Open edX so that you can download, configure, build, and run it. It is in fact the community-supported distribution: If you’re not running Open edX via Tutor, we don’t promise to help you (although we might still try to).
So does Tutor contain… the actual kernel?
I think it’s fair to say Tutor “wraps” parts of Open edX, but I would hesitate to paint it as a “container”, because there are others distributions (“wrappers”) out there:
DEREX and Bitnami both have alternative Open edX distributions that we market on the project site.
The old Native Installation is an Open edX distribution. It’s deprecated, but there are still plenty of people out there using it.
Devstack wraps and runs Open edX, albeit only in dev-mode.
2U/edX’s internal pipelines use various scripts and tools that configure, build, and deploy Open edX, just like Tutor does.
Notably, there is nothing in Open edX application code that expects Tutor to be the thing configuring/building/running it. There is a separation between “Open edX” and “distributions of Open edX”, which I think is a positive thing that adds flexibility to the platform. Personally, I hope that one day 2U adopts Tutor for production deployments, but if they choose to keep using their current tools because they work better for 2U’s scale, that’s fine! The community should not be adding anything into Open edX application codes that mandates Tutor usage.
Given all that, I am torn between putting Tutor in two different places:
Within the Core Product Offering, but outside the Kernel.
It would be a new type of box: “Distribution” instead of “Bundled Extension”
Each Tutor plugin would be an extension, pointing to Tutor from various rings:
The ones used to run the Core Product Offering (e.g.,
tutor-forum,tutor-mfe) would each be “Bundled Extensions”.The ones used to run the Extended Product Offering (e.g.,
tutor-android,tutor-xqueue, etc.) would each be “Official Extensions”.The ones used to run unofficial features (e.g.,
tutor-richie,tutor-figures) would be "Unofficial Extensions".
Justification:
Tutor runs the community-supported installation method and therefore must be part of the Core Product Offering.
Tutor plugins should exist in rings corresponding to the feature that they support.
Outside the Open edX Universe, in a new part of the diagram.
I would create a “Distributions” area of the diagram, with arrows indicating their relationship to the Open edX project.
Tutor, as well as Devstack/DEREX/etc. would go in Distributions area.
Justification:
Tutor is the supported way to run the Core Product Offering, but it’s not the only way. DEREX should also be able to run the CPO.
Distributions/tooling and features are inherently different kinds of things, so it would be unproductive to mix them together in the same diagram.
The Open edX project doesn’t own/control Tutor. We have a great working relationship with @Régis B. , but at the end of the day they are run as separate projects. The diagram should be clear about that separation of responsibility.
Interested to hear any and all thoughts on this.
@Adolfo Brandes Curious what you think especially. Did OpenStack have any scenarios like this, where there were important tools that were associated with the project, yet external to the project?
OpenStack is in a very similar situation, as usual. Its deployment story is firmly in “new part of the diagram” territory. It has an official list of components akin to our Core Product Offering (also subdivided into different categories) and an entirely separate list of deployment tools (which is also separate from the list of commercial distros).
I’m of the opinion we should follow that lead.
Where does the event bus live? If we make event bus a pre-req for Credentials (which is an official extension) it would need to be at that level or deeper. What if it’s a prereq for a bundled extension?
In general, I’d say that if X is a pre-req for Y, then X must be at least as close to the center as Y is. All dependency arrows should be pointing inward.
If Event Bus is a pre-req for Credentials, then Event Bus would be an Official Extension well. If it became a pre-req for a Bundled Extension, like Forums, then Event Bus would need to graduate to Bundled Extension. If it became a pre-req for LMS/CMS, then it’d be either in the Actual Kernel or Ideal Kernel (Of those two, probably Ideal Kernel. If LMS/CMS needed Event Bus, then it’d be fair to say that Event Bus is a sufficiently critical part of the platform to belong in the Ideal Kernel).
Thanks for sharing this, @Kyle McCormick ! A few comments to kick off the discussion and address some of your asks:
The end goal is to decide how the Core Product Offering should relate to:
a theoretical “technical core”
theoretical outer rings of the product
for example: an “Extended Product Offering” made up of “Official Extensions”
what’s included & what’s enabled in named community releases
Jenna: We’ve been using your visual as a reference point in the product narrative subgroup, and it is proving very helpful. We are thinking about it this way:
The Core Product Offering, ie what you get in a basic install (and by extension, this informs inclusion/enabling in named releases, etc) should respond to the primary use case for Open edX. As a starting point, we chose to focus on the use case that Open edX was effectively designed for: delivering online courses asynchronously, and at scale [and recognizing that “scale” is a characteristic that needs some wrangling.]
From this starting point, we are identifying the basic list of features required in order to support that primary use case, using a persona-based approach (for example, what key needs does a course author have in delivering against this use case, and what base features are required to support/address that need?). We will also be doing an assessment of these features, for example, is there a major pain point associated with this feature? Does this feature only work at extreme scales, ie over a million but not well at a few thousand? Does this feature represent broad needs vs edx specific needs?
We anticipate having a first cut of this basic list to share by the next Product WG meeting on Aug 16. Here’s the working document - still very much still a WIP, please take for now with a grain of salt (in my mind, it’s sort of a Platform Map 2.0, but hopefully simplified and oriented around need--->feature).
From that, I wonder if we can anticipate a next step of mapping that list against your tech core and extended product boundaries? From my point of view, in that exercise we’d be asking the “how” questions - how is each feature best implemented? Can we overcome the pain points by shifting to an extension rather than baking it in? Are there clear customization pathways that are better off falling into the extended product offering or open edx universe space? Etc. I think the end goal would be to transform the current dotted line around the Core Product Offering into a solid one and shift it such that it maps against the tech core in reality - does that seem right?
which components we support and at what level
Jenna: As further steps, we plan to do gap analyses for other popular or desired use cases. And I think this is where your distinction between Core Product Offering and Extended Product Offering is really informative. For example, the on-campus focus group has identified two possible on-campus use cases for Open edX - one as an alternative open source LMS solution, and one as a pedagogical tool to use in hybrid/blended teaching contexts. There are themes across the community too, for example multitenancy offerings for “small” trainings in corporate/non-profits, etc. From a product PoV, the goal is to distill use-case specific lists of features for each, and identify gaps in the current offering. And I can see doing a similar mapping exercise against your visual for these too. For example, the basic list of features to support an on-campus LMS use case is best achieved by combining a combination of unofficial extensions and official extensions, which could be packaged together in a way that enables easy customization/add-on, but doesn’t mess with or muddy the core offering. Rather, the core offering should be designed to easily support customization for a broad array of use cases.
cc: @Faqir Bilal @Santiago Suarez