...
Initiating and specifying three major milestone projects: Open Analytics Reference System (OARS), Events over Increasing Message Bus Adoption, and Tracking Log Event Cleanup. These projects arose from the experiences and discussions between Data WG members. But they really took shape once Brian Mesick joined tCRIL to head the Platform Data effort, and Jenna Makowski directed her Product Management lens at the pain points and use cases expressed by the community.
...
Discovery and Specification for OARS V1
We have decided to replace Insights with the Open Analytics Reference System, a light, flexible data pipeline based primarily on 3rd party open source solutions for routing, storing and analyzing Open edX event data. The OARS architecture will be cost effective for small to medium sized Open edX sites, will scale appropriately for their expected data use, and comprise loosely-coupled components which operators can exchange if their deployments require. OARS is composed of several components, and we have investigated and decided on most of the technologies included in the reference implementation, including RedisClickhouse, ClickhouseRalph, and Superset.Reference implementation for OARS V1
Alongside the discovery and specification, we have also begun the reference implementation for OARS in the Tutor environment.
[optional] We continue to engage in a set of ongoing tasks; these tasks are work we need to be sure we’re doing on a regular basis, but we don’t want to lose sight of their importance. Those tasks are:
Ongoing task 1; 1-2 sentences of importance/impact
Ongoing task 2; 1-2 sentences of importance/impact
…
...
Redis Streams as a Message Bus
We have a funded contribution project in progress with OpenCraft to complete a reference implementation of redis streams as a second concrete implementation of the Open edX message bus. This should allow all operators of Open edX to be able to gain the benefits of asynchronous messaging and advance our plans for a less coupled architecture.Google Analytics 4 upgrade
We also have a funded contribution project in progress with Racoon Gang to upgrade our Google Analytics support and expand GA tracking into several microfrontends that did not have GA support added when they were moved out of edx-platform.
Concrete Plans - Next 6 Months
Over the next six months, {working_group_name} has some really exciting work queued up. We hope to drive {key metrics, description of impact, etc}. Our primary goals are to {work on / complete / another word} the following tasks and projects:
If applicable: Links to roadmaps/GitHub projects/important wiki pages/etc
Optional: We’ll provide a deeper dive into (name project(s) you’ll deep dive into) later in this reportthe Data Working Group expects to start seeing the fruits of our planning. We hope for make a functional OARS v1 available for testing and feedback, and make detailed plans for OARS v2 which will be centered around returning processed analytical data to instructors directly in the CMS, reacting to early feedback, and growing the dataset and available reports.
We expect the funded contributions for Redis as a Message Bus and Google Analytics 4 Upgrade to be completed in this time frame, unblocking investigations into using the message bus as a delivery mechanism for analytics events. We will also begin our efforts to make those events more reliable with the Tracking Log Event Cleanup work.
Our high level Initiatives are available on the Open edX Roadmap’s Data tab, and epic-level task tracking happens in the Data Working Group project board.
Future Vision for the Group
...
Moving beyond the next six months, the {working_group_name} is really excited about making progress in {areas of interest}. Describe why these areas are interesting/important, and what their intended impact is. Include as much information as you wish here, but short and sweet is also OK.
Deep Dive: Accomplishment/In-progress Initiative/Ongoing Task/Concrete Plan (Optional)
Use this section to write or link to deep dives into any of your accomplishments or initiatives that would benefit the community to know more about. You can assume readers at this point will have the background knowledge to understand this being written in industry jargon. You may have as few or many “Deep Dive” links as you’d like. If you don’t have a place to post deep-dive content at the moment, consider posting them as blog posts on the Open edX site.
...
Initiative: Replace Insights with OARS, V1
...
...
Data Working Group is looking towards forming a cohesive xAPI profile for Open edX, growing our community’s data capabilities, and collaborating with other working groups on the foundational pieces of adaptive learning. With upcoming projects across the platform for a tagging and taxonomy system, modular learning, and standards-based learning traces in a data lake we will have a solid foundation to push the boundaries of how learner experiences can be tailored to improve educational outcomes!
Deep Dive: Open Analytics Reference System
The following links have details about the high level architectural decisions that have informed the OARS system:
For code-level details on the technical progress of the OARS system, the code lives in the following repositories:
openedx-oars - Documentation and project management, including architectural design docs.
tutor-contrib-superset - For running Apache Superset in Tutor.
tutor-contrib-clickhouse - For running the ClickHouse database in Tutor.
tutor-contrib-ralph - For running the Ralph learning events processor in Tutor.
tutor-contrib-oars - OARS-specific configuration for tying all of the above systems together.
Adds and configures event-routing-backends in the LMS to transform send xAPI statements to Ralph
Adds OAuth configuration so that Superset can authenticate users against the LMS and authorize which course data they have access to
Adds default datasets, dashboards, and charts to Superset
Configures ClickHouse database, tables, views, and users