Introduction
The Aspects learner analytics project is entering the Beta phase! The project has made a lot of progress since it kicked off as OARS at the beginning of 2023 and we’re ready to start gathering community feedback and exercise it in a wider variety of environments.
As a Beta, this project is not ready for production use yet. Large scale changes are still likely as we push toward a production ready v1, which might cause data loss and upgrade complications. We have not completed a full security or privacy pass at this time. We recommend testing this beta on a standalone private environment, especially if you are testing with production data. Use at your own risk! |
Found a bug or have a feature suggestion? We’d love to hear them. You can create a Github issue here, find us on the Open edX Slack in the #aspects channel, or email bmesick@axim.org.
|
Status
2023-12-20
Demo server is upgraded to v0.66.1. Diff from previous version is here. Notable changes:
2023-12-14
Demo server is upgraded to v0.66.0. Diff from previous version is here. Notable changes:
Real, actual translations are in. Set your language to fr_CA in the LMS and log back in to Superset to see
PII mapping tables exist for (optionally) including user profile data and mapping it to the anonymized xAPI actor ids (thanks Cristhian Garcia !)
Event Routing Backends now supports filters to override actors, verbs, and objects on events. This powerful functionality unlocks all manner of flexibility in xAPI event emission (thanks Andrey Cañon!)
Experimental instructor and operator dashboards from EduNEXT are being loaded, and will be available for feedback soon
We’ll be rolling the feedback into the default v1 dashboard designs
These include usage of the new PII data for location, gender etc. where it exists
2023-11-15
Demo server is upgraded to v0.63.0. Diff from previous version is here. Notable changes:
Operator dashboard now has a prototype of Superset introspection charts, allowing operators more visibility into how the system is being used as well as ClickHouse charts, with more details on important ClickHouse metrics.
Superset upgraded to 3.0.1 for a handful of bug fixes.
Translations pipeline is working from end-to-end, new translations welcome!
A substantial limitation in how Superset assets are handled in Kubernetes that limited how big assets could be was worked around.
Help text has been re-worked to allow operators to link out to help documentation per dashboard. By default this will point to the Aspects docs, but can be turned off entirely in configuration.
Fixes have been made to the way answers are sent by event-routing-backends for multi-select (checkbox) problems. This should allow those answers to display correctly in the “Answer Distribution” chart and is the first bug we’ve encountered in beta that will require tracking event replay to fix. 
2023-10-25
Demo server is upgraded to v0.57.3. Diff from previous version is here. There are a number of backend changes, but a couple of important user facing changes:
Beta feedback: XBlock names are now prepended with their section/subsection/unit ids and sorted by them, so that they should sort in the same order as they appear in the course.
Beta feedback: Problems that allow multiple answers (checkboxes) now display each possible answer individually instead of having one waaaay too long label for each permutation of selections. This should make the problem charts actually readable in those situations.
Fix: Course permissions caching got broken in the Superset 3.0 upgrade, when instructors are added or removed from courses the update should show up in Superset within a few minutes instead of… never. Thanks to Zachary Trabookis for finding this one!
Translations: Aspects translations have moved to openedx-translations and the Translations WG has been told that they are available to work on.
There are some outstanding issues with the translation pipeline due to it being on the brand new translation infrastructure, but it’s close.
Filter names aren’t currently translated, but that should be fixed in a few days.
Thanks to Pierre Mailhot for getting the fr_CA translations done within hours of source strings being available!
Dashboard embedding support is now built into Aspects. Currently we don’t implement it anywhere automatically, but EduNEXT is building a plugin on top of this functionality which embeds the Superset dashboard into Studio.
The course completion plugin has been upgraded in edx-platform and should start sending the events we need to power charts, but this didn’t land until after Quince was cut. We’re likely going to hold off implementing these charts until more instances are likely to have the data to support them, or we will have to hide them behind a feature flag.
2023-10-13
Demo server is upgraded to v0.53.0, which also has several non-user-facing changed but does add 2 new forum engagement charts to the instructor dashboard. Work is continuing on getting translations into the Open edX Transifex project, getting Completions updates merged to edx-platform, and various performance and bug fixes.
We also welcomed Chelsea Rathbun as a new Product Manager at Axim, initially focused on helping us get Aspects v1 nailed down and completed!
2023-09-22
Demo server is upgraded to tutor-contrib-aspects v0.45.0. We’ve had a number of non-user-facing changes this week, and a demo on Weds that I didn’t want to mess up, so a fair number of things have gone in. Most importantly a big upgrade to Superset 3.0.0! The biggest obvious change will be that there are all kinds of things happening with chart colors. We’ll be working on getting that settled down (and making the colors customizable along with the rest of the work on UI customization).
Superset 3 has a number of bug fixes for things that we’ve been working around (like the annoying filter bug that caused us to display empty charts until you selected a course / problem etc.). We’ll need to un-work-around those things before we’ll really see the benefits, but those changes should be coming in short order.
This release also includes the last set of plumbing unlocking forum reports. We’re going to take a stab at getting some forum data visible in the next few days.
There were numerous bug fixes, especially in the area of Kubernetes.
Experimental support for ClickHouse Cluster is also in, though very lightly tested. We’re hoping it will eventually allow for even the largest scale instances to use Aspects. If you are a large instance and want to take it for a spin with monumental amounts of data, let us know!
We’ve also merged new tracking log events for completions and login/logout, as well as the xAPI transforms for completions. Since these are entirely new events data will only start being collected for them once they are released in your edx-platform environment. We’re hoping to get them into the next Palm release.
2023-09-15
Demo server is upgraded to tutor-contrib-aspects v0.42.0. This version includes a workaround to the known Superset issues around filter selection. Currently some filters are not “required” (the little red asterisk) but charts will not show up until a course, problem, or video are selected. This prevents problems or videos from incorrect courses from showing up while we wait for a fix from Superset.
Preliminary help text has also been added in the Instructor Dashboard tabs, and descriptions for each chart are available from the “three dot” menu on the top right of the chart. Feedback on whether these are helpful and how we can improve usability is appreciated!
2023-09-13
Demo server is upgraded to tutor-contrib-aspects v0.40.1. This includes the first pass at grading charts! A number of new issues have been opened for v1, but an important one is adding localized chart descriptions and labels to describe how to interpret them. I hope to have the English version of those in this week, so we can start asking the Translations Working Group for help getting our strings translated!
2023-09-08
The demo server (see below) is up and accepting new registrations. It’s running v0.38.0 and loaded with the Demo Course and UMontreal’s ChatGPT101, generously shared with us by Vincent Laberge. We’re primarily looking to get people to interact with the courses and generate real course data to exercise the visualizations and help us iterate on them.
Roadmap
The beta process is expected to take several weeks as we find and fix bugs, field questions, improve docs and existing reports, and add the remaining reports that are considered necessary for the v1 release.
Open issues across the openedx Github organization tagged with aspects v1 are considered blocking the v1 release. This list will grow and shrink as we progress through the beta, but this is the best place to look to find which things are being worked on.
We recently received a draft list of questions from the Product team that will bring a variety of new reports into Aspects. Currently several tasks are in flight to add necessary data pipelines (tracking log events, xAPI transforms, and low level transformations in the database) to support some of those new reports. As we work with Product to define how best to present that information new tasks will be created.
Integrating instructor charts into the LMS is in scope for v1, but will likely be a funded effort managed by Product. It would be utilizing the same charts from the instructor dashboard, but displayed in-context in the course (only for instructors).
Upcoming reports
Report Name | Dashboard | Tracking Logs | xAPI | Charts |
|---|
Forum Engagement | Instructor | |
| |
Grading | Instructor | |
| |
Course Completion | Instructor | | |
|
Special Exams | Instructor |
|
|
|
User Devices (mobile app / web user agent) | Instructor |
|
|
|
Superset Usage | Operator | N/A | N/A |
|
ClickHouse Performance | Operator | N/A | N/A |
|
Demo Server
https://aspects-beta.openedx.org
https://superset.aspects-beta.openedx.org
We expect to be updating the demo server daily during weekdays (if there are any changes) throughout the beta. Student registrations are open to help populate course data, you shouldn’t need to email verify (no matter what the text says, the server doesn’t send mail).
The Superset instance for viewing reports is available if you have joined the beta (see below) and have an instructor and/or admin account. If you can log in there as a student, that’s a bug and you should let us know!
The server itself is an AWS EC2 `t3a.large` (2 vCPUs, 8 GiB RAM) that was actually upgraded from the OARS Demo instance from the 2023 Open edX Conference talk. It’s a Tutor local install.
Known Issues
Superset
Localization - many assets are not localized yet (and there are still likely to be some name changes). Some of these issues are in Superset, some are in Aspects, if you’d like to assist let us know! We’ll also be in contact with the Translations Working Group to ask for help.
Localization - Dashboards, charts, etc. are appended with their language code. This is a temporary development hack to help us identify which version of an asset we’re looking at and assist with translations.
Instructor dashboard - Problem and Video filters sometimes include unexpected problems and videos, including ones that the instructor shouldn’t have access to. This is currently mitigated by making users select options in the filters that aren’t “required”, which is not ideal. (bug)
Instructor dashboard - Some problems cause errors in the “Problem Performance” → “Distribution of Responses” chart. (bug)
Vector
xAPI transforms
Join the Beta
Want to help? Here are a few ways you can contribute:
Jump on the demo server and blast through one or both of the courses to help generate real world data for us to examine. Once you’ve done some coursework, contact us on Slack or email bmesick@axim.org to get permissions to view the reports!
Install and run Aspects locally to help us work out the bugs with installation, configuration, documentation, and customization. (New docs coming soon to help smooth this process!)
Try out migrating production data to test at scale. If you have an existing installation we’d love to work with you to try out the process of getting your historical data loaded into a test Aspects environment to get more testing done on that process and test real world performance at scale.
Help us out with translations! Docs coming, but we’re already in Transifex!
Review our code for security, privacy, and best practices. We’ve been moving fast for the last several months, and fresh eyes are welcome.