Introduction

The Aspects learner analytics project is entering the Beta phase! The project has made a lot of progress since it kicked off as OARS at the beginning of 2023 and we’re ready to start gathering community feedback and exercise it in a wider variety of environments.

Status

2023-12-20

Demo server is upgraded to v0.66.1. Diff from previous version is here. Notable changes:

• Bug fixes that allow us to show the dashboards EduNEXT have put together to best advantage:

  • EduNEXT Instructor Dash v1

  • EduNEXT Operator Dash v1

2023-12-14

Demo server is upgraded to v0.66.0. Diff from previous version is here. Notable changes:

• Real, actual translations are in. Set your language to fr_CA in the LMS and log back in to Superset to see

• PII mapping tables exist for (optionally) including user profile data and mapping it to the anonymized xAPI actor ids (thanks @Cristhian Garcia !)

  • PII data is hooked up to the LMS retirement pipeline, if you run retirments against the LMS user PII entries should be cleared in ClickHouse as part of that process (thanks @Jill Vogel !)

• Event Routing Backends now supports filters to override actors, verbs, and objects on events. This powerful functionality unlocks all manner of flexibility in xAPI event emission (thanks Andrey Cañon!)

• Experimental instructor and operator dashboards from EduNEXT are being loaded, and will be available for feedback soon

  • We’ll be rolling the feedback into the default v1 dashboard designs

  • These include usage of the new PII data for location, gender etc. where it exists

2023-11-15

Demo server is upgraded to v0.63.0. Diff from previous version is here. Notable changes:

• Operator dashboard now has a prototype of Superset introspection charts, allowing operators more visibility into how the system is being used as well as ClickHouse charts, with more details on important ClickHouse metrics.

• Superset upgraded to 3.0.1 for a handful of bug fixes.

• Translations pipeline is working from end-to-end, new translations welcome!

• A substantial limitation in how Superset assets are handled in Kubernetes that limited how big assets could be was worked around.

• Help text has been re-worked to allow operators to link out to help documentation per dashboard. By default this will point to the Aspects docs, but can be turned off entirely in configuration.

• Fixes have been made to the way answers are sent by event-routing-backends for multi-select (checkbox) problems. This should allow those answers to display correctly in the “Answer Distribution” chart and is the first bug we’ve encountered in beta that will require tracking event replay to fix.

  • Actually this change is in event-routing-backends, but didn’t make it to Aspects yet. Next time.

2023-10-25

Demo server is upgraded to v0.57.3. Diff from previous version is here. There are a number of backend changes, but a couple of important user facing changes:

• Beta feedback: XBlock names are now prepended with their section/subsection/unit ids and sorted by them, so that they should sort in the same order as they appear in the course.

• Beta feedback: Problems that allow multiple answers (checkboxes) now display each possible answer individually instead of having one waaaay too long label for each permutation of selections. This should make the problem charts actually readable in those situations.

  • This has caused a new known issue where certain problems are causing errors in Superset, we’re looking into it!

• Fix: Course permissions caching got broken in the Superset 3.0 upgrade, when instructors are added or removed from courses the update should show up in Superset within a few minutes instead of… never. Thanks to @Zachary Trabookis for finding this one!

• Translations: Aspects translations have moved to openedx-translations and the Translations WG has been told that they are available to work on.

  • There are some outstanding issues with the translation pipeline due to it being on the brand new translation infrastructure, but it’s close.

  • Filter names aren’t currently translated, but that should be fixed in a few days.

  • Thanks to @Pierre Mailhot for getting the fr_CA translations done within hours of source strings being available!

• Dashboard embedding support is now built into Aspects. Currently we don’t implement it anywhere automatically, but EduNEXT is building a plugin on top of this functionality which embeds the Superset dashboard into Studio.

  • This could also support things like embedding a dashboard in a private wiki or other internal web page.

• The course completion plugin has been upgraded in edx-platform and should start sending the events we need to power charts, but this didn’t land until after Quince was cut. We’re likely going to hold off implementing these charts until more instances are likely to have the data to support them, or we will have to hide them behind a feature flag.

2023-10-13

Demo server is upgraded to v0.53.0, which also has several non-user-facing changed but does add 2 new forum engagement charts to the instructor dashboard. Work is continuing on getting translations into the Open edX Transifex project, getting Completions updates merged to edx-platform, and various performance and bug fixes.

We also welcomed @Chelsea Rathbun as a new Product Manager at Axim, initially focused on helping us get Aspects v1 nailed down and completed!

2023-09-22

Demo server is upgraded to tutor-contrib-aspects v0.45.0. We’ve had a number of non-user-facing changes this week, and a demo on Weds that I didn’t want to mess up, so a fair number of things have gone in. Most importantly a big upgrade to Superset 3.0.0! The biggest obvious change will be that there are all kinds of things happening with chart colors. We’ll be working on getting that settled down (and making the colors customizable along with the rest of the work on UI customization).

Superset 3 has a number of bug fixes for things that we’ve been working around (like the annoying filter bug that caused us to display empty charts until you selected a course / problem etc.). We’ll need to un-work-around those things before we’ll really see the benefits, but those changes should be coming in short order.

This release also includes the last set of plumbing unlocking forum reports. We’re going to take a stab at getting some forum data visible in the next few days.

There were numerous bug fixes, especially in the area of Kubernetes.

Experimental support for ClickHouse Cluster is also in, though very lightly tested. We’re hoping it will eventually allow for even the largest scale instances to use Aspects. If you are a large instance and want to take it for a spin with monumental amounts of data, let us know!

We’ve also merged new tracking log events for completions and login/logout, as well as the xAPI transforms for completions. Since these are entirely new events data will only start being collected for them once they are released in your edx-platform environment. We’re hoping to get them into the next Palm release.

2023-09-15

Demo server is upgraded to tutor-contrib-aspects v0.42.0. This version includes a workaround to the known Superset issues around filter selection. Currently some filters are not “required” (the little red asterisk) but charts will not show up until a course, problem, or video are selected. This prevents problems or videos from incorrect courses from showing up while we wait for a fix from Superset.

Preliminary help text has also been added in the Instructor Dashboard tabs, and descriptions for each chart are available from the “three dot” menu on the top right of the chart. Feedback on whether these are helpful and how we can improve usability is appreciated!

2023-09-13

Demo server is upgraded to tutor-contrib-aspects v0.40.1. This includes the first pass at grading charts! A number of new issues have been opened for v1, but an important one is adding localized chart descriptions and labels to describe how to interpret them. I hope to have the English version of those in this week, so we can start asking the Translations Working Group for help getting our strings translated!

2023-09-08

The demo server (see below) is up and accepting new registrations. It’s running v0.38.0 and loaded with the Demo Course and UMontreal’s ChatGPT101, generously shared with us by @Vincent Laberge. We’re primarily looking to get people to interact with the courses and generate real course data to exercise the visualizations and help us iterate on them.

Roadmap

The beta process is expected to take several weeks as we find and fix bugs, field questions, improve docs and existing reports, and add the remaining reports that are considered necessary for the v1 release.

Open issues across the openedx Github organization tagged with aspects v1 are considered blocking the v1 release. This list will grow and shrink as we progress through the beta, but this is the best place to look to find which things are being worked on.

We recently received a draft list of questions from the Product team that will bring a variety of new reports into Aspects. Currently several tasks are in flight to add necessary data pipelines (tracking log events, xAPI transforms, and low level transformations in the database) to support some of those new reports. As we work with Product to define how best to present that information new tasks will be created.

Integrating instructor charts into the LMS is in scope for v1, but will likely be a funded effort managed by Product. It would be utilizing the same charts from the instructor dashboard, but displayed in-context in the course (only for instructors).

Upcoming reports

Demo Server

https://aspects-beta.openedx.org

https://superset.aspects-beta.openedx.org

We expect to be updating the demo server daily during weekdays (if there are any changes) throughout the beta. Student registrations are open to help populate course data, you shouldn’t need to email verify (no matter what the text says, the server doesn’t send mail).

The Superset instance for viewing reports is available if you have joined the beta (see below) and have an instructor and/or admin account. If you can log in there as a student, that’s a bug and you should let us know!

The server itself is an AWS EC2 `t3a.large` (2 vCPUs, 8 GiB RAM) that was actually upgraded from the OARS Demo instance from the 2023 Open edX Conference talk. It’s a Tutor local install.

Known Issues

Superset

• Localization - many assets are not localized yet (and there are still likely to be some name changes). Some of these issues are in Superset, some are in Aspects, if you’d like to assist let us know! We’ll also be in contact with the Translations Working Group to ask for help.

• Localization - Dashboards, charts, etc. are appended with their language code. This is a temporary development hack to help us identify which version of an asset we’re looking at and assist with translations.

• Instructor dashboard - Problem and Video filters sometimes include unexpected problems and videos, including ones that the instructor shouldn’t have access to. This is currently mitigated by making users select options in the filters that aren’t “required”, which is not ideal. (bug)

• Instructor dashboard - Some problems cause errors in the “Problem Performance” → “Distribution of Responses” chart. (bug)

Vector

• Fixed: There is a known issue with setting up the Vector-based workflow in Kubernetes. (bug)

xAPI transforms

• Fixed: Multi-question responses do not send all required data for all questions. (bug, WIP PR)

Join the Beta

Want to help? Here are a few ways you can contribute:

• Jump on the demo server and blast through one or both of the courses to help generate real world data for us to examine. Once you’ve done some coursework, contact us on Slack or email bmesick@axim.org to get permissions to view the reports!

• Install and run Aspects locally to help us work out the bugs with installation, configuration, documentation, and customization. (New docs coming soon to help smooth this process!)

• Try out migrating production data to test at scale. If you have an existing installation we’d love to work with you to try out the process of getting your historical data loaded into a test Aspects environment to get more testing done on that process and test real world performance at scale.

• Help us out with translations! Docs coming, but we’re already in Transifex!

• Review our code for security, privacy, and best practices. We’ve been moving fast for the last several months, and fresh eyes are welcome.