Overview

This document outlines the product specs for a V1 release of Aspects. The goal of the V1 release is to deliver an MVP experience for instructional designers and course teams to access a curated set of reports and visualizations needed for analyzing course-level enrollment, engagement, and performance data all in one place and easily from the Open edX platform. Aspects will replace existing and soon-to-be-deprecated Insights with the aim of delivering a user-friendly analytics solution that offers near real-time data, and is modern, secure, easy to set up, less expensive to run, and much more extensible than the existing product.

Because an Operator dashboard was developed for the Aspects beta in Superset that provides information in instance health, instance-level enrollment, courses, and organizations, we will include this dashboard with its current reports in this V1 release in order to ensure that instance administrators can access critical data while further discovery work is conducted to expand and improve these dashboards in a future release.

A separate corresponding document covers more details on the technical approach to this release.

Key Use Cases

To provide focus for this initial release, we will hone in on common use cases discovered through user interviews that are outlined below. These use cases fall into three themes: enrollment, engagement, and performance. In addition to course-wide enrollment, engagement, and performance metrics, we will introduce a few reports that allow users to look at data for individual learners or for a group of learners that meet certain criteria. 

Course Dashboard

Enrollment

As a course author, I want to…

• See the number of total learners enrolled in my course, so that I know how many students have signed up for the course to date.

• See the number and percentage of total enrollees, current enrollees, and enrollees per enrollment track available on my Open edX instance, so that I can easily compare enrollment breakdown from one course to another.

• See the rate of enrollment for my course plotted over time, so that I can easily identify periods of increased enrollment with the goal of understanding where learners are coming from. 

• View the change in number of enrollees over a time period of my choosing including their enrollment track, so that I have more information to identify when learners heard about my course, why they chose the enrollment track they did, and what changes occurred to course enrollment over a time period of my interest.

Learner Engagement

As a course author, I want to…

• See how many learners stopped engaging with viewing course content, watching course videos, and attempting course problems for the course run or a time period of my choosing, so I can identify where the most learners are dropping off in my course. 

  • Opened at least one page for each section/subsection of the course versus how many learners opened all pages in each section/subsection

  • Viewed at least part of at least one video for each section/subsection of the course versus how many learners viewed at least part of all videos in each section/subsection

  • Attempted at least one problem for each section/subsection of the course versus how many learners attempted all problems in each section/subsection

• See how many learners watched each video in my course in full or in part, so that I know which videos my learners watched.

• See how many times each section of each video was watched, so that I’m able to identify any sections of the video that are either unclear or especially significant to learners.

Performance

As a course author, I want to…

Learner Performance

• See the number who have a passing grade or more in my course, so that I have grading information easily accessible alongside other performance metrics.

• See the distribution of current course grade for all learners in my course, so that I have a sense of how learners are performing in my course alongside the rest of the course metrics.

Content Performance

• See the number of learners who attempted each assignment and the percentage of correct versus incorrect answers submitted for each assignment, so that I know where learners are in my course and how they’re performing.

• See the number of learners who attempted each graded and ungraded problem, so that I can see what parts of the course learners are engaging with.

• See the percentage of correct versus incorrect answers submitted for each graded and ungraded problem, so that I can see how learners are performing in the course and diagnose issues with specific problems.

• See the average number of attempts taken for each graded and ungraded problem, so that I can diagnose issues with specific problems and better gauge problem difficulty level. 

Learner Groups and Individual Learner Data

As a course author, I want to view the metrics outlined below for a group of learners (made up of a subset of learners that meet certain criteria) or for an individual learner, so that I’m able to learn more about specific learner’s/learners' engagement, performance, progress, and contact information:

For this V1 release, we will focus on providing one out-of-the box learner group designed to surface enrollment, engagement, and performance information for learners that may be at risk due to low performance and/or engagement in the course. The defining criteria for these two learner groups are defined below:

• Learners who…

  • have enrolled in the course

  • AND have not visited the course in 7 or more days

  • AND have not yet passed the course

  • AND who have visited a page other than the course homepage OR watched a video OR answered a graded or ungraded problem OR posted in the course discussion forum

Deliverables

• Course Dashboard in Superset

• Means by which course authors and instructional designers can securely navigate to the dashboard from the Open edX LMS

• Learner Group/Individual Learner Dashboard in Superset

• Operator Dashboard in Superset (this is the same dashboard that currently exists in Aspects beta with no adjustments - future adjustments will be informed by discovery efforts and tackled in a future release)

Users

The primary users of this initial V1 release of Aspects will be course authors, instructional designers, and course teams that want access to quick, descriptive metrics and visualizations regarding the enrollment, engagement, and performance of learners in a course. We realize instance admins, program administrators, and course authors will also need access to whole-instance and cross-course statistics as well, but these metrics require further discovery and will be the focus of a subsequent release. Because an Operator Dashboard has already been developed as part of the work to create Aspects beta, this dashboard will be included in this v1 release with the caveat that we may expand/edit this dashboard in future releases based on additional discovery work.

In Scope/ Out of Scope

The following table outlines what is in scope and out of scope for this V1 release.

MVP Specs

In order to deliver all components of the Aspects V1 release, the following features will need to be completed:

Nonfunctional Requirements

• Can support instances up to 2M registered learners

• Must be supported in the 19 languages currently supported by the Translations Working Group

  • Notes on language support in Superset:

    • Superset currently only supports the languages listed here.

    • Superset currently does not support right-to-left languages, which will impact a few of the languages for which we have translators. For example: Arabic and Hebrew.

Technical Open Questions

1. Can we limit access to select data visualizations depending on a user’s role? Specifically thinking of Course Editors and Course Authors roles, which may need access to information about content that may need to be revised or updated (an example: videos where specific sections are unclear to users or problems that are not performing well because they may be unclear).

Future Direction

1. Beyond the initial V1 release, further discovery will be conducted to improve the existing whole instance (Operator Dashboard) and inform cross-course reporting and visualizations for program administrators, course authors, and instance administrators.

2. Further improvements to what was delivered in this V1 Aspects release will be considered. Additional possible areas for exploration include:

   1. Embedding dashboard metrics into the Open edX LMS (this may be folded into V1, if possible)

   2. Allowing users to configure their own learner groups

   3. Making it easier to compare individual and learner group metrics against course-wide metrics

   4. Making it easy to contact to a specific group of learners (although this will likely involve parts of the Open edX platform outside of Aspects)

   5. If we find during user testing or receive feedback after releasing that users often find themselves downloading multiple CSV files and combining information from multiple reports to better analyze their course data, we may explore providing more streamlined reports for download that capture the most commonly occurring reporting use cases.

   6. Incorporating demographics data and visualizations use cases that could not achieved in V1

      1. Specific use cases related to demographics information discovered through user interviews are detailed below:

         1. Course Enrollment:

            1. As a course author, I want to see the number and percentage of enrollees per category of the following demographics: learner country (top 7 most occurring, rest), gender, age, education level, so that I have a better sense of who my learners are.

            2. As a course author, I want to view the learner country in the change in number of enrollees over time visualization, so that I have more information to identify where learners heard about my course over a certain time period.

         2. Learner Groups: Providing a visualization that breaks down the following demographic information for the learner group: Geographic location (Country only), Gender, Educational Background, Age (only to be shown if the group is 100 or more learners)

      2. Allowing course authors to see the number of unique learners who clicked on each link to an external resource in my course and the number of times the link was clicked (if applicable), so that they know if learners are using these resources (an initial requirement for V1 that needed to be removed from the initial release due to broken functionality introduced in the move to MFE)