In-Context Metrics for Studio: Product Requirements
- Chelsea Rathbun
Overview
Course delivery teams are busy and often don’t want to have to navigate to a separate full-page dashboard to view metrics for their courses or their content. In an effort to equip users with content engagement and performance metrics where they edit and view their course content, we’ll be adding easy-to-access, easy-to-collapse metrics in context, right in Studio.
Requirements
Developed as a frontend plugin
In-context metrics will be developed utilizing frontend plugin framework by creating a frontend plugin that can be removed, customized, or used by any instance, and frontend plugin slots so the plugin can be rendered in appropriate places on the page(s) that can be removed, customized, or used by any instance (even those that don’t have Aspects)
For instances that have Aspects, in-context metrics are enabled by default. Operators can choose to remove or customize the plugin on their instance.
For instances that don’t have Aspects, what does this plugin look like? It is empty by default.
The plugin slot can be stacked if multiple plugins try to use it (it would be possible for someone to use this slot to display Aspects in-context metrics alongside an LLM summarization plugin, for example)
Overarching functionality
On instances where in-context metrics are enabled:
When enabled, the following django permissions and course roles can view in-context metrics:
Users with superuser or staff django permissions on their instance can view in-context metrics
Active (django) users can view in-context metrics for courses where they are listed as any of the following course staff roles: Course Admin, Course Staff
Active (django) users cannot view in-context metrics for courses where their listed as any of the following course staff roles (as their only role in the course): Limited Staff, Beta Tester, Discussion Admin, Discussion Moderator, Group Community TA, Community TA, Course Data Researcher
Inactive (django) users cannot view in-context metrics in Studio
It must be clear in the UI to Studio users that in-context metrics are available for them to view (when enabled). The in-context metrics are to be hidden by default, but easily accessed and re-hidden by the user.
When visible, the user must be able to view the in-context metrics next to/by the content the metrics relate to.
When disabled or hidden, there are no calls made to Superset and the UI for this is unobtrusive (while also being clear)
Default metrics that are shown must be tailored to the type of content displayed (See specific requirements in the next section).
If a course is re-run or exported, the course metrics are copied over from the copied run. By default, the new course run does not show the copied course metrics, but users are able to view historic (or copied run) in-context metrics if they choose.
The visuals can be viewed without having to zoom or scroll horizontally (the sidebar is wide enough to clearly display visualizations in-full with legible labels, etc)
It should be possible to the default override in-context metrics with custom ones
It should be possible to add new in-context metrics for custom xblock types.
Requirements by content type
Per component, I want to see the following:
For video components:
I want to see where (what timestamps) in the video my learners get to
I want to see where (what timestamps) in the video are being watched multiple times
For single problem components:
I want to see the % correct problem responses (out of all learner submissions, not first attempt or last attempt)
Initial problem response distribution
For single select, multi-select, dropdown, I would imagine this could be a bar graph showing the number of learners that submitted each response as their initial problem response
For text input, we’ll want to show the 7 (if this looks too crunched, we can decrease) most common responses and indicate the number of learners that submitted each response as their initial problem response. If possible, it might be good to give users a way to see more than the top 7 responses either by clicking a view more button, scrolling through the graph, etc.
For numerical input or numerical input, we’ll want to show the responses and indicate the number of learners that submitted each range of response as their initial problem response (I could imagine these being a scatter plot, density plot, or box plot or even a histogram with bars as ranges. Not sure if this is possible with numerical inputs).
For graded subsections
I want to see the average grade on the whole graded subsection among those who submitted at least one response to a problem in the graded subsection.
Note: The grade is based on the learner’s last attempt for each problem. You can configure problems in graded subsections to accept a set number or even unlimited attempts.
I want to see a distribution of the number of learners who attempted each problem in the graded subsection. Can this distribution be a stacked bar to show breakdown of correct/incorrect final responses?