Data Use Cases - Raw Market Feedback
- Jenna Makowski
- Jill Vogel
A space to aggregate and share various use cases associated with data, reporting and analytics needs. Examples might include things like:
User story/persona, ie “As a course operator I want to build a report that includes X, Y, Z…”
Problem/challenge to address: “As a faculty member teaching a blended learning course, I need predictive analytics to help me identify at-risk learners etc etc”
Here is a sortable and filter-able table view of the problem statements aggregated below, with corresponding info about persona and focus area: https://docs.google.com/spreadsheets/d/1NqhhP8FzAz_S0PgWBxrs2QNmCfF75DPyfxvV6s5vTno/edit#gid=0
Need/limitation/problem | Persona | Keywords/Focus area | Source |
Need more nuanced permissions control and security levels. For example currently can’t give instructors access to analytics without also giving them access to Studio, but they don’t want their instructors all having access to Studio. | course admin | access, permissions, security | WGU |
Row level security: the capability to limit access to certain rows (records) to users, based on the content of a field (e.g., allow each teacher to access only their courses) | course admin | access, permissions, security | Aulasneo customers |
Column level security: restrict access to certain fields to users (e.g., course price) | course admin | access, permissions, security | Aulasneo customers |
Analysis of all courses, and all learners, at the same time, is the goal | educator, course admin | aggregate data | HMS |
Initiative for aggregated statistics- aggregated across multiple courses rather than course-level | educator, course admin | aggregate data | UC3M |
As a site admin, I want to compare figures between courses, organizations, editions | course admin | aggregate data | Aulasneo customers |
Generating real-time learner experience pathways based on assessments and other feedback loops | learner | learner engagement data | WHO |
Need both static and dynamic attributes to determine what content students see. Need deep event engagement data to answers questions like - is this student taking a long time because they are struggling, or because they are distracted? How many videos have they watched? How long were they engaged with the video? | educator, instructional designer | learner engagement data | WGU |
What is the correctness delta between the first assessment attempt and the second attempt? | educator, instructional designer | learner engagement data - assessments | HMS |
Content reuse data - -descriptive and dynamic metadata (dynamic like time on task and difficulty it gives students). Dynamic data examples: atomic time - how much time did a student spend on a problem. Molecular time - additional time needed to reference textbooks. The ideal would be to measure learning per unit time. | educator, instructional designer | learner engagement data - assessments | MIT |
What was the learner page view path through their course? Is there a theme, are pages frequently revisited, and which page has the longest dwell time? | educator, instructional designer | learner engagement data - page data | HMS |
How many total seconds were spent viewing a page? Basically, any learner or administrative activity in the logs is within the scope of our research. | educator, instructional designer | learner engagement data - page data | HMS |
Page views and amount of time spend per page (total seconds) | educator, course admin | learner engagement data - page data | Penn State |
As a teacher, I want to see in which units do students spend more time | educator/course admin | learner engagement data - page data | Aulasneo customers |
“We want to look at a course with 10k people and be able to say: In XYZ year, this is how many people engaged with this page and this is how much time they spent” | educator, course admin | learner engagement data - page data | Penn State |
Predictive reports/analytics to identify at-risk students. | educator, course admin | learner engagement data - predictive | MIT |
Predictive analytics | educator, instructional designer | learner engagement data - predictive | UC3M |
As a teacher, I want to see how many videos have been watched, out of the units that were released in the last period (e.g. week) | educator | learner engagement data - video | Aulasneo customers |
As a course author, I want to see which videos were most seen, which were seen at double speed, which were replayed, which segment of the video was most seen, replayed and skipped | educator | learner engagement data - video | Aulasneo customers |
Their need is to to connect in-person and online experiences into one cohesive learning experience. Use case: Learner completes a 5-hour online training course on landscaping, then has their pruning skills assessed in-person. Need 1) Seamless purchase of a blended learning experience; 2) Tie the off-campus event to the platform |
| offline events | Penn State |
Data and analysis to track progress of cohorts/groups, especially for blended contexts | educator, course admin | progress data (cohort/group) | anonymized |
Individual data - for in-person/live courses. An instructor/educator needs to understand which individual is doing what and their individual progress in real-time | educator, course admin | progress data (individual) | anonymized |
Easier ways to track individual student progress | educator, course admin | progress data (individual) | UBx |
As a site admin, I want to track all activities of a specific student | course admin | progress data (individual) | Aulasneo customers |
As a teacher, I need a progress report for each student | educator, course admin | progress data (individual) | Aulasneo customers |
Real-time progress reports at individual level - both at progress-across-the-course level and at the individual problem/assessment level | educator, instructional designer | real time progress data (individual) | anonymized |
Need to know who is where and what they are doing at all times - better tracking of progress at individual student level for engagement | educator, course admin | real time progress data (individual) | WGU |
Real-time, in-context analytics | educator, course admin | real time progress data (individual) | UC3M |
“Quick views” for educators. For educators, quick insights into #enrollments, courses by enrollment, courses by completion, quick stats by course etc | educator, course admin | reporting | anonymized |
"Quick view" for learners. For learners, courses enrolled, progress by course, completions | learner | reporting | anonymized |
Better visualization of real-time data | educator, course admin | reporting | anonymized |
An important need for enterprise instances is to be able to align their enterprise training to internal KPIs |
| reporting | anonymized |
As a student, I want to see the courses I’m enrolled | learner | reporting | Aulasneo customers |
As a student, I want to see the courses I’ve completed and my grade | learner | reporting | Aulasneo customers |
As a teacher, in courses with zoom live sessions, I need an attendance report per student for each session | educator | reporting | Aulasneo customers |
Make it easy to export data into third-party business analytics tools - tie to google analytics for example | educator, course admin | reporting - exporting | anonymized |
Easy integration of data - particularly grades/outcomes - to campuswide LMS | educator, course admin | reporting - exporting | UBx |
Challenge is integration - getting accurate course completion data. Need a more elegant way to define course completion. The only way around their integration setup is to have students complete a yes/no quiz saying they completed the course. They rely on the course completions to report out CEUs or credential. They use the completion counts to measure impact of programs and if no CEU attached, there’s not a lot of motivation to complete the quiz. | course admin | reporting - exporting | Penn State |
As a teacher, I want to see how many activities have been completed, out of the units that were released in the last period (e.g. week) | educator | reporting - student activity | Aulasneo customers |
Daily reports of student activity from the day prior. | educator | reporting - student activity | MIT |
As a teacher, I want to see how much activity is happening in the forums, out of the units that were released in the last period (e.g. week) | educator/course admin | reporting - student activity | Aulasneo customers |
@Matt Bunch at HMS shared the following data analysis questions/topics:
Analysis of all courses, and all learners, at the same time, is the goal.
What is the correctness delta between the first assessment attempt and the second attempt?
What was the learner page view path through their course?
Is there a theme, are pages frequently revisited, and which page has the longest dwell time?
How many total seconds were spent viewing a page?
Basically, any learner or administrative activity in the logs is within the scope of our research.
Jenna’s compiled use cases - no particular order
Anonymized sources:
Real-time progress reports at individual level - both at progress-across-the-course level and at the individual problem/assessment level
Individual data - for in-person/live courses. An instructor/educator needs to understand which individual is doing what and their individual progress in real-time
Data and analysis to track progress of cohorts/groups, especially for blended contexts
“Quick views” for educators and learners. For educators, quick insights into #enrollments, courses by enrollment, courses by completion, quick stats by course etc. For learners, courses enrolled, progress by course, completions
[Reporting] Better visualization of real-time data
[Reporting] Make it easy to export data into third-party business analytics tools - tie to google analytics for example
[Reporting] An important need for enterprise instances is to be able to align their enterprise training to internal KPIs
Specific use cases:
UBx: Want easier ways to track individual student progress
UBx: Want easy integration of data - particularly grades/outcomes - to campuswide LMS (canvas): Use case - the way faculty use UBx on campus is to assign homework, assignments, etc with particular learning objectives like “learn python basics”. They want assessment results to flow seamlessly into their central LMS. “Can I take a quiz on Open edX for my Math 309 course, get the grades, and have the grades flow into Brightspace?” He sees this as a huge incentive for getting other departments using UBx more broadly for blended learning initiatives
Penn State: Page views and amount of time spend per page (total seconds). Also meaningful ways to conglomerate/report. “We want to look at a course with 10k people and be able to say: In XYZ year, this is how many people engaged with this page and this is how much time they spent”
Penn State: Challenge is integration - getting accurate course completion data. Need a more elegant way to define course completion. The only way around their integration setup is to have students complete a yes/no quiz saying they completed the course. They rely on the course completions to report out CEUs or credential. They use the completion counts to measure impact of programs and if no CEU attached, there’s not a lot of motivation to complete the quiz.
Penn State: Their need is to to connect in-person and online experiences into one cohesive learning experience. Use case: Learner completes a 5-hour online training course on landscaping, then has their pruning skills assessed in-person. Need 1) Seamless purchase of a blended learning experience; 2) Tie the off-campus event to the platform
WGU: Need to know who is where and what they are doing at all times - better tracking of progress at individual student level for engagement
WGU: [RBAC/permissions] Need more nuanced permissions control and security levels. For example currently can’t give instructors access to analytics without also giving them access to Studio, but they don’t want their instructors all having access to Studio.
WGU: Data for adaptive learning contexts: Need both static and dynamic attributes to determine what content students see. Need deep event engagement data to answers questions like - is this student taking a long time because they are struggling, or because they are distracted? How many videos have they watched? How long were they engaged with the video?
WHO: Adaptive learning branching - generating real-time learner experience pathways based on assessments and other feedback loops
MIT Blended: Predictive reports/analytics to identify at-risk students. Daily reports of student activity from the day prior.
MIT Blended: Content reuse data - -descriptive and dynamic metadata (dynamic like time on task and difficulty it gives students). Dynamic data examples: atomic time - how much time did a student spend on a problem. Molecular time - additional time needed to reference textbooks. The ideal would be to measure learning per unit time.
Spanish consortium: Initiative for aggregated statistics- aggregated across multiple courses rather than course-level
Spanish consortium: Real-time, in-context analytics
Spanish consortium: Predictive analytics