Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

A space to aggregate and share various use cases associated with data, reporting and analytics needs. Examples might include things like:

  • User story/persona, ie “As a course operator I want to build a report that includes X, Y, Z…”

  • Problem/challenge to address: “As a faculty member teaching a blended learning course, I need predictive analytics to help me identify at-risk learners etc etc”

...

Here is a sortable and filter-able table view of the problem statements aggregated below, with corresponding info about persona and focus area: https://docs.google.com/spreadsheets/d/1NqhhP8FzAz_S0PgWBxrs2QNmCfF75DPyfxvV6s5vTno/edit#gid=0

Need/limitation/problem

Persona

Keywords/Focus area

Source

Need more nuanced permissions control and security levels. For example currently can’t give instructors access to analytics without also giving them access to Studio, but they don’t want their instructors all having access to Studio.

course admin

access, permissions, security

WGU

Row level security: the capability to limit access to certain rows (records) to users, based on the content of a field (e.g., allow each teacher to access only their courses)

course admin

access, permissions, security

Aulasneo customers

Column level security: restrict access to certain fields to users (e.g., course price)

course admin

access, permissions, security

Aulasneo customers

Analysis of all courses, and all learners, at the same time, is the goal

educator, course admin

aggregate data

HMS

Initiative for aggregated statistics- aggregated across multiple courses rather than course-level

educator, course admin

aggregate data

UC3M

As a site admin, I want to compare figures between courses, organizations, editions

course admin

aggregate data

Aulasneo customers

Generating real-time learner experience pathways based on assessments and other feedback loops

learner

learner engagement data

WHO

Need both static and dynamic attributes to determine what content students see. Need deep event engagement data to answers questions like - is this student taking a long time because they are struggling, or because they are distracted? How many videos have they watched? How long were they engaged with the video?

educator, instructional designer

learner engagement data

WGU

What is the correctness delta between the first assessment attempt and the second attempt?

educator, instructional designer

learner engagement data - assessments

HMS

Content reuse data - -descriptive and dynamic metadata (dynamic like time on task and difficulty it gives students). Dynamic data examples: atomic time - how much time did a student spend on a problem. Molecular time - additional time needed to reference textbooks. The ideal would be to measure learning per unit time.

educator, instructional designer

learner engagement data - assessments

MIT

What was the learner page view path through their course? Is there a theme, are pages frequently revisited, and which page has the longest dwell time?

educator, instructional designer

learner engagement data - page data

HMS

How many total seconds were spent viewing a page? Basically, any learner or administrative activity in the logs is within the scope of our research.

educator, instructional designer

learner engagement data - page data

HMS

Page views and amount of time spend per page (total seconds)

educator, course admin

learner engagement data - page data

Penn State

As a teacher, I want to see in which units do students spend more time

educator/course admin

learner engagement data - page data

Aulasneo customers

“We want to look at a course with 10k people and be able to say: In XYZ year, this is how many people engaged with this page and this is how much time they spent”

educator, course admin

learner engagement data - page data

Penn State

Predictive reports/analytics to identify at-risk students.

educator, course admin

learner engagement data - predictive

MIT

Predictive analytics

educator, instructional designer

learner engagement data - predictive

UC3M

As a teacher, I want to see how many videos have been watched, out of the units that were released in the last period (e.g. week)

educator

learner engagement data - video

Aulasneo customers

As a course author, I want to see which videos were most seen, which were seen at double speed, which were replayed, which segment of the video was most seen, replayed and skipped

educator

learner engagement data - video

Aulasneo customers

Their need is to to connect in-person and online experiences into one cohesive learning experience. Use case: Learner completes a 5-hour online training course on landscaping, then has their pruning skills assessed in-person. Need 1) Seamless purchase of a blended learning experience; 2) Tie the off-campus event to the platform

offline events

Penn State

Data and analysis to track progress of cohorts/groups, especially for blended contexts

educator, course admin

progress data (cohort/group)

anonymized

Individual data - for in-person/live courses. An instructor/educator needs to understand which individual is doing what and their individual progress in real-time

educator, course admin

progress data (individual)

anonymized

Easier ways to track individual student progress

educator, course admin

progress data (individual)

UBx

As a site admin, I want to track all activities of a specific student

course admin

progress data (individual)

Aulasneo customers

As a teacher, I need a progress report for each student

educator, course admin

progress data (individual)

Aulasneo customers

Real-time progress reports at individual level - both at progress-across-the-course level and at the individual problem/assessment level

educator, instructional designer

real time progress data (individual)

anonymized

Need to know who is where and what they are doing at all times - better tracking of progress at individual student level for engagement

educator, course admin

real time progress data (individual)

WGU

Real-time, in-context analytics

educator, course admin

real time progress data (individual)

UC3M

“Quick views” for educators. For educators, quick insights into #enrollments, courses by enrollment, courses by completion, quick stats by course etc

educator, course admin

reporting

anonymized

"Quick view" for learners. For learners, courses enrolled, progress by course, completions

learner

reporting

anonymized

Better visualization of real-time data

educator, course admin

reporting

anonymized

An important need for enterprise instances is to be able to align their enterprise training to internal KPIs

reporting

anonymized

As a student, I want to see the courses I’m enrolled

learner

reporting

Aulasneo customers

As a student, I want to see the courses I’ve completed and my grade

learner

reporting

Aulasneo customers

As a teacher, in courses with zoom live sessions, I need an attendance report per student for each session

educator

reporting

Aulasneo customers

Make it easy to export data into third-party business analytics tools - tie to google analytics for example

educator, course admin

reporting - exporting

anonymized

Easy integration of data - particularly grades/outcomes - to campuswide LMS

educator, course admin

reporting - exporting

UBx

Challenge is integration - getting accurate course completion data. Need a more elegant way to define course completion. The only way around their integration setup is to have students complete a yes/no quiz saying they completed the course. They rely on the course completions to report out CEUs or credential. They use the completion counts to measure impact of programs and if no CEU attached, there’s not a lot of motivation to complete the quiz.

course admin

reporting - exporting

Penn State

As a teacher, I want to see how many activities have been completed, out of the units that were released in the last period (e.g. week)

educator

reporting - student activity

Aulasneo customers

Daily reports of student activity from the day prior.

educator

reporting - student activity

MIT

As a teacher, I want to see how much activity is happening in the forums, out of the units that were released in the last period (e.g. week)

educator/course admin

reporting - student activity

Aulasneo customers

...

Matt Bunch at HMS shared the following data analysis questions/topics:

  • Analysis of all courses, and all learners, at the same time, is the goal.

  • What is the correctness delta between the first assessment attempt and the second attempt?

  • What was the learner page view path through their course?

    • Is there a theme, are pages frequently revisited, and which page has the longest dwell time?

  • How many total seconds were spent viewing a page?
    Basically, any learner or administrative activity in the logs is within the scope of our research.

...

Jenna’s compiled use cases - no particular order

Anonymized sources:

  • Real-time progress reports at individual level - both at progress-across-the-course level and at the individual problem/assessment level

  • Individual data - for in-person/live courses. An instructor/educator needs to understand which individual is doing what and their individual progress in real-time

  • Data and analysis to track progress of cohorts/groups, especially for blended contexts

  • “Quick views” for educators and learners. For educators, quick insights into #enrollments, courses by enrollment, courses by completion, quick stats by course etc. For learners, courses enrolled, progress by course, completions

  • [Reporting] Better visualization of real-time data

  • [Reporting] Make it easy to export data into third-party business analytics tools - tie to google analytics for example

  • [Reporting] An important need for enterprise instances is to be able to align their enterprise training to internal KPIs

Specific use cases:

  • UBx: Want easier ways to track individual student progress

  • UBx: Want easy integration of data - particularly grades/outcomes - to campuswide LMS (canvas): Use case - the way faculty use UBx on campus is to assign homework, assignments, etc with particular learning objectives like “learn python basics”. They want assessment results to flow seamlessly into their central LMS. “Can I take a quiz on Open edX for my Math 309 course, get the grades, and have the grades flow into Brightspace?” He sees this as a huge incentive for getting other departments using UBx more broadly for blended learning initiatives

  • Penn State: Page views and amount of time spend per page (total seconds). Also meaningful ways to conglomerate/report. “We want to look at a course with 10k people and be able to say: In XYZ year, this is how many people engaged with this page and this is how much time they spent”

  • Penn State: Challenge is integration - getting accurate course completion data. Need a more elegant way to define course completion. The only way around their integration setup is to have students complete a yes/no quiz saying they completed the course. They rely on the course completions to report out CEUs or credential. They use the completion counts to measure impact of programs and if no CEU attached, there’s not a lot of motivation to complete the quiz. 

  • Penn State: Their need is to to connect in-person and online experiences into one cohesive learning experience. Use case: Learner completes a 5-hour online training course on landscaping, then has their pruning skills assessed in-person. Need 1) Seamless purchase of a blended learning experience; 2) Tie the off-campus event to the platform

  • WGU: Need to know who is where and what they are doing at all times - better tracking of progress at individual student level for engagement

  • WGU: [RBAC/permissions] Need more nuanced permissions control and security levels. For example currently can’t give instructors access to analytics without also giving them access to Studio, but they don’t want their instructors all having access to Studio.

  • WGU: Data for adaptive learning contexts: Need both static and dynamic attributes to determine what content students see. Need deep event engagement data to answers questions like - is this student taking a long time because they are struggling, or because they are distracted? How many videos have they watched? How long were they engaged with the video?

  • WHO: Adaptive learning branching - generating real-time learner experience pathways based on assessments and other feedback loops

  • MIT Blended: Predictive reports/analytics to identify at-risk students. Daily reports of student activity from the day prior.

  • MIT Blended: Content reuse data - -descriptive and dynamic metadata (dynamic like time on task and difficulty it gives students). Dynamic data examples: atomic time - how much time did a student spend on a problem. Molecular time - additional time needed to reference textbooks. The ideal would be to measure learning per unit time.

  • Spanish consortium: Initiative for aggregated statistics- aggregated across multiple courses rather than course-level

  • Spanish consortium: Real-time, in-context analytics

  • Spanish consortium: Predictive analytics

...