Canvas - Learner Progress Tracking
- Matthew Harrington
Summary & Lessons Learned from Canvas
Canvas’ admin analytics are very pass/fail oriented, rather than progress, because unlike other LMSes where content is provided to be consumed, Canvas’ role is designed to augment in person classroom training. Their admin analytics in particular are interesting to me because it essentially feels more like they’re geared towards watching for underperforming teachers than anything. They report on whether teachers are using a variety of tools, whether teachers are providing content on the course, and whether their learners are performing in graded activities. They do have some features to help learners, but I get the distinct impression that it’s almost in order to catch which teachers are failing their learners from the way it’s taught to admins.
Is this what schools want? I can’t really speak to that. It does make sense though, given the context of K-12 classroom education, where particularly older teachers are resistant to introducing technology into their classroom.
Canvas’ tracking features are essentially very highly tailored to the classroom use case, as you might expect. They care about learners showing up and submitting graded activities, because that’s the primary point of Canvas in the classroom - to provide a place for graded activities to take place. There’s no measurement of assimilative content because it’s not necessary - the teaching is taking place somewhere else, for the most part. This means we can’t base the needs of online courses on Canvas, but it is a good case for the blended use case, and something to clearly understand and ask if we want to support - i.e. what happens if the bulk of the learning content isn’t actually on our site? Is that a mode we want to explicitly support, or not?
The “time spent active” issue is an interesting one. Canvas used to lean on this metric heavily, but in more recent dashboards it’s demoted in importance. There are various conversations about this metric, how it’s measured, how it’s sometimes very inaccurate, and other complaints about its lack of transparency. For example, at one point (I do not know is this is true today), the activity timer started only after 2 minutes of a learner viewing a page, and was not tied to any actions on the user’s part. This meant that sequences of short pages did not start the timer, and learners who left pages open for significant time while not actually engaging (such as looking at their phone) could run up the timer and appear to be active
I actually strongly believe that this metric is incredibly useful, but trying to implement it in such a way that it is reliable and trustworthy is a mammoth problem that I’m assuming Instructure just didn’t want to spend effort on solving.
It feels like Canvas expects teachers to handle the burden of explaining what masteries actually mean to learners, when it would’ve been pretty easy to add that to the UI. Making sure that things like grading and progress are clear to learners is crucially important for ensuring learner satisfaction.
Administrator Learner Progress Tracking
Canvas administrators can download CSV data dumps of a large amount of data from the reports area of the admin dashboard:
As mentioned in the reporting and analytics research doc (where you’ll find more reporting information), it’s unclear what exactly is in these CSVs, but at the very least we know that it can be used to see grades for all learners across all courses.
The student dashboard of the admin analytics dashboard contains a lot of data about the grades of students:
But not necessarily their progress through courses. The courses dashboard shows a variety of aggregate data about courses:
The most interesting for the purposes of progress tracking is the bottom half, where it displays courses with low average grades and low average student activity. Canvas essentially puts the focus at the admin level on grade output and participation - are learners taking part in the course? Are learners submitting late? Their admin dashboard essentially allows admins to track whether learners are disengaged or failing, rather than anything predictive such as whether learners are on track to pass, underperforming versus norms (other than a 70% pass rate), or behind in course content, because it’s assumed that all learners are attending in-person classes alongside their Canvas course.
Course Staff Learner Progress Tracking
In a similar vein, Canvas doesn’t provide much in the way of progress tracking in terms of “which content has the learner accessed out of available content” as that’s not what Canvas is for. Instead the focus is on two main areas:
Grade - How the learner has performed in graded tasks
This is measured by simple grade calculations and reports on all graded assignments, as well as charts and graphs displaying different visualisations of this data:
Participation - How much the learner has engaged with content in the course
This is measured in three main ways - page views, interactions, and literal time spent active in content
Page views are counted if the learner loads a page
Interactions are specific interactions that are recorded for specific activities, such as creating a wiki page, posting a message in discussions, and submitting an assignment
Time spent in content is very inaccurate, and it actually seems like this measure is being actively removed and demoted in importance by Canvas because of this. It was intended to record exactly how long learners spend studying in order to show exactly who is engaging fully or not.
Alongside this, much like Moodle, Canvas features a gradebook for all learner grades in all graded activities. What’s interesting about the gradebook is it has multiple modes to choose between.
The standard mode of the Canvas gradebook simply reports on grades in graded assignments:
The other mode it can be viewed in is the Learning Mastery gradebook:
This mode shows learner performance against mastery benchmarks (such as grades in specific grading categories) which are defined in the course settings, as well as the average performance of the cohort against those benchmarks, making it possible to see where each specific learner is performing below average, or where learners on the whole are performing above or below average.
The individual gradebook allows an instructor to hone in on a specific learner’s assignments and masteries for targeted interventions and further details about that learner. The individual gradebook is accessed through using extremely specific filters to help guide staff to specific entries, rather than presenting an overview:
Learner Progress Tracking
Learners in Canvas can access a list of their grades, which shows all assignments in the course:
If the course uses masteries, then their status on those masteries can also be viewed:
As masteries are somewhat unclear to learners, there’s information available on what the calculation methods actually mean:
But the explanations aren’t super clear, at least from the screenshots and documentation.
Other than these, there’s very little in the way of tracking course progress and completion other than dates and due date nudges, reminding learners when assignments are overdue, because once again - Canvas is primarily intended for instructor-led, classroom training, and anything outside of this use case is secondary.