Canvas - Course-level Analytics and Reporting

Canvas has recently (since at least 2020) moved to using an LTI tool for in-course analytics that Instructure refers to as “New Analytics”, with a goal of deprecating the old analytics tools. Interestingly, as they went with an LTI tool for this, it has all the limitations of functioning as an LTI tool, such as requiring third-party cookies and scripts. 

New analytics is an extremely comprehensive suite of course analytics and reporting tools including data tables, interactive charts and other user-friendly visualisations. It updates every 24 hours, but “Course Activity Report data may be delayed by 40 hours”, whatever that means. Presumably they have some technical issue that slows down that data.

The New Analytics dashboard has 5 tabs -

• Course grade - For plotting the performance of learners across the sections of the course

• Weekly online activity - Showing activity metrics by week

• Students - Reporting on learners enrolled in the course

• Reports - For general downloadable reports

• Attendance - For logging learner attendance

The average course grade is also displayed above all tabs, along with the last updated date, which displays to the user how old all the data is, which is definitely required in my experience, as staff will frequently expect data to contain events that happened more recently, and then wonder why they’re not showing up. This at least reminds them that the data is outdated.

Open

Course Grade

Open

The first set of filters allows staff to put in up to 3 filters, regardless of type. Filters can be at multiple levels - by section (which is a group of learners, not a content thing in Canvas), by learner, and by assignment name. Filtering by section and learner adds coloured dots representing that filter to the graph, where different assignments through the course are shown along the X-axis. Clicking the dot, or filtering by assignment, immediately opens up that assignment on the chart, where further data can be seen.

The buttons to the right of the filters allow staff to message all learners captured in the currently filtered segment (✉️), or download the data behind the report as a CSV file. Clicking the three dots allows the user to switch between the visualisation and a tabular view of the data behind the visualisation.

The checkboxes allow you to filter broader assignment views by different types of assessed materials - Assignments, Discussions, and Quizzes.

Clicking into a grade on the table or a dot on the visualisation opens up details for that entry. For a section, it appears as above, with the option to view which learners have yet to submit the assignment and message them, or download a CSV list of them:

Clicking into an assignment for a specific learner instead shows details for that learner:

It shows their submission dates, access dates, due dates, and how they scored compared to the average grade for that assignment.

Weekly Online Activity

The weekly Online Activity Charts have the same filter options as the course grade chart. 

The data shown once filtered is Page Views and Participation events by week. Participation events are triggered by all tools that record participation, which at the time of writing is only:

• Announcements: posts a new comment to an announcement

• Assignments: submits an assignment

• Collaborations: loads a collaboration to view/edit a document

• Conferences: joins a web conference

• Discussions: posts a new comment to a discussion

• Pages: creates a page

• Quizzes: submits a quiz

• Quizzes: starts taking a quiz

So it’s specifically those tracked interactions. Interestingly, this limited list is apparently based on the US federal guidelines for what counts as attendance in an online course (thankfully listed on page 1).

As both graphs show averages, this makes it possible to compare a single learner against the course average, but not view the raw numbers for the course as a whole in this view.

By clicking one of the dots on the graph, you can open up a summary for a specific week:

As before, clicking the ✉️ icon allows you to email all learners captured by the filter (specific learners or sections of learners), switch to a tabular view, or download the underlying data as a CSV.

Using the message function on the online activity chart shows a range of options for which subsets of learners to message:

• Viewed

• Didn’t View

• Participated

• Didn’t Participate

A specific resource can also be selected to finer tune which subset of learners should be messaged.

Students

The Students view allows you to view data for students in the course:

As before, up to 3 filters for specific learners and sections can be added

The following data is shown:

• Student Name

• Grade - the learner’s current grade

• % on time - the percentage of assignments with deadlines that were submitted before the deadline

• Last participation - the date of their last tracked event (from a very limited list, what counts as participation listed above)

• Last page view - the date the learner last loaded any page of the course

• Page views - the number of times the learner has loaded any page of the course

• Participations - the total number of participation events the learner has triggered

By clicking a learner’s name, it opens up a view of data specific to that learner, including:

• A course grade chart similar to the main course grade chart discussed above:

• Weekly online activity charts for that learner (again, per the regular activity chart, just filtered to specifically this learner):

• Finally, the communications tab shows the amount of conversation that has taken place between a learner and their instructors, as a bar chart:

• And as a table, where you can click in to see specific details for a given week:

Notably, these three pages of detailed learner analytics can also be given to learners, so that they can view their own activity, grades, and communication records. 

Reports

The reports dashboard allows you to download 5 types of predefined reports in CSV format:

• Missing Assignments - a list of assignments that have not been submitted yet

• Late Assignments - a list of assignments that have been submitted late

• Excused Assignments - a list of assignments that are excused

• Class Roster - a list of students enrolled in the course or section with student contact information such as email and SIS ID

• Course Activity - a list of daily user views and participations

Clicking the Run Report button provides some options for applying filters to the data:

The filter options are not documented, and unfortunately I’ve been unable to find a conclusive list (New Analytics is only available in courses that have a minimum volume of data, so the testing site can’t access anything). I’ve been able to find the following options mentioned in various locations:

• By assignment

• By section name

• By learner

• By due dates

• By course - implying this report can be run from the account level, though this isn’t mentioned in the Canvas documentation that I’ve noticed

Online Attendance

Previously labelled Attendance, this was changed to Online Attendance in response to community complaints that it didn’t handle in-person attendance. The purpose of the report is to record when learners have and haven’t accessed courses on school days.

What counts as attendance can be configured by course staff:

If a student meets any of the selected criteria, they’re marked as “attending” that day The events that can trigger attendance being marked are for learners to:

• View a page in a course

• Post a comment on an announcement or discussion

• Submit an assignment

• Join a BigBlueButton session

• Create a page

• Start or submit a quiz

This is always an OR filter, so completing any of the selected options will result in the learner being marked as attending that day.  Staff can also mark which days the class is in session in order to set which days their “attendance” actually matters:

The requirements for attendance and active days can be defined at the account level, and if they are, staff cannot change them on the course level.

The report itself lists learners, the section they belong to, how many days they’ve missed, and shows a simple true ✅ or false ❌ list for each day of the week in the chosen time frame.