Moodle - Course-Level Analytics and Reporting
- Matthew Harrington
Analytics
Moodle’s course analytics functionality is much more robust than their site-level reporting tools. Many third-party tools exist to expand course level analytics, and out of the box, Moodle includes many different dashboards and models that can be used to track and measure course and learner performance.
Analytics & Insights
One of Moodle’s main features for generating analytics uses analytics models to generate what they call insight reports. These are first defined at the administrator level:
These analytics models can use a variety of factors, from simple categorisation based on data (such as “Students who have not accessed the course yet”) to much more complex machine-learning algorithms that allow the system to conduct analyses of learner data to determine things like “Students at risk of dropping out”, which is one of the few default models that actually has meaningful documentation to show how it determines the learners who are at risk.
Within courses, staff have the option to generate reports based on these analytics models, and take actions such as sending a message to all learners flagged by the algorithm, or automated actions can be taken using those models as triggers (such as sending messages to all learners who have assignments overdue, or due soon). Learners are able to flag the intervention as useful or not useful, and the actions taken and their reception are reported back to the site administrator.
The machine learning-based models are not functional until a site has enough data to train the model on a particular site, and the broader system is reliant on institutions designing and building their own tools to provide custom-tailored analytics. My cynical gut feeling is that the reason for this is simply institutional ego - few universities want to be told how to work with their data, so Moodle have focused on providing a toolkit for education researchers to build their own tools for their institution, as shown in these diagrams from their documentation:
The API and methodology for these analytics is beyond my expertise without much deeper research than I have time for, and extremely in-depth, so it’s worth reviewing the documentation for this in more depth, as I don’t want to inaccurately summarise the information:
Analytics API documentation (the source of the diagrams above, as Google docs has eaten some of the image quality)
Course Reporting
In addition to reports from analytics models, Moodle course staff can review a set of preset reports from their course dashboard.
Competency Breakdown
Moodle has the concept of competencies, which are tied to specific content and automatically or manually evaluated based on learner performance. A teacher in a course can view the competency statuses of a learner as one report from their course dashboard:
Logs
Log reports are of activity logs for both learners and staff that show all actions taken by and affecting a user. An example of this could be a learner posting in a discussion forum, or a staff member grading a learner submission.
These log reports can be filtered by course, participant, time period, activity type, action types (such as teaching events or participation events), and event names. The report itself contains the following information:
Date/time
Full name of the user that took the action (linking to that user’s profile)
Full name of the user their action affected (typically blank unless staff affecting another user)
Event context - a link to the page on which the user took the action
Component - the category the event belongs to, such as system events or assignment events.
Event name - the actual action that was taken by the user
Description - a semi-human readable summary of what happened.
Origin - where the action took place, device-wise, such as web, or mobile app.
IP address - the IP address of the user at the time of taking the action.
These tools seem somewhat unusually placed, as they can be used to access logs from any course, and are more of an administrative tool, yet they are clearly shown as being part of the course dashboard. They are also accessible through the admin dashboard, and my suspicion is that they simply decided to allow course staff access to the same tools in order to review suspicious activity in courses, without actually tailoring it to course staff.
Live logs behave the same way, but instead show a live feed of actions taking place within the course, which can be filtered in the same way as regular logs, or paused to review recent actions.
Activity Report
The activity report provides staff with a summary of how many views by how many users each part of their course has received.
Each activity and resource records the following information:
Activity name
Number of views by how many unique users
Related blog entries (if tied to a blog post)
The date the item was last accessed by a user
This report will eventually lose data, as it only covers a period of time as set by a course admin.
In addition to this, the Complete Activity Report is available from the same location, which provides a full, in-depth summary of all of a learner’s submissions, activity completion, and content access. If configured, learners can be given access to their own activity report. This fundamentally serves a completely different purpose to the original activity report, so them living in the same section is very weird, but that’s Moodle.
Course Participation
Participation reports:
Generate a list of who has participated in a given activity, and how many times.
Can be filtered by role, group, and action (View or Post).
Allow individuals or groups students (e.g. those who have not participated) to be easily messaged.
These have a lot of overlap with analytics and insights reports, and may be an earlier version of those tools. The documentation on this feature is extremely poor, but the way it’s phrased makes me assume the primary purpose of this is monitoring discussion engagement and messaging learners who haven’t posted:
Statistics
Statistics is an optional core plugin component that adds a basic data visualisation to courses when enabled:
This shows the time period, along with how many views the course has gotten from different groups of users. It links directly to logs scoped to that time period, and relies on logs, making it essentially a visualisation of the logging functionality.
It’s honestly extremely likely that I am missing tools and reports available to course staff in Moodle, as their documentation is extremely poorly structured and incomplete in a lot of places, and it appears almost every institution with their own documentation on reporting uses third-party or in-house plugins to augment their reporting experience. This leads me to the assumption that the out-of-the-box reporting experience of core Moodle is not fit for purpose, especially as the Moodle documentation itself refers users to third-party plugins.