Updated: 25th May, 2023

Project leads (2U): @Asad Azam (tech), @Aamir Ayub (product)

Background

An initiative is underway at 2U for building a comprehensive notifications framework that would allow notifying users for chosen activities in the LMS or Studio. The aim is to deliver timely and relevant information to our users and increase their engagement with the platform. We plan to support three notification channels – web, email, and push notifications, in the long run.

We plan to test and deliver the first release of this framework using web notifications for the edX discussion forum. The ultimate aim is to establish a foundation for a framework that can expand to include other platform areas and notification channels.

This document discusses some concepts beyond web notifications and edX discussion forums, in order to capture a reasonable amount of potential future improvements and make provisions for them in our current implementation.

Notifications preferences

Types of preferences

Notification preferences will be either general or course-specific. General preferences refer to notifications about platform updates, marketing and promotional messages, progress updates etc. Course-specific preferences allow users to customise notifications for each course they are enrolled in, such as new content notifications, discussion forum updates etc.

Preferences management

Users will be able to customise their notification preferences for each course they are enrolled in. This will be done through a dedicated preferences page that can be accessed through a direct URL, a link on the Account Settings page, or a button on the notifications tray. This feature will provide users with greater control over the notifications they receive and ensure that they only receive notifications that are relevant to their learning experience.

Open

A button in notifications tray to navigate to the preferences page.

Open

A link to preferences page on Account Settings page.

Preferences page

The notification preferences page allows users to manage their notification settings for each course they're enrolled in. Key features of this interface include:

1. Grouped preferences: Preferences are organised by platform area (e.g. Discussions, Coursework) to help users quickly find and manage the settings they care about. For example, the "Core notifications" preference covers activity types 5-9 in the forum notifications list, while the "Content reported" preference covers activity types 10-12.

2. Important notifications: Certain types of notifications are considered important and cannot be individually opted out of. However, users can opt out of the entire platform area to stop receiving notifications for that activity. For example, users cannot opt out of "Core notifications" for Discussions, but they can turn off notifications for the entire Discussions forum.

3. Email and push notifications: In addition to web notifications, users can opt in to receive email and/or push notifications for some activity types. For example, the option to enable email and push notifications is not available for likes on authored posts and responses.

4. Role-based visibility: The visibility of preferences for each activity type is based on roles. For example, only the users with Course Staff and Course Admin roles will see preferences related to course authoring.

5. Default notification settings: Default notification settings are assigned to each preference based on the user's role in the course. For example, users with Discussion Admin, Discussion Moderator, Community TA, and Group Community TA roles will be automatically subscribed to web notifications for the "Content reported" preference by default. However, push and email notifications will not be automatically enabled for this preference.

   1. Since roles are assigned after a user has enrolled in a course, and some may change afterwards, we will need to link the default settings for a user, to changes in its role in the course.  

Preference creation logic

General preferences for a user will be created as soon as they sign up on the platform. Course specific preferences will be created for a user after they enrol in a course.  Users will start receiving notifications as soon as their preferences have been created.

Notifications tray

Users will be able to see and interact with web notifications via a notifications tray. This tray will be accessible via a bell button in the top right corner, alongside the username, as seen below. Clicking the button will open the notifications tray on the right side of the page that the user is currently viewing. Clicking elsewhere on the page will close the tray.

Layout of notifications tray

1. Link to preferences: The button to navigate to the notification preferences page on the notifications tray helps users review and set preferences conveniently.

2. Tabs: To improve readability and prevent context switching, notifications will be organised and accessed through tabs representing different platform areas. This will allow users to quickly find and view notifications related to specific areas, such as discussion forums or grading, instead of having to search through a long, mixed-up list of notifications.

   1. All tabs will be visible at all times to allow users to track older notifications. Sorting the tabs based on the number of unread notifications may cause confusion for users, so the tabs will remain in their fixed positions.

   2. UI will be responsive such that if horizontal space is insufficient, the remaining tabs will be listed in a dropdown menu as seen below.

3. Notifications list: After selecting a specific tab representing a platform area, notifications for relevant activities within that area will be listed. More details are discussed below.

Format of a notification

Each notification item contains the following elements:

1. Temporal relevance: The amount of time that has passed since the notification was created.

2. Context: A brief description of what the notification is related to, could be a course or a feature announcement etc.

3. Notification: Detailed information about the activity that triggered the notification.

4. Tracking: A status indicator that shows whether the notification has been read or not.

5. Icon: The icon serves as a visual cue that quickly conveys the type of activity.

6. URL: Link to the content that notification is about.

For forum notifications, the context will be the name of the course to which the forum belongs. A list of potential forum activities that will trigger a notification and the corresponding notification format, is presented later in this section.

Unseen notifications

Unseen notifications refer to notifications that have been generated but not yet viewed by the user. The count of unseen notifications will help capture the user's attention and ensure that important updates are not missed. Count of unseen notifications will be tracked for each platform area separately. The count will be visible alongside the name of the tab for that platform area. Once the tab is clicked, the count will disappear and set to zero.

Count of all unseen notifications: The count of all unseen notifications will be the sum of unseen counts for all platform areas. The count will be shown alongside the bell icon. Once the notifications tray is opened, the count will change according to the user clicking each tab. If all tabs have been opened, the count will disappear and set to zero.

Difference between unseen and unread: It's important to distinguish between 'unseen' and 'unread' notifications. 'Unseen' notifications refer to those that the user hasn't viewed at all, while 'unread' notifications have been viewed but not yet clicked to access the associated URL.

Unread notifications (versus unseen)

Notifications that have not been clicked by the user are considered unread. Unlike unseen notifications, the count of unread notifications is not maintained or displayed. However, each notification will have an identifier for its read/unread state. This allows the user to easily identify which notifications they have already viewed and which ones they still need to read.

Structure of forum notifications

Sort

Notifications will be sorted in descending order based on the date they were generated, with the newest appearing at the top of the list. To improve readability, the notification list will be divided into two sections: “Last 24 hours” and "Earlier" (for all notifications generated before the last 24 hours).

Loading notifications

To prevent any impact on page loading, notifications will not be loaded until the user opens the notifications tray. To reduce loading time of notifications tray, the following measures will be implemented:

1. Upon opening the tray, notifications will be loaded (max: 20) for only the platform area whose tab opens by default.

2. When another tab is opened, notifications for that tab will be loaded (max: 20).

3. A "Load more" button will be available in every tab to allow users to load older notifications in batches of 20. Having a button is necessary (as opposed to endless scrolling) for accessibility reasons.

Mark all as read

To enable users to quickly mark all notifications as read for a specific platform area, each tab will include a “Mark all as read” button. This button will change the state of all notifications for that platform area to “read”, even for notifications that have not been loaded yet.

Interacting with a notification

Notifications may or may not be associated with a URL within the platform. For example, discussion notifications will always be linked to a specific post, while some types of notifications, such as system alerts or announcements, may not have an associated URL. When a notification is clicked, the user will be taken directly to the corresponding URL.

• URLs will open in the new tab, so the user will be taken away from the notifications tray.

• The notifications tray will remain open when the user clicks on a notification and navigates to the corresponding URL. This allows the user to easily return to the notifications tray to continue viewing or managing their notifications.

Icon for notification type

Icons on notifications help users quickly determine the nature of the notification before reading it in full. For example, an announcement icon can be used for a notification about a new feature in the platform.

When will the notifications tray be updated?

Notifications will not be live and will not update in real-time. Instead, an API call will be made periodically or on trigger to fetch new notifications.

When new notifications are fetched:

• If the notifications tray is already open, they will be added to it in real-time without requiring the user to refresh the page or click any buttons. To indicate that new notifications have arrived, the tray or individual notifications could be styled differently, e.g. by changing their colour or adding a badge.

• If the notifications tray is closed, the unread count displayed on the notifications button will be updated. Additionally, the button itself could be animated (e.g. by shaking) to draw the user's attention to the fact that new notifications have arrived.

Tracking metrics

We plan to add tracking events that’ll help us answer the following questions:

1. What activity types had notifications enabled [or disabled] by default but were later disabled [or enabled] by users? Do these users belong to a particular course role?

2. What is the click % of notifications for each activity that has notifications enabled [or disabled] by default? Do these users belong to a particular course role?

3. What is the distribution of total notifications/day received by users? Is it manageable by all users? If not, why?

4. What is the distribution of click % of notifications for users? Is it too low for a group of users? Why?

5. What % of active users open the notifications tray? Or in what percentage of sessions is the notifications tray opened?

6. What % of active users have disabled notifications for the entire Discussions area?

Storage cost optimisation

As notifications accumulate in the database, the amount of storage needed to maintain them can grow rapidly over time. To manage this growth, we plan to implement an automatic deletion system that will regularly remove notifications that are older than a specified number of days. This system will run daily, and the number of days used as the cutoff for deletion will be configurable by the development team. By implementing this system, we can ensure that our storage requirements stay (reasonably) under control, without sacrificing the ability to maintain an adequate history of notifications.

Notification delivery start/stop

User Perspective

Our priority is to maintain an intuitive notifications system. To achieve this, users will commence receiving notifications related to a course run upon enrolment, which will persist throughout their active engagement with the course.

Platform Perspective

Our framework will start delivering notifications from the moment a course run is created and continue up until X days following the course run's end date. This will enable us to dispense course-authoring updates prior to the course's start and facilitate relevant post-completion interactions and queries after the course concludes.

Future enhancements and optimisation

Optimising preferences

Optimising defaults for all users: After the rollout of web notifications, we will closely monitor the click rates of notifications for each activity type and rank them. If an activity type's notifications receive a click rate above a certain threshold, we will turn on the web notification preference for that activity type by default. Conversely, if an activity type's notifications receive a click rate below a certain threshold, we will turn off the web notification preference by default. Additionally, we will keep track of the activity types that users subscribe to the most and use this information to help determine default preferences for all users.

Our goal is to gather enough data to optimise forum notifications and create a set of guidelines for other teams to follow when optimising default preferences for their platform areas. This will ensure that users receive the most relevant notifications and can easily manage their subscriptions. By doing so, we hope to increase user engagement and improve the overall learning experience on our platform.

Optimising preferences for each user: We plan to use a recommendation algorithm to optimise notification preferences for each user after the rollout. The algorithm will analyse the notifications clicked on by the user and the notifications the user is subscribed to. Based on this data, the algorithm will recommend whether users should opt-in or opt-out of specific notifications that may be helpful. This will ensure that each user receives notifications that are most relevant to them, improving their overall learning experience.

Notifications page with sorts and filters

A dedicated notifications page will help us create a UX filtering, sorting, marking, deleting, and bulk action features for notifications. This will help users easily manage their notifications, find the ones that interest them, and quickly perform actions on multiple notifications at once, such as marking them as read or deleting them. With these features, users can more easily stay organized and maintain control over their notifications log, which can help improve their overall experience on the platform.

Grouping notifications

Several types of forum notifications can be consolidated into a single notification to reduce the number of notifications users receive. Here are some examples:

1. Responses to a post: Multiple responses to a post can be consolidated into a single notification to inform the user that there have been multiple responses to their post.

2. Comments on a response: Multiple comments on a response can be consolidated into a single notification to inform the user that there have been multiple comments on their response.

3. Likes on a post or response: Multiple likes on a post or response can be consolidated into a single notification to inform the user that there have been multiple likes on their post or response.

By consolidating related notifications into a single one, users can reduce the number of notifications they receive, making it easier for them to manage their notifications and stay informed about activity in the forum.

Managing surplus notifications

In the approach outlined in the notification delivery start/stop section, we see a potential risk of delivering surplus notifications. This can occur in two scenarios:

1. Inactive users: Users who, despite enrolling, haven't interacted with a course for an extended period (e.g., 3 months). For them, frequent course notifications might be redundant or even intrusive.

2. Course completers: These are users who have successfully completed a course. While they may still appreciate some updates, routine course notifications might not be relevant to them and could contribute to notification fatigue.

To make the user experience an optimal over time, we will monitor and manage surplus notifications through the following strategies:

1. Notification click-through rate: We will monitor how often our notifications are being clicked on. A low click-through rate might indicate that our notifications are not engaging or relevant to the users receiving them, necessitating an adjustment in our approach.

2. Active vs. inactive users: We will monitor the ratio of notifications sent to active versus inactive users. If inactive users are receiving a significant chunk of notifications, we will revisit our approach.

3. User feedback and engagement: We will actively seek user feedback and monitor user engagement with notifications and preferences.

Potential risks associated with web notifications

A criticality analysis of risks involved in web notifications is presented below. Each one is assigned a consequence rating and a probability rating from 1, 2 and 3 with 3 being the highest. Addressing the risks with highest criticality rating is essential. It is noteworthy that this analysis is subjective and there will always be room for improvement.

Next milestones

Upon successful rollout of web notifications for forums, we have identified the following areas for further consideration:

1. Push notifications for edX mobile apps: We are exploring options to incorporate push notifications into our mobile applications. This may be accomplished via third-party integration or in-house development.

2. Email notifications: We plan to implement an email notification system, possibly using a third-party notification system.

3. Preventing notification fatigue: We aim to develop a mechanism to regulate notification volume and frequency for each user, thereby preventing notification overload while ensuring that critical updates are delivered. This will become increasingly crucial as more platform apps utilise our notification framework and incorporate push notifications.

4. New feature announcements: We're considering the development of an announcement system for new releases and updates within the edX platform, ensuring that all users are notified of these updates.