Discussions Metrics and OKRs

Owned by Diana Huang
Last updated: Jun 30, 2016 by Robert Raposa

See GA Dashboard: Forum OKRs

OKRs

Counts of:

  • Posts (thread) created

  • Responses created

  • Comments created

  • Search

Possible Metrics

  • number of forums responses
  • number of users responding to a post
  • number of responses per topic
  • number of users posting (e.g. relative to the number of active)
  • number of posts read (relative to the number of users reading and number of users active)
  • forums posts read to completion (per student?)
  • are learners reading/posting through inline discussions or the discussions tab?

  • Discussion metrics by course

  • # of courses with active discussion forums

Existing Events

  • posted
  • upvoted
  • searched

 

Possible New Events / Modifications to existing events

  • thread is followed
  • thread is viewed
  • distinguishing between discussion tab vs. inline discussions?
  • distinguishing between mobile and non-mobile

 

General Questions


  • What percentage of courses use the edX forums?

    • If use another platform, why?

    • If no forum at all, why?

  • How do courses use forums?

    • What are the main types of activities on the forum? “thread intent” (instructor-specified discussion activities, student support, ask a question about the content, introductions, etc.)

      • % breakdowns of thread intent by course

    • Do course staff closely guide forum setup and organize interaction -- or do they let students take charge?

    • How often do students initiate threads vs. course staff?

    • Do course staff pin threads/use the forum Categories?

    • Does this differ by country/partner/subject?

  • Why do students go to the forum?

    • Do they post a question, participate in an activity, meet other students, jump in on a topic to discuss, or just browse?

    • Do they search for answers?

    • Do they follow frequently?

    • Do they upvote/pin frequently?

    • Do they revisit conversations?

      • If they asked a question, how do they find out there was a response?

      • How many responses do they read?

    • Do they flag posts?

  • What kind of posts appear on the forum?

    • Post length?

    • Number of responses/comments?

    • Tied to the courseware?

    • In categories?

  • How often do students visit the course forum?

    • How is this correlated to completion?

  • If a student is frustrated, do they find what they are looking for on the forum?

    • How often do students search? Do they need to change their search string?

    • How often do students post the same question?

    • How soon are questions answered? How do students know their question was answered?

  • How is forum performance? Is it fast to load/post/search?


User Feedback


  • Do students find the forum easy to navigate? Easy to compose a post in? Informative?

  • Do course staff find the forum easy to manage? Easy to organize? Conducive?

  • Does student usage of the forum meet course team intent?

  • What are the top requested features?


Basic Metrics


  • Adoption: How many courses on edX use forums?

    • Number of forums

    • Number of currently live forums

    • Breakdown by partner/geolocation

  • Activity: How much content is there on the forum, and how often is it visited?

    • Number of posts/pins/votes/views

      • aggregate and per course

      • breakdown of threads/responses/comments

    • % active students who visited the forum at least once.

      • % repeat visitors.

      • % posters vs. just viewers

    • Number of times each post was viewed / revisited by students

  • Editing: How easy is it to author a post?

    • Average number of times a post is edited.

    • Average time spent composing a post / editing a response

  • Organization: How do course staff organize the forum?

    • Number of discussion components.

      • Average number of threads per component

    • Number of course staff created categories

    • Number of pinned threads

  • Curation, Search, and Navigation:

    • % of students voting. % of staff pinning

    • % of forum users/sessions ordering threads by “top voted”

    • % of forum users/sessions who search. % of “successful” searches.

    • Number of times students/staff click “Load all responses” in a thread, “Load more” in the left-hand thread nav, or change their nav filter.

  • Performance:

    • How long does it take to load a discussion thread? the entire forum?

  • Retention:

    • % of dropout from forum users vs. all active students last week

    • % of forum users from verified active users vs. all active users


Advanced Analytics


  • Correlation of forum activity with completion? Distribution of forum usage beyond the first intro exercise

  • Distribution of forum “depth” (number of responses and comments)

  • % posts/views from discussion components vs. the forum.

  • % breakdown of threads by category  (instructor-specified activities, platform support, content questions, introductions)

  • Number of searches. Distribution of subsequent threads viewed. % subsequent posts/answers

  • Distribution of post length / thread depth -- per course

    • breakdown by subject/intent


Examples -- Interesting examples of how courses use forums.