/
[Proposal] AI Supported Grading in Discussions

[Proposal] AI Supported Grading in Discussions

TL;DR

We recommend extending the existing xblock-discussion-grading tool with AI-powered qualitative grading using a rubric and LLM integration. This keeps all discussion logic in one place, aligns with instructors’ grading workflows, and avoids bloating unrelated XBlocks. We’ll reuse LiteLLM from xblock-ai-evaluation as a shared wrapper to handle AI prompts and model switching.

Overview

ASU is exploring AI tools to help instructors grade discussion posts in Open edX without reviewing every individual comment. This initiative builds on earlier MVP plans for the [Proposal] Discussion Grader XBlock (MVP), which focuses on post counting and thread weighting. We now propose a separate AI grading plugin to handle qualitative scoring, as a follow-on phase to that MVP.

This lets us:

  • Keep the MVP focused and modular

  • Separate funding and development of the AI component

  • Deliver a pluggable approach to discussion grading

Tool Landscape: What We Explored

External LTI tools

These are general-purpose AI grading tools that connect to Open edX using LTI which is a standard way to link educational tools together. While powerful, they weren’t built specifically for grading discussion posts, so extra development work would be needed.

  • IBM MARK (Github repo): an Apache-2.0 platform from IBM Skills Network that accepts free-text or file/github URL inputs and returns grades through LTI 1.3.
    It does not support discussions type input and has no built-in way to fetch Open edX forum data, so custom development would be required. However, it is an external tool so custom development related to Open edX is likely not going to be accepted.

    • ✅ Pros: Open-source, already supports LTI

    • ❌ Cons: Doesn’t support discussion posts; no way to pull in discussion forum data automatically from Open edX. Custom development would be required, and because it’s external, changes might not be accepted by the Open edX community.

  • OnMicro AI (Github repo): a no-code assessment builder that also relies on LTI launch and grade pass-back. Like MARK, it would need custom work to ingest forum posts from Open edX.

    • ✅ Pros: Easy to use interface, prompt templates

    • ❌ Cons: Like IBM MARK, it lacks native support for Open edX discussions and would need custom development to connect.

Commercial LTI discussion platforms

  • Packback offers GPT-based “Curiosity Scores” and pushes grades back via LTI grade-book sync, but is closed-source and licence-based.

    • ✅ Pros: Purpose-built for discussion forums; ready to use

    • ❌ Cons: Closed-source and license-based which means it’s not customizable and adds ongoing costs. It also would involve moving away from native open edx discussions forum.

Native Open edX XBlocks

  • xblock-ai-evaluation already provides rubric editing, an LLM client (LiteLLM), and grade pass-back for short-answer and coding items.
    It can be extended to support discussions grading using a forum-fetch helper, but that risks turning the repo into a “junk drawer” of unrelated question types and confusing UX for course authors.

    • ✅ Pros: Already has AI capabilities built-in; supports rubric creation and editing; handles grade pass-back to Open edX

    • ❌ Cons: Not designed for discussion grading; adding forums would stretch the tool’s scope and might confuse UX; mixing use cases increases maintenance complexity

  • xblock-discussion-grading is purpose-built for grading discussions, counting posts and weighting threads. It currently lacks any AI layer for qualitative scoring, which will need to be added, along with a helper to fetch forum posts (not just stats) using the new discussions v2 api (Python and mysql based instead of Ruby).

    • ✅ Pros: Purpose-built for discussions; aligned with current instructor grading workflows; keeps all discussion grading logic in one focused place

    • ❌ Cons: Lacks qualitative AI scoring and would need to be added; needs a helper to fetch full forum post content;  requires new UI for rubrics and grade pass-back

Recommendation Summary

We propose extending the existing xblock-discussion-grading tool:

  • The block already knows which posts belong to which learner and aligns with instructors’ existing workflow for grading participation.

  • Adding an AI quality rubric keeps all discussion-specific logic in one component, avoiding a “kitchen-sink” XBlock that mixes forums with unrelated short-answer and code use-cases.

  • We can move the LiteLLM wrapper (in xblock-ai-evaluation) as a separate wrapper library and import it rather than reinventing the entire LLM plumbing. LiteLLM is a plug-and-play adapter that will let us easily connect and manage different AI models in a unified way.

An alternative approach is to attach forum support directly to xblock-ai-evaluation, which already handles AI grading for other question types. While this might be faster initially, it will stretch the purpose of the tool beyond its original design. Over time, this could lead to a bloated, hard-to-maintain component.

Next steps

  • Secure funding from Open edX community members for [Proposal] Discussion Grader XBlock (MVP)

  • Once the MVP is built and merged, we will revisit this discovery work and expand it into a full product proposal for the AI grading plugin.

  • That future proposal will include detailed product scope, and technical evaluation.