Discovery - Discussion Redesign

Product

• Product Requirements: Forum Navigation Improvements
• UX Designs (Invision)
• TNL-3743 - Getting issue details... STATUS

Engineering Docs

• Comment Service Python Port
• Discussion Redesign Architecture
• Discussion XBlock Meeting Notes - 09-30-2015
• Greg's Musings on the Future of the Forums Code
• Forums v2

Notes on past Ruby to Python port status

Current Concerns Regarding Discussions

The following is not meant to replace existing docs above, nor the docs from the Discussions (Forums) Home Page.  Below is an attempt to consolidate and summarize some of the more pressing issues.

Discussions Related Product Backlog

• Top Issues/Priorities:

  • No clear organization or hierarchy
  • No unity between inline and course wide discussions
  • No way to easily surface interesting content
  • No notifications (or UI) to see user's own activity and relevant updates

• Useful Docs:

  • Forum Navigation Requirements

  • Discussion Category Usage and Management

• Backlog Spikes:

  • TNL-3117: Spike: Discussion API gap analysis

  • TNL-3118: Spike: Ability to toggle between new and old Discussion Forum UIs

  • TNL-3119: Spike: Aggregating Thread Data

  • TNL-3120: Spike: Discussion Topics (commentables) as first-class citizens

Performance Issues

• Discussion API Performance Testing (by Mobile team)

• Timeout errors in the discussion forums for many +5s first-time calls with edx-platform timeout of 5s.  More details here. 

• A current sampling of slow queries:

• Getting a thread and counting it’s comments

• Getting thread list

• Counts are slow to calculate.  Existing Production pinned threads of 3000-4000 comments currently time out consistently.

• Possibly use cached comment_count to estimate count for large threads

• Getting active threads

• Getting all threads a user is subscribed to

• NoSQL/MongoDB related performance issues may require:

• Additional caching

• Adjusting functionality (e.g. estimating counts for large threads)

• Stored Javascript (like stored procedures), or

• Moving to a relational database; used by all major third-party forum solutions

• Improved pagination will require some fixes mentioned above

• Inexact performance testing (issues surfacing in Production only)

Ruby Maintainability

• Uses outdated library for ElasticSearch with no clear upgrade path

• Engineers have fear of getting into the Ruby code

• Inhibits strategy toward all-Python IDA (if that is the ultimate direction)

Forked XBlock

• XBlock required for McKinsey must either be merged or replaced

Other Considerations for Refactor/Replacement

General Maintainability

• Minimize future maintenance cost on functionality outside of edX core capabilities (buy vs build?)

• Simplify code to use more consistent terminology where possible

• Consolidate Discussions related code (separate repo or directory)

Discussions XBlock

• Current implementation for McKinsey exists on fork of edX

• Estimate: ~4 developer weeks to merge into the platform

• XBlock plan and delivery date for edx-platform will determine whether Solutions proceeds with merge of fork

• Requires solution in within 6 months (by April/May 2016)

• Current UI enhancements due in June 2016

Convert Ruby to Python

• Moving out of Ruby will likely require this rewrite, unless we move to a new solution

• MongoDB team forked Discussions and moved Ruby code into Django Comment Client

• Forked 3 years ago

• Dropped Elastisearch and replaced with Mongo search

• Dropped separate deployment

• Dropped code/functionality they weren’t using

• Jim Abramson has a PR from an initial edX attempt at this rewrite

• Peer Learning Working Group

• Project planned for rewriting Ruby in Python to make it extensible for some Peer Intervention related requirements (Piotr)

IDA Dependencies

• Requires standard way to handle Advanced Settings

• Requires managing and handling roles (Dave Ormsbee)

• User API

• What has Jim created for XSeries? Is this just for oAuth?

Integration Points for Third-Party Solutions

Research Data

• Does research dump of Discussion data influence refactoring possibilities?

Other Resources

Related edX Architectural Links

• Open Edx Architecture
• Extending edX Platform (Confluence)
• Creating Django based Independently Deployable Applications (IDAs)
• Programs Documentation (IDA under development)

Third-party Forums Resources

• Piazza (used by Berkley Class in edx)
• Django-based Forums Software
• Comparison of Forum Software (Wikipedia)
  

Rough Draft Architectural Diagram (Future)