Discussion API Results

Owned by Christopher Lee (Deactivated)
Last updated: Nov 25, 2015
3 min read

Performance Improvements:

  • ~2,000,000 invalid items in mongo were created during a migration during Sept. 2013. There are plans to remove these entries during the mongo upgrade. 

OPS-1062 - Getting issue details... STATUS

  • Forums testing was very difficult due to the unpredictability of the forums. The noise in the forums made it very hard to pinpoint individual parameters. It was discovered that in the default view in the browser and app, the index for  "last_activity_at" or "date" filter was found to be missing in the loadtest environment and read replica.  After adding this index on the loadtest environment, forums requests seemed to be more stable as we saw less spikes and less timeouts and response times seemed to improve across all courses. In addition to this index, there are other indexes that we may need to support such as "unread", "unanswered" and "asc". Hopefully, we will see the same effect on production when applied. 

DEVOPS-3123 - Getting issue details... STATUS

MA-1504 - Getting issue details... STATUS

    • Why was this not found earlier? This missing index was discovered when trying to query against one of the largest courses for posts. On production, a timeout error would be returned rather than an out of memory sorting error. Smaller course that were also affected by this, were small enough to get away with queries with the missing index.

Concerns:

  • It was found that thread retrieval depended on the number of comments in a thread. The more comments there are in a thread, the longer the response time. This may be an issue because of the potential usage patterns leaning towards more requests to comment heavy threads. Raw data here.

MA-1503 - Getting issue details... STATUS

Notes:

  • Browser vs. Discussion API overhead: this was a criteria that we could not obtain accurate information from these tests since the behaviour of the discussion forums changed significantly after adding the indexes. Other factors included being unable to produce normal non discussion forums load and a different mongo configuration. In general, after the performance improvements, the discussion API performance seems to be better than that of the browser in terms of response time and stability. 

     Apples to oranges Loadtest vs Production

    GET Thread List

    One the left is a 30 minute loadtest for a course with ~80,000 posts on the loadtest env with the added indexes. On the right is a 7 day snapshot of the production data across all courses without the added indexes.

    GET Thread

    One the left is a 30 minute loadtest for a course with ~80,000 posts on the loadtest env with the added indexes. On the right is a 7 day snapshot of the production data across all courses without the added indexes.