Initial Tests:

Note: Locust user wait times were not correctly set here. Ignore user number and wait time and assume requests were being sent as soon as one of the users are ready.

    In the initial testing phase of the discussion api, the wait time between each top-level task was 500ms. If a task has multiple calls, such as PATCH comment, the wait time between each of those calls were 1000ms. These waits are very short and represent an accelerated usage of the discussions api. The first test that was run was against BerkeleyX/ColWri2.2x/1T2014 (~38,000 threads, ~40,000 comments) with 10 locust users. A high percentage of the responses were failures and the response times were in the 1000-3000ms range. Under the assumption that a course with more posts would have worse performance, it would be expected that this course which is the 5th biggest course in our system, would not perform. To get a better baseline, we tried a smaller course.

    The next course used was BerkeleyX/CS.CS169.1x/3T2013 (783 threads, 3385 comments at the time) which is about 2000 more posts than the median. In this test, we set up 10 locust users (same as the previous) for 1 hour and yielded better results. It was noted that during the 1 hour, the response times gradually got worse and less consistent as more spikes showed up on the charts. To see if the issue was too much load, another test was run against the "same" course (~5000 threads, ~10,000 commentS) with 5 locust users. This test showed that the response times were actually much worse than the previous test. After analyzing both sets of data, it was realized that the creation of posts in the course had affected the performance of that course. It was found that the number of posts in the course had increase about 2-3 times in an hour. During a week, about 50,000 posts are made against ALL courses. In this 1 hour, a fifth of that was added to a SINGLE COURSE. This significant increase in posts lead to the slower response times in the second test.

Another test was to be run against a small course, without any thread creation.

    In this test, the course HarvardX/1368.2x/2T2015 (92 threads, 1697 comments) was used with 10 locust users for 30 minutes without an post creation. As expected, we observed very consistent and lower response times.  

    In this test, we would like to compare a course that was similar to the size of the initial BerkeleyX/CS.CS169.1x/3T2013 (783 threads, 3385 comments at the time) and one that is bigger than HarvardX/1368.2x/2T2015 (92 threads, 1697 comments). The course used SMES/ASLCx/1T2015 (1700 Threads, 3047 comments). As expected, the response times were comparable to BerkeleyX/CS.CS169.1x/3T2013 (783 threads, 3385 comments at the time)  and slower than HarvardX/1368.2x/2T2015 (92 threads, 1697 comments). An additional test was run against a tiny course  TsinghuaX/00690242_2x/3T2014 (29 threads, 155 comments). As expected,  the response times for TsinghuaX/00690242_2x/3T2014 (29 threads, 155 comments) were the best by far.

Tests while Mixing Courses:

    In this series of tests, we will be testing against different sized courses with different ratios. There are spikes in response times in production and one possibility may be that the few very large courses may be the reason. In this initial test, we ran it for 3 hours, with a small course vs large course on a 20:1 ratio. The large course used is BerkeleyX/ColWri2.2x/1T2014 (~38,000 threads, ~40,000 comments) , while the small course is SMES/ASLCx/1T2015 (1700 Threads, 3047 comments). We can see that there are spikes in the response times. There have been many 500 responses from the large course which needs to be addressed. There were many unexpected 500 errors during this test. Upon further investigation, it was found that there was a memory issue due to missing indexes.

Addressing the 500s:

    Although only present on the larger courses, a significant amount of 500s were being returned. A solution has been proposed to include an index to account for this but for now in order to continue the load tests, the offending parameters "last_activity_at" and "asc" will be removed from out random selection. This also means that our thread retrieval for PATCHing (and DELETEing which has been disabled) will be less random, but sufficiently random. By removing these parameters, we hope to get rid of these 500s and see a performance improvement for large and small courses.

Test reruns:

    These tests have the non-indexed fields removed just to verify that those were causing the 500s. All tests were run with 10 locust users for 30 minutes. Response times were all better, no spikes, and response times were better the smaller the course.

After adding indices:

Page size vs. Response time: 

    In a new test, it was found that SMES/ASLCx/1T2015 (1700 Threads, 3047 comments) was slowing down over time. To identify what was happening, a new course was seeded which we will call DAPI (1000 Threads, 500 comments). In the analysis of the forums, we saw that the median body size was 250 characters. Our PATCH operations can be 4, 250, 1000, 5000, or 10000 characters. This test was run over 10 hours. The decrease in response time needed to be addressed.

GETting a single threads vs comment_count 

    This tests takes the course BerkeleyX/ColWri2.2x/1T2014 (~38,000 threads, ~40,000 comments) and then retrieves a single thread. This thread is then grouped by its comment_count. This test was run for 1 hour. As expected, because of pagination issues, threads with more comments are taking longer to return.