Initial Loadtesting
Initial Tests:
Note: Locust user wait times were not correctly set here. Ignore user number and wait time and assume requests were being sent as soon as one of the users are ready.
In the initial testing phase of the discussion api, the wait time between each top-level task was 500ms. If a task has multiple calls, such as PATCH comment, the wait time between each of those calls were 1000ms. These waits are very short and represent an accelerated usage of the discussions api. The first test that was run was against BerkeleyX/ColWri2.2x/1T2014 (~38,000 threads, ~40,000 comments) with 10 locust users. A high percentage of the responses were failures and the response times were in the 1000-3000ms range. Under the assumption that a course with more posts would have worse performance, it would be expected that this course which is the 5th biggest course in our system, would not perform. To get a better baseline, we tried a smaller course.
The next course used was BerkeleyX/CS.CS169.1x/3T2013 (783 threads, 3385 comments at the time) which is about 2000 more posts than the median. In this test, we set up 10 locust users (same as the previous) for 1 hour and yielded better results. It was noted that during the 1 hour, the response times gradually got worse and less consistent as more spikes showed up on the charts. To see if the issue was too much load, another test was run against the "same" course (~5000 threads, ~10,000 commentS) with 5 locust users. This test showed that the response times were actually much worse than the previous test. After analyzing both sets of data, it was realized that the creation of posts in the course had affected the performance of that course. It was found that the number of posts in the course had increase about 2-3 times in an hour. During a week, about 50,000 posts are made against ALL courses. In this 1 hour, a fifth of that was added to a SINGLE COURSE. This significant increase in posts lead to the slower response times in the second test.
Another test was to be run against a small course, without any thread creation.
The next course used was BerkeleyX/CS.CS169.1x/3T2013 (783 threads, 3385 comments at the time). Unlike the loadtest with the very large course "BerkeleyX/ColWri2.2x/1T2014", the server was able to handle the requests without returning 500s. These numbers seem more reasonable. The test ran for 1 hour.
Name | reqs | # | fails | Avg | Min | Max | Median | req/s | 95% |
|---|---|---|---|---|---|---|---|---|---|
GET | GET_comment_list | 1718 | 0(0.00%) | 459 | 163 | 1783 | 410 | 0.1 | 950 |
GET | GET_thread | 106253 | 0(0.00%) | 195 | 153 | 2302 | 180 | 10.3 | 270 |
GET | GET_thread_list | 61009 | 0(0.00%) | 595 | 155 | 4064 | 460 | 6.8 | 1700 |
PATCH | PATCH_comment | 1719 | 0(0.00%) | 593 | 208 | 2122 | 520 | 0.1 | 1100 |
PATCH | PATCH_thread | 1883 | 19(1.00%) | 403 | 166 | 2283 | 360 | 0.3 | 810 |
POST | POST_comment_comment | 1373 | 0(0.00%) | 966 | 591 | 4261 | 900 | 0.1 | 1600 |
POST | POST_comment_response | 4135 | 0(0.00%) | 928 | 546 | 4194 | 860 | 0.5 | 1600 |
POST | POST_thread | 4688 | 0(0.00%) | 492 | 424 | 4102 | 460 | 0.4 | 610 |
As the loadtest continued, the 95th and 50th percentile gradually became slower and slower. The throughput was very high compared to that of production (~100rpm for a similar call /django_comment_client.forum.views:single_thread) so the test will be rerun with less load.
After analyzing this data, it was realized that the number of posts in this course had actually altered the response times of the http requests. The course had increased about 2-3 times. In a week, about 20,000 threads, 20,000 responses, and 10,000 comments are created in ALL COURSES. In this 1 hour test, we have created a fifth of that value against a single course. Take note that the following tests were against the same course, but with the new increased post count.
The test below was against the same course but at half the throughput. As shown on the graph, the response times were slower. This test ran for 30 minutes. Again, we see that the response times increases slowly as the RPMs decreases appropriately.
Name | reqs | # | fails | Avg | Min | Max | Median | req/s | 95% |
|---|---|---|---|---|---|---|---|---|---|
GET | GET_comment_list | 306 | 0(0.00%) | 517 | 179 | 1227 | 490 | 0.1 | 910 |
GET | GET_thread | 19602 | 0(0.00%) | 198 | 150 | 2403 | 180 | 4.2 | 320 |
GET | GET_thread_list | 11150 | 1(0.01%) | 915 | 156 | 4026 | 770 | 3.8 | 1900 |
PATCH | PATCH_comment | 306 | 0(0.00%) | 768 | 423 | 2007 | 690 | 0 | 1200 |
PATCH | PATCH_thread | 370 | 2(0.54%) | 508 | 253 | 2043 | 480 | 0 | 890 |
POST | POST_comment_comment | 268 | 0(0.00%) | 1139 | 775 | 2556 | 1100 | 0.1 | 1500 |
POST | POST_comment_response | 735 | 0(0.00%) | 1108 | 715 | 2573 | 1000 | 0.1 | 1500 |
POST | POST_thread | 833 | 0(0.00%) | 478 | 416 | 1925 | 450 | 0 | 600 |
In this test, the course HarvardX/1368.2x/2T2015 (92 threads, 1697 comments) was used with 10 locust users for 30 minutes without an post creation. As expected, we observed very consistent and lower response times.