...
Another test was to be run against a small course, without any thread creation. | Anchor |
|---|
| ramping_course |
|---|
| ramping_course |
|---|
|
| Expand |
|---|
| title | Tests for smaller course which gradually grew |
|---|
|
The next course used was BerkeleyX/CS.CS169.1x/3T2013 (783 threads, 3385 comments at the time). Unlike the loadtest with the very large course "BerkeleyX/ColWri2.2x/1T2014", the server was able to handle the requests without returning 500s. These numbers seem more reasonable. The test ran for 1 hour. | | | | | | | | | 95% |
|---|
| GET | GET_comment_list | 1718 | 0(0.00%) | 459 | 163 | 1783 | 410 | 0.1 | 950 | | GET | GET_thread | 106253 | 0(0.00%) | 195 | 153 | 2302 | 180 | 10.3 | 270 | | GET | GET_thread_list | 61009 | 0(0.00%) | 595 | 155 | 4064 | 460 | 6.8 | 1700 | | PATCH | PATCH_comment | 1719 | 0(0.00%) | 593 | 208 | 2122 | 520 | 0.1 | 1100 | | PATCH | PATCH_thread | 1883 | 19(1.00%) | 403 | 166 | 2283 | 360 | 0.3 | 810 | | POST | POST_comment_comment | 1373 | 0(0.00%) | 966 | 591 | 4261 | 900 | 0.1 | 1600 | | POST | POST_comment_response | 4135 | 0(0.00%) | 928 | 546 | 4194 | 860 | 0.5 | 1600 | | POST | POST_thread | 4688 | 0(0.00%) | 492 | 424 | 4102 | 460 | 0.4 | 610 |
As the loadtest continued, the 95th and 50th percentile gradually became slower and slower. The throughput was very high compared to that of production (~100rpm for a similar call /django_comment_client.forum.views:single_thread) so the test will be rerun with less load. After analyzing this data, it was realized that the number of posts in this course had actually altered the response times of the http requests. The course had increased about 2-3 times. In a week, about 20,000 threads, 20,000 responses, and 10,000 comments are created in ALL COURSES. In this 1 hour test, we have created a fifth of that value against a single course. Take note that the following tests were against the same course, but with the new increased post count. 
The test below was against the same course but at half the throughput. As shown on the graph, the response times were slower. This test ran for 30 minutes. Again, we see that the response times increases slowly as the RPMs decreases appropriately. | Name | reqs | # | fails | Avg | Min | Max | Median | req/s | 95% |
|---|
| GET | GET_comment_list | 306 | 0(0.00%) | 517 | 179 | 1227 | 490 | 0.1 | 910 | | GET | GET_thread | 19602 | 0(0.00%) | 198 | 150 | 2403 | 180 | 4.2 | 320 | | GET | GET_thread_list | 11150 | 1(0.01%) | 915 | 156 | 4026 | 770 | 3.8 | 1900 | | PATCH | PATCH_comment | 306 | 0(0.00%) | 768 | 423 | 2007 | 690 | 0 | 1200 | | PATCH | PATCH_thread | 370 | 2(0.54%) | 508 | 253 | 2043 | 480 | 0 | 890 | | POST | POST_comment_comment | 268 | 0(0.00%) | 1139 | 775 | 2556 | 1100 | 0.1 | 1500 | | POST | POST_comment_response | 735 | 0(0.00%) | 1108 | 715 | 2573 | 1000 | 0.1 | 1500 | | POST | POST_thread | 833 | 0(0.00%) | 478 | 416 | 1925 | 450 | 0 | 600 |
|
...
| Expand |
|---|
| title | Rerun of mixed courses test after first index was added |
|---|
|
This is before adding the "asc" index and after adding the "last_activity_at" index. The major difference is the lack of spikes, consistent RPM, better response times especially for the large course and no 500s. This test was run for 3 minutes with a small course vs large course on a 20:1 ratio. The large course used is BerkeleyX/ColWri2.2x/1T2014 (~38,000 threads, ~40,000 comments) , while the small course is SMES/ASLCx/1T2015 (1700 Threads, 3047 comments). 
| Name | reqs | # | fails | Avg | Min | Max | Median | req/s | 95% |
|---|
| Large | GET_comment_list | 46 | 0(0.00%) | 443 | 207 | 1155 | 370 | 0 | 840 | | | GET_thread | 2404 | 0(0.00%) | 225 | 168 | 1275 | 210 | 0.7 | 290 | | | GET_thread_list | 1296 | 0(0.00%) | 489 | 181 | 1104 | 480 | 0.6 | 830 | | | PATCH_comment | 45 | 0(0.00%) | 490 | 221 | 2060 | 450 | 0 | 780 | | | PATCH_thread | 53 | 0(0.00%) | 489 | 174 | 1912 | 410 | 0.1 | 1100 | | Small | GET_comment_list | 877 | 0(0.00%) | 211 | 154 | 1072 | 200 | 0.3 | 300 | | | GET_thread | 55710 | 2(0.00%) | 198 | 141 | 1562 | 190 | 28.2 | 260 | | | GET_thread_list | 29694 | 0(0.00%) | 449 | 147 | 3919 | 430 | 13.2 | 810 | | | PATCH_comment | 875 | 4(0.46%) | 340 | 184 | 1584 | 320 | 0.2 | 560 | | | PATCH_thread | 1038 | 9(0.86%) | 259 | 150 | 1055 | 240 | 0.5 | 420 |
|
In a new test, it was found that SMES/ASLCx/1T2015 (1700 Threads, 3047 comments) was slowing down over time. To identify what was happening, a new course was seeded which we will call DAPI (1000 Threads, 500 comments). In the analysis of the forums, we saw that the median body size was 250 characters. Our PATCH operations can be 4, 250, 1000, 5000, or 10000 characters. This test was run over 10 hours. The decrease in response time needed to be addressed.
| Expand |
|---|
| title | page size vs response time |
|---|
|
Over time DAPI (1000 threads, 500 comments) slowed down over time. 
| Name | reqs | # | fails | Avg | Min | Max | Median | req/s | 95% |
|---|
| GET | GET_comment_list | 15793 | 0(0.00%) | 220 | 162 | 3116 | 200 | 0.2 | 340 | | GET | GET_thread | 955097 | 1(0.00%) | 214 | -8 | 4509 | 200 | 23.8 | 290 | | GET | GET_thread_list | 510921 | 0(0.00%) | 445 | 15 | 5595 | 420 | 11.6 | 810 | | PATCH | PATCH_comment | 15792 | 0(0.00%) | 337 | 187 | 3851 | 320 | 0.2 | 490 | | PATCH | PATCH_thread | 17333 | 164(0.94%) | 273 | 161 | 3019 | 250 | 0.5 | 400 |
When running a similar test with a more verbose output, we clearly saw that the page size had an effect on the response time. Although not shown here, larger content size was somewhat correlated to slower response times, but page_size had the better correlation. | Name | reqs | # | fails | Avg | Min | Max | Median | req/s | 95% |
|---|
| GET | page=1&page_size=100 | 238 | 0(0.00%) | 791 | 655 | 1990 | 750 | 0.4 | 960 | | GET | page=2&page_size=100 | 282 | 0(0.00%) | 762 | 659 | 1526 | 730 | 0.4 | 910 | | GET | page=1&page_size=75 | 240 | 0(0.00%) | 651 | 544 | 1038 | 620 | 0.2 | 830 | | GET | page=2&page_size=75 | 255 | 0(0.00%) | 677 | 573 | 1628 | 650 | 0 | 820 | | GET | page=7&page_size=100 | 260 | 0(0.00%) | 568 | 461 | 2637 | 520 | 0.4 | 710 | | GET | page=6&page_size=100 | 232 | 0(0.00%) | 559 | 466 | 843 | 520 | 0.1 | 710 | | GET | page=9&page_size=100 | 222 | 0(0.00%) | 568 | 460 | 1907 | 530 | 0.5 | 700 | | GET | page=3&page_size=100 | 262 | 0(0.00%) | 556 | 462 | 1807 | 520 | 0.2 | 700 | | GET | page=5&page_size=100 | 259 | 0(0.00%) | 556 | 463 | 1375 | 510 | 0.3 | 700 | | GET | page=4&page_size=100 | 276 | 0(0.00%) | 555 | 456 | 840 | 520 | 0.2 | 690 | | GET | page=8&page_size=100 | 236 | 0(0.00%) | 548 | 468 | 903 | 520 | 0.1 | 680 | | GET | page=3&page_size=75 | 251 | 0(0.00%) | 543 | 445 | 1348 | 510 | 0.3 | 670 | | GET | page=2&page_size=50 | 278 | 0(0.00%) | 507 | 415 | 1141 | 480 | 0.5 | 670 | | GET | page=1&page_size=50 | 251 | 0(0.00%) | 512 | 425 | 1043 | 490 | 0.1 | 660 | | GET | page=8&page_size=75 | 237 | 0(0.00%) | 481 | 397 | 1644 | 440 | 0.1 | 650 | | GET | page=3&page_size=50 | 247 | 0(0.00%) | 523 | 451 | 903 | 500 | 0.4 | 640 | | GET | page=6&page_size=75 | 238 | 0(0.00%) | 492 | 398 | 1634 | 450 | 0.3 | 640 | | GET | page=4&page_size=75 | 218 | 0(0.00%) | 480 | 395 | 886 | 440 | 0.2 | 630 | | GET | page=7&page_size=75 | 227 | 0(0.00%) | 474 | 391 | 795 | 440 | 0.5 | 620 | | GET | page=5&page_size=75 | 239 | 0(0.00%) | 466 | 388 | 1394 | 440 | 0.2 | 610 | | GET | page=4&page_size=50 | 237 | 0(0.00%) | 453 | 371 | 1010 | 420 | 0.2 | 610 | | GET | page=9&page_size=75 | 216 | 0(0.00%) | 467 | 397 | 996 | 430 | 0.4 | 600 | | GET | page=6&page_size=50 | 252 | 0(0.00%) | 391 | 321 | 1065 | 360 | 0.4 | 540 | | GET | page=5&page_size=25 | 249 | 0(0.00%) | 392 | 323 | 1367 | 370 | 0.4 | 530 | | GET | page=7&page_size=50 | 243 | 0(0.00%) | 392 | 319 | 997 | 360 | 0.4 | 530 | | GET | page=9&page_size=50 | 221 | 0(0.00%) | 391 | 330 | 658 | 360 | 0 | 520 | | GET | page=8&page_size=50 | 241 | 0(0.00%) | 388 | 317 | 1300 | 360 | 0.4 | 520 | | GET | page=5&page_size=50 | 204 | 0(0.00%) | 386 | 322 | 723 | 360 | 0.1 | 520 | | GET | page=2&page_size=25 | 259 | 0(0.00%) | 363 | 299 | 1124 | 340 | 0.4 | 520 | | GET | page=1&page_size=25 | 242 | 0(0.00%) | 373 | 296 | 1142 | 350 | 0.4 | 510 | | GET | page=4&page_size=25 | 238 | 0(0.00%) | 361 | 292 | 1171 | 340 | 0.3 | 500 | | GET | page=6&page_size=25 | 256 | 0(0.00%) | 365 | 312 | 687 | 350 | 0.3 | 480 | | GET | page=3&page_size=25 | 232 | 0(0.00%) | 356 | 302 | 752 | 340 | 0.9 | 480 | | GET | page=7&page_size=25 | 261 | 0(0.00%) | 362 | 310 | 590 | 350 | 0.1 | 470 | | GET | page=9&page_size=25 | 274 | 0(0.00%) | 304 | 254 | 797 | 280 | 0.3 | 430 | | GET | page=8&page_size=25 | 232 | 0(0.00%) | 300 | 253 | 644 | 280 | 0 | 420 | | GET | page=8&page_size=1 | 235 | 0(0.00%) | 228 | 191 | 1025 | 210 | 0 | 320 | | GET | page=3&page_size=1 | 255 | 0(0.00%) | 235 | 185 | 1106 | 220 | 0.2 | 310 | | GET | page=9&page_size=1 | 248 | 0(0.00%) | 234 | 184 | 1147 | 210 | 0.3 | 310 | | GET | page=5&page_size=1 | 247 | 0(0.00%) | 233 | 186 | 851 | 220 | 0.5 | 300 | | GET | page=7&page_size=1 | 231 | 0(0.00%) | 234 | 188 | 2104 | 220 | 0.3 | 290 | | GET | page=2&page_size=1 | 251 | 0(0.00%) | 226 | 187 | 972 | 220 | 0.4 | 290 | | GET | page=1&page_size=1 | 255 | 0(0.00%) | 223 | 186 | 461 | 210 | 0.3 | 290 | | GET | page=4&page_size=1 | 260 | 0(0.00%) | 222 | 188 | 400 | 210 | 0.3 | 290 | | GET | page=6&page_size=1 | 266 | 0(0.00%) | 221 | 187 | 373 | 220 | 0.1 | 280 |
|
This tests takes the course BerkeleyX/ColWri2.2x/1T2014 (~38,000 threads, ~40,000 comments) and then retrieves a single thread. This thread is then grouped by its comment_count. This test was run for 1 hour. As expected, because of pagination issues, threads with more comments are taking longer to return.
| Expand |
|---|
| title | GET thread vs comment_count |
|---|
|
Note that the slow transactions are the same thread. Comment_count is also linearly correlated. There is a case where the comment_count was ~4000. The approximate formula to calculate the response time for this context is (comment_count*1.4+360). A thread with ~4000 comments would take approximately 8200ms.  Image Removed Image Added
| Name | reqs | # | fails | Avg | Min | Max | Median | req/s | 95% |
|---|
| GET | Comment_Range:0-25 | 21567 | 0(0.00%) | 330 | 263 | 2050 | 320 | 5.7 | 420 | | GET | Comment_Range:25-50 | 3648 | 0(0.00%) | 358 | 280 | 932 | 340 | 1.1 | 460 | | GET | Comment_Range:50-75 | 802 | 0(0.00%) | 391 | 308 | 919 | 370 | 0.7 | 510 | | GET | Comment_Range:75-100 | 446 | 0(0.00%) | 424 | 325 | 859 | 400 | 0.1 | 570 | | GET | Comment_Range:100-125 | 1114 | 0(0.00%) | 444 | 349 | 1613 | 420 | 0.2 | 590 | | GET | Comment_Range:125-150 | 56 | 0(0.00%) | 486 | 391 | 762 | 470 | 0 | 700 | | GET | Comment_Range:150-175 | 1745 | 0(0.00%) | 518 | 403 | 1029 | 500 | 0.3 | 720 | | GET | Comment_Range:175-200 | 1610 | 0(0.00%) | 532 | 400 | 1487 | 510 | 0.5 | 730 | | GET | Comment_Range:225-250 | 853 | 0(0.00%) | 595 | 456 | 1389 | 570 | 0.2 | 830 | | GET | Comment_Range:350-375 | 850 | 0(0.00%) | 740 | 560 | 1853 | 700 | 0.1 | 1000 | | GET | Comment_Range:375-400 | 828 | 0(0.00%) | 772 | 573 | 1926 | 720 | 0.3 | 1100 | | GET | Comment_Range:1575-1600 | 879 | 0(0.00%) | 2194 | 1740 | 4731 | 2100 | 0.2 | 3100 |
|
...