With the addition of new query param (i.e. requested_fields=profile_image) to discussion API GET endpoints for thread and comment; the discussion APIs got exposed to the possibility of performance degradation because this new param fetches and prepares the profile image related data of all the users that are participating in thread or comment.
We have gathered and compared two sets of load test results. The former results (i.e. with profile_image) are run on the latest changes and the later are run on older implementation (i.e. without profile_image).
There appears to be anomaly in the results below (highlighted 99%) where the new implementation is faster than the old. Where as, for other results the percentile of new implementation increases significantly with the increase in load. For example, the first pair of results where no. of clients = 48, the difference is in tens or hundrend and for the last pair of results where no. of clients = 610, the difference is in thousands.
Logically; there should only be increase in response time for GET endpoints, but we see in results that all other endpoints show differences too. This is because all PATCH, POST and DELETE endpoints first call GET endpoint to retrieve an 'id' of thread and/or comment and then make further processing on it.
New Relic:
Here is the new relic permaLink_without_profile and permaLink_with_profile where the average rpm for former is 1.55k and for later is 1.48k and 84.5% of requests are made for AccountViewSet.list (i.e. user accounts API for multiple usernames) for later.
...
...
...
No. of clients =
req/s =
| Methods | median response time | 95% | 99% |
|---|
DELETE_comment | | | |
DELETE_thread | | | |
GET_comment_list | | | |
GET_thread | | | |
GET_thread_list | | | |
PATCH_comment | | | |
PATCH_thread | | | |
POST_comment_comment | | | |
POST_comment_response | | | |
POST_thread | | | |
auto_auth | | | |
No. of clients =
req/s =
| Methods | median response time | 95% | 99% |
|---|
DELETE_comment | | | |
DELETE_thread | | | |
GET_comment_list | | | |
GET_thread | | | |
GET_thread_list | | | |
PATCH_comment | | | |
PATCH_thread | | | |
POST_comment_comment | | | |
POST_comment_response | | | |
POST_thread | | | |
auto_auth | | | |
No. of clients =
req/s =
| Methods | median response time | 95% | 99% |
|---|
| DELETE_comment | | | |
| DELETE_thread | | | |
| GET_comment_list | | | |
| GET_thread | | | |
| GET_thread_list | | | |
| PATCH_comment | | | |
| PATCH_thread | | | |
| POST_comment_comment | | | |
| POST_comment_response | | | |
| POST_thread | | | |
| auto_auth | | | |
No. of clients =
req/s =
...
No. of clients =
req/s =
| Methods | median response time | 95% | 99% |
|---|
DELETE_comment | | | |
DELETE_thread | | | |
GET_comment_list | | | |
GET_thread | | | |
GET_thread_list | | | |
PATCH_comment | | | |
PATCH_thread | | | |
POST_comment_comment | | | |
POST_comment_response | | | |
POST_thread | | | |
auto_auth | | | |
No. of clients =
req/s =
| Methods | median response time | 95% | 99% |
|---|
DELETE_comment | | | |
DELETE_thread | | | |
GET_comment_list | | | |
GET_thread | | | |
GET_thread_list | | | |
PATCH_comment | | | |
PATCH_thread | | | |
POST_comment_comment | | | |
POST_comment_response | | | |
POST_thread | | | |
auto_auth | | | |
No. of clients =
req/s =
| Methods | median response time | 95% | 99% |
|---|
DELETE_comment | 270 | 350 | 1300 |
DELETE_thread | 180 | 210 | 280 |
GET_comment_list | 170 | 250 | 300 |
GET_thread | 170 | 230 | 260 |
GET_thread_list | 240 | 590 | 740 |
PATCH_comment | 270 | 350 | 450 |
PATCH_thread | 190 | 260 | 370 |
POST_comment_comment | 330 | 440 | 650 |
POST_comment_response | 280 | 350 | 400 |
POST_thread | 170 | 230 | 250 |
auto_auth | 220 | 230 | 230 |
No. of clients =
req/s =
| Methods | median response time | 95% | 99% |
|---|
DELETE_comment | 270 | 340 | 380 |
DELETE_thread | 180 | 240 | 360 |
GET_comment_list | 150 | 210 | 330 |
GET_thread | 160 | 220 | 300 |
GET_thread_list | 170 | 340 | 480 |
PATCH_comment | 260 | 340 | 370 |
PATCH_thread | 150 | 260 | 270 |
POST_comment_comment | 330 | 410 | 550 |
POST_comment_response | 280 | 360 | 440 |
POST_thread | 170 | 230 | 270 |
auto_auth | 210 | 210 | 210 |
No. of clients = 192
req/s = 16.10
...
DELETE_comment
...
DELETE_thread
...
GET_comment_list
...
GET_thread
...
GET_thread_list
...
PATCH_comment
...
PATCH_thread
...
POST_comment_comment
...
POST_comment_response
...
POST_thread
...
auto_auth
...
With the addition of new query param (i.e. requested_fields=profile_image) to discussion API GET endpoints for thread and comment; the discussion APIs got exposed to the possibility of performance degradation because this new param fetches and prepares the profile image related data of all the users that are participating in thread or comment.
We have gathered and compared two sets of load test results. The former results (i.e. with profile_image) are run on the latest changes and the later are run on older implementation (i.e. without profile_image).
There appears to be anomaly in the results below (highlighted 99%) where the new implementation is faster than the old. Where as, for other results the percentile of new implementation increases significantly with the increase in load. For example, the first pair of results where no. of clients = 48, the difference is in tens or hundrend and for the last pair of results where no. of clients = 610, the difference is in thousands.
Logically; there should only be increase in response time for GET endpoints, but we see in results that all other endpoints show differences too. This is because all PATCH, POST and DELETE endpoints first call GET endpoint to retrieve an 'id' of thread and/or comment and then make further processing on it.
New Relic:
Here is the new relic permaLink_without_profile and permaLink_with_profile where the average rpm for former is 1.55k and for later is 1.48k and 84.5% of requests are made for AccountViewSet.list (i.e. user accounts API for multiple usernames) for later.
| |
|---|
No. of clients = 192 req/s = 16 | Methods | median response time | 95% | 99% |
|---|
| | | | | DELETE_comment | 280 | 360 430 | | DELETE_thread | 180 | 290 290 | | GET_comment_list | 150 | 210 290 | | GET_thread | 160 | 220 310 | | GET_thread_list | 180 | 350 490 | | PATCH_comment | 250 | 340 450 | | PATCH_thread | 190 | 260 | 430 | POST_comment_comment | 330 | 430 | 550 | POST_comment_response | 280 | 370 | 470 | POST_thread | 170 | 230 290 | | auto_auth | 220 | 230 230 | |
| No. of clients = 240 req/s = 18.70 | Methods | median response time | 95% | 99% |
|---|
DELETE_comment | 290 | 390 470 | | DELETE_thread | 180 | 250 350 | | GET_comment_list | 180 | 270 400 | | GET_thread | 180 | 240 300 | | GET_thread_list | 240 | 620 800 | | PATCH_comment | 270 | 400 860 | | PATCH_thread | 200 | 270 2300 | | POST_comment_comment | 340 | 470 760 | | POST_comment_response | 290 | 390 760 | | POST_thread | 180 | 240 390 | | auto_auth | 220 | 230 230 | |
|
No. of clients = | 240req/s = | 19.7| Methods | median response time | 95% | 99% |
|---|
| DELETE_comment |
| 280 | 370 | 460 | 180 | 250 | 300 | 150 | 220 | 280 | 160 | 230 | 280 | 180 | 350 | 500 | 260 | 380 | 500 | 190 | 250 | 380 | 330 | 420 | 500 | 280 | 370 | 460 | 180 | 240 | 340 | 210 | 220 | 220
| No. of clients = | 288req/s = | 23.20| Methods | median response time | 95% | 99% |
|---|
| DELETE_comment |
| 290 | 380 | 460 | 190 | 290 | 380 | 180 | 270 | 340 | 180 | 240 | 300 | 250 | 630 | 790 | 260 | 350 | 440 | 170 | 240 | 280 | 340 | 430 | 560 | 290 | 390 | 470 | 180 | 240 | 320 | 210 | 210 | 210
|
No. of clients = 288 req/s = 22.2 | Methods | median response time | 95% | 99% |
|---|
DELETE_comment | 280 | 420 | 1400 | DELETE_thread | 180 | 240 270 | | GET_comment_list | 160 | 240 | 660 | GET_thread | 160 | 240 | 490 | GET_thread_list | 180 | 360 570 | | PATCH_comment | 270 | 360 | 740 | PATCH_thread | 190 | 260 | 630 | POST_comment_comment | 340 | 450 | 620 | POST_comment_response | 290 | 440 | 1300 | POST_thread | 180 | 240 | 1500 | auto_auth | 220 220 | 220 | |
| No. of clients = 336 req/s = 26.70 | Methods | median response time | 95% | 99% |
|---|
DELETE_comment | 290 | 410 | 570 | DELETE_thread | 190 | 320 1200 | | GET_comment_list | 190 | 270 | 370 | GET_thread | 180 | 250 | 350 | GET_thread_list | 250 | 630 800 | | PATCH_comment | 240 | 370 | 410 | PATCH_thread | 180 | 270 | 330 | POST_comment_comment | 350 | 460 | 560 | OSTPOST_comment_response | 300 | 390 | 470 | POST_thread | 180 | 250 320 | | auto_auth | 240 | 250 250 | |
|
No. of clients = 336 req/s = 28.2 | Methods | median response time | 95% | 99% |
|---|
DELETE_comment | 290 | 440 | 2400 | DELETE_thread | 190 | 260 440 | | GET_comment_list | 160 | 240 | 550 | GET_thread | 170 | 240 | 690 | GET_thread_list | 180 | 370 540 | | PATCH_comment | 270 | 380 | 1700 | PATCH_thread | 190 | 280 | 1500 | POST_comment_comment | 360 | 480 | 2400 | OSTPOST_comment_response | 300 | 430 | 1300 | POST_thread | 180 | 240 290 | | auto_auth | 240 | 260 260 | |
| No. of clients = 384 req/s = 31 | Methods | median response time | 95% | 99% |
|---|
DELETE_comment | 300 | 400 890 | | DELETE_thread | 190 | 300 | 320 | GET_comment_list | 190 | 280 410 | | GET_thread | 180 | 260 360 | | GET_thread_list | 260 | 660 830 | | PATCH_comment | 230 | 380 | 430 | PATCH_thread | 200 | 260 400 | | POST_comment_comment | 360 | 500 590 | | POST_comment_response | 300 | 410 | 470 | POST_thread | 180 | 250 | 290 | auto_auth | 290 | 360 360 | |
|
No. of clients = 384 req/s = 31.4 | Methods | median response time | 95% | 99% |
|---|
DELETE_comment | 300 | 430 550 | | DELETE_thread | 190 | 250 | 720 | GET_comment_list | 160 | 240 350 | | GET_thread | 170 | 240 330 | | GET_thread_list | 190 | 360 510 | | PATCH_comment | 230 | 380 | 500 | PATCH_thread | 180 | 270 320 | | POST_comment_comment | 360 | 480 590 | | POST_comment_response | 100 | 430 | 540 | POST_thread | 180 | 250 | 370 | auto_auth | 210 | 270 270 | |
| No. of clients = 480 req/s = 36.80 | Methods | median response time | 95% | 99% |
|---|
DELETE_comment | 320 | 470 | 600 | DELETE_thread | 200 | 290 1300 | | GET_comment_list | 210 | 320 450 | | GET_thread | 190 | 280 410 | | GET_thread_list | 270 | 710 900 | | PATCH_comment | 290 | 450 | 530 | PATCH_thread | 200 | 300 | 380 | POST_comment_comment | 390 | 550 | 680 | POST_comment_response | 330 | 470 | 600 | POST_thread | 190 | 280 | 370 | auto_auth | 340 | 350 350 | |
|
No. of clients = 480 req/s = 38.8 | Methods | median response time | 95% | 99% |
|---|
DELETE_comment | 320 | 440 | 860 | DELETE_thread | 200 | 300 | 1300 | GET_comment_list | 170 | 260 | 390 | GET_thread | 170 | 260 | 370 | GET_thread_list | 190 | 390 | 550 | PATCH_comment | 280 | 420 | 640 | PATCH_thread | 200 | 290 | 410 | POST_comment_comment | 380 | 530 | 1400 | POST_comment_response | 320 | 450 | 610 | POST_thread | 190 | 260 | 410 | auto_auth | 260 | 270 270 | |
| No. of clients = | 510req/s = | 41.40| Methods | median response time | 95% | 99% |
|---|
DELETE_comment |
| 320 | 480 | 570 | 200 | 340 | 470 | 210 | 330 | 440 | 190 | 290 | 390 | 280 | 710 | 880 | 280 | 400 | 460 | 190 | 320 | 420 | 390 | 560 | 690 | 330 | 490 | 590 | 190 | 270 | 400 | 230 | 250 | 250
|
No. of clients = | 510req/s = | 40.4| Methods | median response time | 95% | 99% |
|---|
DELETE_comment |
| 320 | 510 | 780 | 190 | 250 | 360 | 170 | 280 | 450 | 180 | 280 | 630 | 200 | 400 | 640 | 290 | 460 | 740 | 200 | 340 | 850 | 400 | 580 | 820 | 330 | 510 | 780 | 190 | 270 | 460 | 240 | 250 | 250
| No. of clients = | 544req/s = | 28.10| Methods | median response time | 95% | 99% |
|---|
DELETE_comment |
| 340 | 1500 | 4700 | 210 | 1900 | 2900 | 220 | 1400 | 3200 | 200 | 2100 | 5400 | 290 | 1600 | 3300 | 290 | 1800 | 3200 | 220 | 1200 | 2900 | 410 | 1900 | 3800 | 350 | 1800 | 3800 | 200 | 1400 | 2900 | 260 | 270 | 270
|
No. of clients = | 544req/s = | 43.50| Methods | median response time | 95% | 99% |
|---|
DELETE_comment |
| 320 | 490 | 710 | 200 | 340 | 500 | 170 | 280 | 550 | 180 | 300 | 870 | 210 | 420 | 700 | 290 | 500 | 670 | 200 | 340 | 440 | 400 | 600 | 840 | POST | 340 | 530 | 1000 | 190 | 290 | 590 | 210 | 250 | 250
| No. of clients = | 578req/s = | 45.90| Methods | median response time | 95% | 99% |
|---|
DELETE_comment |
| 370 | 2100 | 5500 | 210 | 950 | 1600 | 250 | 1700 | 4100 | 210 | 2800 | 5700 | 310 | 1800 | 4200 | 330 | 1100 | 2600 | 230 | 1700 | 4200 | 440 | 1900 | 5700 | POST | 380 | 2000 | 5400 | 210 | 1700 | 5300 | 530 | 5300 | 5300
|
No. of clients = | 578req/s = | 47.7| Methods | median response time | 95% | 99% |
|---|
DELETE_comment |
| 330 | 540 | 2300 | 200 | 320 | 470 | 180 | 310 | 1100 | 180 | 330 | 1900 | 210 | 450 | 1200 | 300 | 610 | 3600 | 210 | 360 | 940 | 400 | 600 | 1300 | 340 | 550 | 1400 | 190 | 310 | 1200 | 240 | 280 | 280
| No. of clients = | 610req/s = | 36| Methods | median response time | 95% | 99% |
|---|
| DELETE_comment |
| 1500 | 5600 | 7600 | 1300 | 5400 | 6500 | 1200 | 5300 | 6600 | 1900 | 6700 | 9100 | 1300 | 5400 | 6900 | 1400 | 6000 | 6800 | 1100 | 4500 | 5900 | 1600 | 6000 | 7600 | 1300 | 5600 | 7600 | 1200 | 5200 | 6400 | 1300 | 5800 | 5800
|
No. of clients = | 610req/s = | 48.9| Methods | median response time | 95% | 99% |
|---|
DELETE_comment |
| 360 | 630 | 1900 | 210 | 650 | 1600 |
GET_comment_list | 180 | 370 | 1300 |
GET_thread | 190 | 400 | 2100 |
GET_thread_list | 220 | 500 | 1200 |
PATCH_comment | 310 | 610 | 1100 |
PATCH_thread | 220 | 480 | 1400 |
POST_comment_comment | 430 | 890 | 2400 |
POST_comment_response | 360 | 680 | 1800 |
POST_thread | 200 | 420 | 1800 |
auto_auth | 270 | 430 | 430 |
UPDATE:
To narrow down the behaviour of feature endpoints on high traffic and huge data, I have conducted some more tests and here are the results.
Case 1:
I ran a few tests with a fresh new course for each run and initial data of 100 threads, 10 responses each thread and 7 comments to each response in each course. The percentile shows acceptable numbers as opposed to above results where data was increasing in a single course with each run (see above "with profile image" column). Hence we know the increasing data in any course has directly proportional effect to response percentile.
...
| | | GET_comment_list | | | | GET_thread | | | | GET_thread_list | | | | PATCH_comment | | | | PATCH_thread | | | | POST_comment_comment | | | | POST_comment_response | | | | POST_thread | | | | auto_auth | | | |
| No. of clients = req/s = | Methods | median response time | 95% | 99% |
|---|
| DELETE_comment |
| 200 | 300 | 380 | 190 | 270 | 360 |
| GET_thread_list | 270 | 680 | 850 |
| PATCH_comment | 220 | 400 | 500 |
| PATCH_thread | 210 | 290 | 330 |
| POST_comment_comment | 380 | 520 | 590 |
| POST_comment_response | 330 | 460 | 540 |
| POST_thread | 190 | 260 | 300 |
| auto_auth | 240 | 320 | 320 |
No. of clients = 510: (error rate = 560 (1.61%) )
...
No. of clients = 544: (error rate = 364 (1.57%) )
...
Case 2:
I had created a new fresh course populated in it a huge number of threads, responses and comments (in thousands) and then run GET thread and comment endpoints for both with profile image and without (as these are the only endpoints this changes is reflected in).
Comparing the two set of results; we can see the difference in 99% for with profile image but I believe the numbers are acceptable, only the last two cases that I have highlighted shows anomaly.
- For profile image: when no. of clients = 510 shows greater response time than no. of clients = 544. Its reason that I could assume is the users involved in thread and comments for later test are less in number possibly.
- For without profile image:
- 99% for no. of clients = 544 is greater than that of with profile image which is weird.
- there is an instant rise for clients=544 than for clients=510; to see if its a valid increase I used no. of clients somewhere between the two numbers i.e. 522, but the 99% was even higher than for 544, again weird.
| With Profile Image | Without Profile Image| _list | | | | | PATCH_comment | | | | | PATCH_thread | | | | | POST_comment_comment | | | | | POST_comment_response | | | | | POST_thread | | | | | auto_auth | | | |
|
|---|
No. of clients = req/s = | Methods | median response time | 95% | 99% |
|---|
DELETE_comment | | | | DELETE_thread | | | | GET_comment_list | | | | GET_thread | | | | GET_thread_list | | | | PATCH_comment | | | | PATCH_thread | | | | POST_comment_comment | | | | POST_comment_response | | | | POST_thread | | | | auto_auth | | | |
| No. of clients = req/s = | Methods | median response time | 95% | 99% |
|---|
DELETE_comment | | | | DELETE_thread | | | | GET_comment_list | | | | GET_thread | | | | GET_thread_list | | | | PATCH_comment | | | | PATCH_thread | | | | POST_comment_comment | | | | POST_comment_response | | | | POST_thread | | | | auto_auth | | | |
|
No. of clients = req/s = | Methods | median response time | 95% | 99% |
|---|
DELETE_comment | | | | DELETE_thread | | | | GET_comment_list | | | | GET_thread | | | | GET_thread_list | | | | PATCH_comment | | | | PATCH_thread | | | | POST_comment_comment | | | | POST_comment_response | | | | POST_thread | | | | auto_auth | | | |
| No. of clients = | 48req/s = | 4.10error rate = 1(0.03%)
Methods | median response time | 95% | 99%| Methods | median response time | 95% | 99% |
|---|
DELETE_comment | | | | DELETE_thread | | | | GET_comment_list |
| 210 | 290 | 370 | 170 | 230 | 260 | 250 | 650 | 780 | auto | auth | 210 | 220 | 220 | No. of clients = 48
req/s = 4
error rate = 0
|---|
| Methods | median response time | 95% | 99% |
|---|
| GET_comment_list | 170 | 220 | 250 |
| GET_thread | 170 | 230 | 250 |
GET_thread_list | 180 | 360 | 480 | | PATCH_thread | | | | POST_comment_comment | | | | POST_comment_response | | | | POST_thread | | | | auto_auth |
| 210 | 220 | 220 | No. of clients = | 144req/s = | 12 | error rate = 0
| Methods | median response time |
|---|
| 95% | 99% | 95% | 99% |
|---|
DELETE_comment | | | | DELETE_thread | | | | GET_comment_list |
| 210 | 290 | 360 | 170 | 240 | 270 | 250 | 690 | 830 | auto | auth | 230 | 230 | 230 | No. of clients = 144
req/s = 11.4
error rate = 0
|---|
| Methods | median response time | 95% | 99% |
|---|
| GET_comment_list | 160 | 230 | 280 |
| GET_thread | 170 | 240 | 260 |
GET_thread_list | 180 | 360 | 500 | | PATCH_thread | | | | POST_comment_comment | | | | POST_comment_response | | | | POST_thread | | | | auto_auth |
| 220 | 290 | 290 | No. of clients = | 288req/s = | 23.9error rate = 0| Methods | median response time | 95% | 99% |
|---|
DELETE_comment | | | | DELETE_thread | | | | GET_comment_list |
| 220 | 310 | 420 | 180 | 240 | 280 | 260 | 710 | 870 | auto | auth | 220 | 230 | 230 | No. of clients = 288
req/s = 23.3
error rate = 0
| Methods | median response time | 95% | 99% |
|---|
| GET_comment_list | 170 | 230 | 310 |
| GET_thread | 180 | 240 | 280 |
GET_thread_list | 190 | 370 | 510 | | PATCH_thread | | | | POST_comment_comment | | | | POST_comment_response | | | | POST_thread | | | | auto_auth |
| 230 | 230 | 230 | No. of clients = | 510req/s = | 42.3error rate = 1(0.00%)| Methods | median response time | 95% | 99% |
|---|
DELETE_comment | | | | DELETE_thread | | | | GET_comment_list |
| 240 | 410 | 1900 | 190 | 360 | 2600 | 290 | 880 | 2000 |
| auto_auth | 300 | 390 | 390 |
No. of clients = 510
req/s = 40.8
error rate = 0
| Methods | median response time | 95% | 99% |
|---|
| GET_comment_list | 180 | 270 | 580 |
| GET_thread | 190 | 290 | 860 |
GET_thread_list | 200 | 420 | 640 | | | | PATCH_comment | | | | PATCH_thread | | | | POST_comment_comment | | | | POST_comment_response | | | | POST_thread | | | | auto_auth |
| 320 | 1300 | 1300 | No. of clients = | 544req/s = | 44.40 | error rate = 0
| Methods | median response time | 95% | 99% |
|---|
DELETE_comment | | | | DELETE_thread | | | | GET_comment_list |
| 250 | 360 | 480 | 200 | 290 | 390 | 290 | 810 | 970 | | auto_auth | 280 | 310 | 310 |
No. of clients = 544
req/s = 45.4
error rate = 0
| Methods | median response time | 95% | 99% |
|---|
| GET_comment_list | 190 | 420 | 2500 |
| GET_thread | 200 | 480 | 4500 |
GET_thread_list | 210 | 550 | 2500 | | PATCH_comment | | | | PATCH_thread | | | | POST_comment_comment | | | | POST_comment_response | | | | POST_thread | | | | auto_auth |
| 280 | 290 | 290 |