These tests are testing the same endpoints in block+navigation as well as the endpoint that will be replacing the old endpoints. There are some variations to this test.
Details:
- Old endpoint: /api/course_structure/v0/courses/{course_id}/blocks+navigation
- New endpoint:
/api/courses/v1/blocks/?course_id={course_id}&user={user}&depth=all&requested_fields=graded&requested_fields=format&requested_fields=student_view_multi_device&block_counts=video&student_view_data=video
- The loadtest environment has been set up differently which have better response times for mongo queries.
- There may have been some performance changes since the test. To adjust for this, we will rerun the tests for the old endpoint.
- course-v1:edx+DemoX.1+LT_SPLIT and course-v1:MITx+6.002x_6x+1T2015 were not available this time around so equivalent courses were used as the replacement.
- The old tests did not use the edx fork of locust, the new test will also not be using the edx fork.
- Numbers of users: 2
- Wait time: 1000ms
- Endpoint/courses are requested on an even distribution.
Notes:
- Nit: Need to modify test since the course/endpoint didn't cycle properly.
| HTTP Request | Course | API | # Request | # fails | Avg | Median | 95% |
|---|---|---|---|---|---|---|---|
| GET | MITx/6.002x/2013_Spring | Old | 159 | 0 | 4554 | 4500 | 4900 |
| GET | MITx/6.002x/2013_Spring | New | 458 | 0 | 599 | 540 | 700 |
| GET | course-v1:MITx+6.002x+2013 | Old | 159 | 0 | 9595 | 9500 | 9900 |
| GET | course-v1:MITx+6.002x+2013 | New | 458 | 0 | 1768 | 200 | 1700 |
| GET | edX/DemoX.1/2014 | Old | 159 | 0 | 254 | 230 | 330 |
| GET | edX/DemoX.1/2014 | New | 458 | 0 | 71 | 63 | 180 |
| GET | course-v1:edX+DemoX.1+split | Old | 159 | 0 | 254 | 230 | 1100 |
| GET | course-v1:edX+DemoX.1+split | New | 458 | 0 | 217 | 200 | 360 |
X rays:
Old: https://rpm.newrelic.com/accounts/88178/key_transactions/14280/x_rays/1909
New: https://rpm.newrelic.com/accounts/88178/key_transactions/15622/x_rays/1910
Other data:
Below is a step test for the new API.
/usr/local/bin/locustauto --load_time 200 --ramp_up_time 200 --max_users 450 --max_user_hatch_rate 450 --steps 15 --directory courses_api --host=https://courses-loadtest.edx.org
{'LOCUST_TASK_SET': 'NewApiTask', 'COURSE': 'demo_draft', 'RESOURCE': 'blocks+navigation locust --host=https://courses-loadtest.edx.org'}
| users | RPS | 50% | 66% | 75% | 80% | 90% | 95% | 98% | 99% | 100% |
|---|---|---|---|---|---|---|---|---|---|---|
| 30 | 16.4 | 617 | 620 | 625 | 645 | 681 | 723 | 1507 | 1935 | 2104 |
| 60 | 36.1 | 648 | 658 | 677 | 688 | 740 | 788 | 849 | 924 | 1322 |
| 90 | 49.1 | 830 | 916 | 962 | 995 | 1042 | 1135 | 1207 | 1255 | 1444 |
| 120 | 55.3 | 1126 | 1201 | 1285 | 1311 | 1480 | 1588 | 1698 | 1778 | 2064 |
| 150 | 57.1 | 1564 | 1742 | 1847 | 1889 | 2074 | 2249 | 2399 | 2474 | 2824 |
| 180 | 57.6 | 2075 | 2300 | 2450 | 2550 | 2750 | 2950 | 3100 | 3150 | 3567 |
| 210 | 25.3 | 2700 | 2850 | 3025 | 3125 | 3400 | 5050 | 7800 | 8524 | 9367 |
| 240 | 37.5 | 6545 | 7070 | 7220 | 7370 | 7595 | 7795 | 8020 | 8145 | 8447 |
| 270 | 49.1 | 5854 | 6804 | 7230 | 7554 | 8155 | 8529 | 9155 | 9455 | 10455 |
| 300 | 55.2 | 4374 | 4949 | 5474 | 8774 | 11249 | 16749 | 29501 | 34249 | 35379 |
| 330 | 39.4 | 6676 | 7176 | 7501 | 7676 | 8051 | 8551 | 10202 | 12677 | 13705 |
| 360 | 29.1 | 4729 | 6380 | 7329 | 7729 | 9829 | 11004 | 12503 | 12755 | 20657 |
| 390 | 14.2 | 4046 | 4771 | 5121 | 5446 | 9545 | 12244 | 19492 | 24998 | 33624 |
| 420 | 28.6 | 6720 | 7195 | 7570 | 7795 | 8171 | 9688 | 11889 | 14995 | 15264 |
| 450 | 16 | 4489 | 4914 | 5414 | 5964 | 10238 | 15023 | 25535 | 32014 | 35936 |
Raw data:
Name # reqs # fails Avg Min Max | Median req/s
--------------------------------------------------------------------------------------------------------------------------------------------
GET GET MITx%2F6.002x%2F2013_Spring, using the New API 458 0(0.00%) 599 476 2571 | 540 0.10
GET GET MITx/6.002x/2013_Spring, using the Old API 159 0(0.00%) 4554 4234 6139 | 4500 0.10
GET GET course-v1%3AMITx%2B6.002x%2B2013, using the New API 457 0(0.00%) 1768 1517 5112 | 1700 0.10
GET GET course-v1%3AedX%2BDemoX.1%2Bsplit, using the New API 458 0(0.00%) 217 185 598 | 200 0.10
GET GET course-v1:MITx+6.002x+2013, using the Old API 159 0(0.00%) 9495 9017 12543 | 9500 0.00
GET GET course-v1:edX+DemoX.1+split, using the Old API 159 0(0.00%) 868 762 1300 | 830 0.00
GET GET edX%2FDemoX.1%2F2014, using the New API 457 0(0.00%) 71 57 246 | 63 0.10
GET GET edX/DemoX.1/2014, using the Old API 160 0(0.00%) 254 221 515 | 230 0.00
GET create user 1 0(0.00%) 352 352 352 | 350 0.00
--------------------------------------------------------------------------------------------------------------------------------------------
Total 2468 0(0.00%) 0.50
1 0(0.00%) 352 352 352 | 350 0.00
Percentage of the requests completed within given times
Name # reqs 50% 66% 75% 80% 90% 95% 98% 99% 100%
--------------------------------------------------------------------------------------------------------------------------------------------
GET GET MITx%2F6.002x%2F2013_Spring, using the New API 458 540 570 600 630 670 700 2300 2400 2571
GET GET MITx/6.002x/2013_Spring, using the Old API 159 4500 4600 4600 4700 4700 4900 5100 5400 6139
GET GET course-v1%3AMITx%2B6.002x%2B2013, using the New API 457 1700 1700 1700 1800 1800 1900 4800 5000 5112
GET GET course-v1%3AedX%2BDemoX.1%2Bsplit, using the New API 458 200 200 210 210 270 360 440 550 598
GET GET course-v1:MITx+6.002x+2013, using the Old API 159 9500 9600 9600 9700 9700 9900 10000 10000 12543
GET GET course-v1:edX+DemoX.1+split, using the Old API 159 830 880 900 940 1000 1100 1100 1200 1300
GET GET edX%2FDemoX.1%2F2014, using the New API 457 63 65 66 68 80 150 180 210 246
GET GET edX/DemoX.1/2014, using the Old API 160 230 250 280 300 320 320 330 450 515
GET create user 1 350 350 350 350 350 350 350 350 352
-----------------------------------------------------------------------------------------------------------------------------------------