After performance changes

Owned by Christopher Lee (Deactivated)
Last updated: Oct 07, 2015

These tests are testing the same endpoints in block+navigation as well as the endpoint that will be replacing the old endpoints. There are some variations to this test.

Details:

  • Old endpoint: /api/course_structure/v0/courses/{course_id}/blocks+navigation
  • New endpoint: 

    /api/courses/v1/blocks/?course_id={course_id}&user={user}&depth=all&requested_fields=graded&requested_fields=format&requested_fields=student_view_multi_device&block_counts=video&student_view_data=video

  • The loadtest environment has been set up differently which have better response times for mongo queries. 
  • There may have been some performance changes since the test. To adjust for this, we will rerun the tests for the old endpoint.
  • course-v1:edx+DemoX.1+LT_SPLIT and course-v1:MITx+6.002x_6x+1T2015 were not available this time around so equivalent courses were used as the replacement.
  • The old tests did not use the edx fork of locust, the new test will also not be using the edx fork. 
  • Numbers of users: 2
  • Wait time: 1000ms
  • Endpoint/courses are requested on an even distribution.

 

Notes:

  • Nit: Need to modify test since the course/endpoint didn't cycle properly. 
HTTP RequestCourseAPI# Request# failsAvgMedian95%
GETMITx/6.002x/2013_SpringOld159 0455445004900
GETMITx/6.002x/2013_SpringNew4580 599540700
GETcourse-v1:MITx+6.002x+2013Old1590 959595009900
GETcourse-v1:MITx+6.002x+2013New4580 176817001700
GETedX/DemoX.1/2014Old1590 254230330
GETedX/DemoX.1/2014New4580 7163180
GETcourse-v1:edX+DemoX.1+splitOld1590 2542301100
GETcourse-v1:edX+DemoX.1+splitNew4580 217200360

 

X rays:

Old: https://rpm.newrelic.com/accounts/88178/key_transactions/14280/x_rays/1909

New: https://rpm.newrelic.com/accounts/88178/key_transactions/15622/x_rays/1910

Other data:

Below is a step test for the new API for commit 16557901e9386ef736c834e9ecab0ec6898861af

This test was run very quickly and may not represent accurate data. Analyze with caution.

locustauto command and YAML settings
/usr/local/bin/locustauto --load_time 200 --ramp_up_time 200 --max_users 450 --max_user_hatch_rate 450 --steps 15 --directory courses_api --host=https://courses-loadtest.edx.org
{'LOCUST_TASK_SET': 'NewApiTask', 'COURSE': 'demo_draft', 'RESOURCE': 'blocks+navigation locust --host=https://courses-loadtest.edx.org'}
usersRPS50%66%75%80%90%95%98%99%100%
3016.4617620625645681723150719352104
6036.16486586776887407888499241322
9049.183091696299510421135120712551444
12055.3112612011285131114801588169817782064
15057.1156417421847188920742249239924742824
18057.6207523002450255027502950310031503567
21025.3270028503025312534005050780085249367
24037.5654570707220737075957795802081458447
27049.15854680472307554815585299155945510455
30055.243744949547487741124916749295013424935379
33039.4667671767501767680518551102021267713705
36029.14729638073297729982911004125031275520657
39014.24046477151215446954512244194922499833624
42028.6672071957570779581719688118891499515264
4501644894914541459641023815023255353201435936


Raw data:

Name # reqs # fails Avg Min Max | Median req/s
--------------------------------------------------------------------------------------------------------------------------------------------
GET GET MITx%2F6.002x%2F2013_Spring, using the New API 458 0(0.00%) 599 476 2571 | 540 0.10
GET GET MITx/6.002x/2013_Spring, using the Old API 159 0(0.00%) 4554 4234 6139 | 4500 0.10
GET GET course-v1%3AMITx%2B6.002x%2B2013, using the New API 457 0(0.00%) 1768 1517 5112 | 1700 0.10
GET GET course-v1%3AedX%2BDemoX.1%2Bsplit, using the New API 458 0(0.00%) 217 185 598 | 200 0.10
GET GET course-v1:MITx+6.002x+2013, using the Old API 159 0(0.00%) 9495 9017 12543 | 9500 0.00
GET GET course-v1:edX+DemoX.1+split, using the Old API 159 0(0.00%) 868 762 1300 | 830 0.00
GET GET edX%2FDemoX.1%2F2014, using the New API 457 0(0.00%) 71 57 246 | 63 0.10
GET GET edX/DemoX.1/2014, using the Old API 160 0(0.00%) 254 221 515 | 230 0.00
GET create user 1 0(0.00%) 352 352 352 | 350 0.00
--------------------------------------------------------------------------------------------------------------------------------------------
Total 2468 0(0.00%) 0.50

1 0(0.00%) 352 352 352 | 350 0.00

Percentage of the requests completed within given times
Name # reqs 50% 66% 75% 80% 90% 95% 98% 99% 100%
--------------------------------------------------------------------------------------------------------------------------------------------
GET GET MITx%2F6.002x%2F2013_Spring, using the New API 458 540 570 600 630 670 700 2300 2400 2571
GET GET MITx/6.002x/2013_Spring, using the Old API 159 4500 4600 4600 4700 4700 4900 5100 5400 6139
GET GET course-v1%3AMITx%2B6.002x%2B2013, using the New API 457 1700 1700 1700 1800 1800 1900 4800 5000 5112
GET GET course-v1%3AedX%2BDemoX.1%2Bsplit, using the New API 458 200 200 210 210 270 360 440 550 598
GET GET course-v1:MITx+6.002x+2013, using the Old API 159 9500 9600 9600 9700 9700 9900 10000 10000 12543
GET GET course-v1:edX+DemoX.1+split, using the Old API 159 830 880 900 940 1000 1100 1100 1200 1300
GET GET edX%2FDemoX.1%2F2014, using the New API 457 63 65 66 68 80 150 180 210 246
GET GET edX/DemoX.1/2014, using the Old API 160 230 250 280 300 320 320 330 450 515
GET create user 1 350 350 350 350 350 350 350 350 352
-----------------------------------------------------------------------------------------------------------------------------------------