Page Comparison

These tests are testing the same endpoints in block+navigation as well as the endpoint that will be replacing the old endpoints. There are some variations to this test.

Details:

• Old endpoint: /api/course_structure/v0/courses/{course_id}/blocks+navigation
• New endpoint: 

  /api/courses/v1/blocks/?course_id={course_id}&user={user}&depth=all&requested_fields=graded&requested_fields=format&requested_fields=student_view_multi_device&block_counts=video&student_view_data=video

• The loadtest environment has been set up differently which have better response times for mongo queries. 
• There may have been some performance changes since the test. To adjust for this, we will rerun the tests for the old endpoint.
• course-v1:edx+DemoX.1+LT_SPLIT and course-v1:MITx+6.002x_6x+1T2015 were not available this time around so equivalent courses were used as the replacement.
• The old tests did not use the edx fork of locust, the new test will also not be using the edx fork. 
• Numbers of users: 2
• Wait time: 1000ms
• Endpoint/courses are requested on an even distribution.

Notes:

• Nit: Need to modify test since the course/endpoint didn't cycle properly. 

X rays:

Old: https://rpm.newrelic.com/accounts/88178/key_transactions/14280/x_rays/1909

New: https://rpm.newrelic.com/accounts/88178/key_transactions/15622/x_rays/1910

Other data:

Below is a step test for the new API for commit 16557901e9386ef736c834e9ecab0ec6898861af

This test was run very quickly and may not represent accurate data. Analyze with caution.

/usr/local/bin/locustauto --load_time 200 --ramp_up_time 200 --max_users 450 --max_user_hatch_rate 450 --steps 15 --directory courses_api --host=https://courses-loadtest.edx.org
{'LOCUST_TASK_SET': 'NewApiTask', 'COURSE': 'demo_draft', 'RESOURCE': 'blocks+navigation locust --host=https://courses-loadtest.edx.org'}

Raw data:

Name # reqs # fails Avg Min Max | Median req/s
--------------------------------------------------------------------------------------------------------------------------------------------
GET GET MITx%2F6.002x%2F2013_Spring, using the New API 458 0(0.00%) 599 476 2571 | 540 0.10
GET GET MITx/6.002x/2013_Spring, using the Old API 159 0(0.00%) 4554 4234 6139 | 4500 0.10
GET GET course-v1%3AMITx%2B6.002x%2B2013, using the New API 457 0(0.00%) 1768 1517 5112 | 1700 0.10
GET GET course-v1%3AedX%2BDemoX.1%2Bsplit, using the New API 458 0(0.00%) 217 185 598 | 200 0.10
GET GET course-v1:MITx+6.002x+2013, using the Old API 159 0(0.00%) 9495 9017 12543 | 9500 0.00
GET GET course-v1:edX+DemoX.1+split, using the Old API 159 0(0.00%) 868 762 1300 | 830 0.00
GET GET edX%2FDemoX.1%2F2014, using the New API 457 0(0.00%) 71 57 246 | 63 0.10
GET GET edX/DemoX.1/2014, using the Old API 160 0(0.00%) 254 221 515 | 230 0.00
GET create user 1 0(0.00%) 352 352 352 | 350 0.00
--------------------------------------------------------------------------------------------------------------------------------------------
Total 2468 0(0.00%) 0.50

1 0(0.00%) 352 352 352 | 350 0.00

Percentage of the requests completed within given times
Name # reqs 50% 66% 75% 80% 90% 95% 98% 99% 100%
--------------------------------------------------------------------------------------------------------------------------------------------
GET GET MITx%2F6.002x%2F2013_Spring, using the New API 458 540 570 600 630 670 700 2300 2400 2571
GET GET MITx/6.002x/2013_Spring, using the Old API 159 4500 4600 4600 4700 4700 4900 5100 5400 6139
GET GET course-v1%3AMITx%2B6.002x%2B2013, using the New API 457 1700 1700 1700 1800 1800 1900 4800 5000 5112
GET GET course-v1%3AedX%2BDemoX.1%2Bsplit, using the New API 458 200 200 210 210 270 360 440 550 598
GET GET course-v1:MITx+6.002x+2013, using the Old API 159 9500 9600 9600 9700 9700 9900 10000 10000 12543
GET GET course-v1:edX+DemoX.1+split, using the Old API 159 830 880 900 940 1000 1100 1100 1200 1300
GET GET edX%2FDemoX.1%2F2014, using the New API 457 63 65 66 68 80 150 180 210 246
GET GET edX/DemoX.1/2014, using the Old API 160 230 250 280 300 320 320 330 450 515
GET create user 1 350 350 350 350 350 350 350 350 352
-----------------------------------------------------------------------------------------------------------------------------------------