Page Comparison

...

Old endpoint: /api/course_structure/v0/courses/{course_id}/blocks+navigation
New endpoint:
/api/courses/v1/blocks/?course_id={course_id}&user={user}&depth=all&requested_fields=graded&requested_fields=format&requested_fields=student_view_multi_device&block_counts=video&student_view_data=video
The loadtest environment has been set up differently which have better response times for mongo queries.
There may have been some performance changes since the test. To adjust for this, we will rerun the tests for the old endpoint.
course-v1:edx+DemoX.1+LT_SPLIT and course-v1:MITx+6.002x_6x+1T2015 were not available this time around so equivalent courses were used as the replacement.
The old tests did not use the edx fork of locust, the new test will also not be using the edx fork.
Numbers of users: 2
Wait time: 1000ms
Endpoint/courses are requested on an even distribution.

Raw dump, format later:

Notes:

Nit: Need to modify test since the course/endpoint didn't cycle properly.

HTTP Request	Course	API	# Request	Avg	Median	95%
GET	MITx/6.002x/2013_Spring	Old	159	4554	4500	4900
GET	MITx/6.002x/2013_Spring	New	458	599	540	700
GET	course-v1:MITx+6.002x+2013	Old	159	9595	9500	9900
GET	course-v1:MITx+6.002x+2013	New	458	1768	200	1700
GET	edX/DemoX.1/2014	Old	159	254	230	330
GET	edX/DemoX.1/2014	New	458	71	63	180
GET	course-v1:edX+DemoX.1+split	Old	159	254	230	1100
GET	course-v1:edX+DemoX.1+split	New	458	217	200	360

X rays:

Old: https://rpm.newrelic.com/accounts/88178/key_transactions/14280/x_rays/1909

New: https://rpm.newrelic.com/accounts/88178/key_transactions/15622/x_rays/1910

Other data:

Below is a step test for the new API.

Code Block

title	locustauto command and YAML settings

/usr/local/bin/locustauto --load_time 200 --ramp_up_time 200 --max_users 450 --max_user_hatch_rate 450 --steps 15 --directory courses_api --host=https://courses-loadtest.edx.org
{'LOCUST_TASK_SET': 'NewApiTask', 'COURSE': 'demo_draft', 'RESOURCE': 'blocks+navigation locust --host=https://courses-loadtest.edx.org'}

users,	RPS,	50%,	66%,	75%,	80%,	90%,	95%,	98%,	99%,	100%
30,	16.40,	617,	620,	625,	645,	681,	723,	1507,	1935,	2104
60,	36.10,	648,	658,	677,	688,	740,	788,	849,	924,	1322
90,	49.10,	830,	916,	962,	995,	1042,	1135,	1207,	1255,	1444
120,	55.30,	1126,	1201,	1285,	1311,	1480,	1588,	1698,	1778,	2064
150,	57.10,	1564,	1742,	1847,	1889,	2074,	2249,	2399,	2474,	2824
180,	57.60,	2075,	2300,	2450,	2550,	2750,	2950,	3100,	3150,	3567
210,	25.30,	2700,	2850,	3025,	3125,	3400,	5050,	7800,	8524,	9367
240,	37.50,	6545,	7070,	7220,	7370,	7595,	7795,	8020,	8145,	8447
270,	49.10,	5854,	6804,	7230,	7554,	8155,	8529,	9155,	9455,	10455
300,	55.20,	4374,	4949,	5474,	8774,	11249,	16749,	29501,	34249,	35379
330,	39.40,	6676,	7176,	7501,	7676,	8051,	8551,	10202,	12677,	13705
360,	29.10,	4729,	6380,	7329,	7729,	9829,	11004,	12503,	12755,	20657
390,	14.20,	4046,	4771,	5121,	5446,	9545,	12244,	19492,	24998,	33624
420,	28.60,	6720,	7195,	7570,	7795,	8171,	9688,	11889,	14995,	15264
450,	16.00,	4489,	4914,	5414,	5964,	10238,	15023,	25535,	32014,	35936

Raw data:

Name # reqs # fails Avg Min Max | Median req/s
--------------------------------------------------------------------------------------------------------------------------------------------
GET GET MITx%2F6.002x%2F2013_Spring, using the New API 458 0(0.00%) 599 476 2571 | 540 0.10
GET GET MITx/6.002x/2013_Spring, using the Old API 159 0(0.00%) 4554 4234 6139 | 4500 0.10
GET GET course-v1%3AMITx%2B6.002x%2B2013, using the New API 457 0(0.00%) 1768 1517 5112 | 1700 0.10
GET GET course-v1%3AedX%2BDemoX.1%2Bsplit, using the New API 458 0(0.00%) 217 185 598 | 200 0.10
GET GET course-v1:MITx+6.002x+2013, using the Old API 159 0(0.00%) 9495 9017 12543 | 9500 0.00
GET GET course-v1:edX+DemoX.1+split, using the Old API 159 0(0.00%) 868 762 1300 | 830 0.00
GET GET edX%2FDemoX.1%2F2014, using the New API 457 0(0.00%) 71 57 246 | 63 0.10
GET GET edX/DemoX.1/2014, using the Old API 160 0(0.00%) 254 221 515 | 230 0.00
GET create user 1 0(0.00%) 352 352 352 | 350 0.00
--------------------------------------------------------------------------------------------------------------------------------------------
Total 2468 0(0.00%) 0.50

...

Percentage of the requests completed within given times
Name # reqs 50% 66% 75% 80% 90% 95% 98% 99% 100%
--------------------------------------------------------------------------------------------------------------------------------------------
GET GET MITx%2F6.002x%2F2013_Spring, using the New API 458 540 570 600 630 670 700 2300 2400 2571
GET GET MITx/6.002x/2013_Spring, using the Old API 159 4500 4600 4600 4700 4700 4900 5100 5400 6139
GET GET course-v1%3AMITx%2B6.002x%2B2013, using the New API 457 1700 1700 1700 1800 1800 1900 4800 5000 5112
GET GET course-v1%3AedX%2BDemoX.1%2Bsplit, using the New API 458 200 200 210 210 270 360 440 550 598
GET GET course-v1:MITx+6.002x+2013, using the Old API 159 9500 9600 9600 9700 9700 9900 10000 10000 12543
GET GET course-v1:edX+DemoX.1+split, using the Old API 159 830 880 900 940 1000 1100 1100 1200 1300
GET GET edX%2FDemoX.1%2F2014, using the New API 457 63 65 66 68 80 150 180 210 246
GET GET edX/DemoX.1/2014, using the Old API 160 230 250 280 300 320 320 330 450 515
GET create user 1 350 350 350 350 350 350 350 350 352
-----------------------------------------------------------------------------------------------------------------------------------------

X rays:

Old: https://rpm.newrelic.com/accounts/88178/key_transactions/14280/x_rays/1909New: https://rpm.newrelic.com/accounts/88178/key_transactions/15622/x_rays/1910

Versions Compared

Old Version 7

New Version 8

Key