Page Comparison

...

Old endpoint: /api/course_structure/v0/courses/{course_id}/blocks+navigation
New endpoint:
/api/courses/v1/blocks/?course_id={course_id}&user={user}&depth=all&requested_fields=graded&requested_fields=format&requested_fields=student_view_multi_device&block_counts=video&student_view_data=video
The loadtest environment has been set up differently which have better response times for mongo queries.
There may have been some performance changes since the test. To adjust for this, we will rerun the tests for the old endpoint.
course-v1:edx+DemoX.1+LT_SPLIT and course-v1:MITx+6.002x_6x+1T2015 were not available this time around so equivalent courses were used as the replacement.
The old tests did not use the edx fork of locust, the new test will also not be using the edx fork.
Numbers of users: 2
Wait time: 1000ms
Endpoint/courses are requested on an even distribution.

Notes:

Nit: Need to modify test since the course/endpoint didn't cycle properly.

...

Raw dump, format later:

...

HTTP Request	Course	API	# Request	Avg	Median	95%
GET	MITx/6.002x/2013_Spring	Old	159	4554	4500	4900
GET	MITx/6.002x/2013_Spring	New	458	599	540	700
GET	course-v1:MITx+6.002x+2013	Old	159	9595	9500	9900
GET	course-v1:MITx+6.002x+2013	New	458	1768	1700	1700
GET	edX/DemoX.1/2014	Old	159	254	230	330
GET	edX/DemoX.1/2014	New	458	71	63	180
GET	course-v1:edX+DemoX.1+split	Old	159	254	230	1100
GET	course-v1:edX+DemoX.1+split	New	458	217	200	360

X rays:

Old: https://rpm.newrelic.com/accounts/88178/key_transactions/14280/x_rays/1909

New: https://rpm.newrelic.com/accounts/88178/key_transactions/15622/x_rays/1910

Other data:

Below is a step test for the new API for commit 16557901e9386ef736c834e9ecab0ec6898861af

This test was run very quickly and may not represent accurate data. Analyze with caution.

Code Block

title	locustauto command and YAML settings

/usr/local/bin/locustauto --load_time 200 --ramp_up_time 200 --max_users 450 --max_user_hatch_rate 450 --steps 15 --directory courses_api --host=https://courses-loadtest.edx.org
{'LOCUST_TASK_SET': 'NewApiTask', 'COURSE': 'demo_draft', 'RESOURCE': 'blocks+navigation locust --host=https://courses-loadtest.edx.org'}

users	RPS	50%	66%	75%	80%	90%	95%	98%	99%	100%
30	16.4	617	620	625	645	681	723	1507	1935	2104
60	36.1	648	658	677	688	740	788	849	924	1322
90	49.1	830	916	962	995	1042	1135	1207	1255	1444
120	55.3	1126	1201	1285	1311	1480	1588	1698	1778	2064
150	57.1	1564	1742	1847	1889	2074	2249	2399	2474	2824
180	57.6	2075	2300	2450	2550	2750	2950	3100	3150	3567
210	25.3	2700	2850	3025	3125	3400	5050	7800	8524	9367
240	37.5	6545	7070	7220	7370	7595	7795	8020	8145	8447
270	49.1	5854	6804	7230	7554	8155	8529	9155	9455	10455
300	55.2	4374	4949	5474	8774	11249	16749	29501	34249	35379
330	39.4	6676	7176	7501	7676	8051	8551	10202	12677	13705
360	29.1	4729	6380	7329	7729	9829	11004	12503	12755	20657
390	14.2	4046	4771	5121	5446	9545	12244	19492	24998	33624
420	28.6	6720	7195	7570	7795	8171	9688	11889	14995	15264
450	16	4489	4914	5414	5964	10238	15023	25535	32014	35936

Raw data:

Name # reqs # fails Avg Min Max | Median req/s
--------------------------------------------------------------------------------------------------------------------------------------------
GET GET MITx%2F6.002x%2F2013_Spring, using the New API 458 0(0.00%) 599 476 2571 | 540 0.10
GET GET MITx/6.002x/2013_Spring, using the Old API 159 0(0.00%) 4554 4234 6139 | 4500 0.10
GET GET course-v1%3AMITx%2B6.002x%2B2013, using the New API 457 0(0.00%) 1768 1517 5112 | 1700 0.10
GET GET course-v1%3AedX%2BDemoX.1%2Bsplit, using the New API 458 0(0.00%) 217 185 598 | 200 0.10
GET GET course-v1:MITx+6.002x+2013, using the Old API 159 0(0.00%) 9495 9017 12543 | 9500 0.00
GET GET course-v1:edX+DemoX.1+split, using the Old API 159 0(0.00%) 868 762 1300 | 830 0.00
GET GET edX%2FDemoX.1%2F2014, using the New API 457 0(0.00%) 71 57 246 | 63 0.10
GET GET edX/DemoX.1/2014, using the Old API 160 0(0.00%) 254 221 515 | 230 0.00
GET create user 1 0(0.00%) 352 352 352 | 350 0.00
--------------------------------------------------------------------------------------------------------------------------------------------
Total 2468 0(0.00%) 0.50

...

Percentage of the requests completed within given times
Name # reqs 50% 66% 75% 80% 90% 95% 98% 99% 100%
--------------------------------------------------------------------------------------------------------------------------------------------
GET GET MITx%2F6.002x%2F2013_Spring, using the New API 458 540 570 600 630 670 700 2300 2400 2571
GET GET MITx/6.002x/2013_Spring, using the Old API 159 4500 4600 4600 4700 4700 4900 5100 5400 6139
GET GET course-v1%3AMITx%2B6.002x%2B2013, using the New API 457 1700 1700 1700 1800 1800 1900 4800 5000 5112
GET GET course-v1%3AedX%2BDemoX.1%2Bsplit, using the New API 458 200 200 210 210 270 360 440 550 598
GET GET course-v1:MITx+6.002x+2013, using the Old API 159 9500 9600 9600 9700 9700 9900 10000 10000 12543
GET GET course-v1:edX+DemoX.1+split, using the Old API 159 830 880 900 940 1000 1100 1100 1200 1300
GET GET edX%2FDemoX.1%2F2014, using the New API 457 63 65 66 68 80 150 180 210 246
GET GET edX/DemoX.1/2014, using the Old API 160 230 250 280 300 320 320 330 450 515
GET create user 1 350 350 350 350 350 350 350 350 352
-----------------------------------------------------------------------------------------------------------------------------------------

X rays:

Old: https://rpm.newrelic.com/accounts/88178/key_transactions/14280/x_rays/1909New:

https://rpm.newrelic.com/accounts/88178/key_transactions/15622/x_rays/1910

Versions Compared

Old Version 4

New Version Current

Key