Part 1: Confirm performance factors for Discussion API

MA-1099 - Getting issue details... STATUS

Endpoints:

/courses/(course_id)/
- GET
/course_topics/(course_id)
- GET
/threads/
- GET
- POST
/threads/(thread_id)
/comments
- GET
- POST
/comments/(comment_id)
- GET not implemented
- PATCH
- DELETE

Notes and considerations:

Thread and Comment pool:

Various methods of select post data were considered.

Selecting threads from a smaller pool or selecting the same thread
- Rather than getting the entire list of
Retrieving the list of thread ids when starting locust.
- This was my initial

Staff vs. Normal User:

Using users with staff access was thrown into consideration as it would be make some of the permissions a bit more difficult for some discussion forums actions such as editing the body of a thread. Some tests were ran to see if there was a difference. No difference was found the tests that were designed to check for a difference.

/threads/

GET:

Invalid: Old test that was invalid due to the way mongo is set up

Using 10 locust users with min/max time of 1000/2000ms, GET requests for sent for either a single thread or randomly from a selection of 10% of the threads in a course. This was tested against a sandbox.

It seems like getting a random thread vs. the same thread does not make a reliably noticeable difference.
As number of posts in the database increase, the response time also increases.
Number of posts in a course does not seem to be matter for GETting a post.

Update: This conclusion is invalid as the sandbox does not have the proper indexes.

Approx Total Posts	Name	# requests	# fails	Median	Average	Min	Max	Content Size	# reqs/sec
1000	GET a random Thread out of 1000 Posts	2671	0	140	150	125.5698204	410.5699062	1178	5.8
1000	GET a single Thread out of 1000 Posts	2262	0	140	151	126.8491745	337.1069431	1178	5.9
11000	GET a random Thread out of 10000 Posts	1522	0	180	196	164.470911	447.6130009	1178	6
11000	GET a single Thread out of 10000 Posts	1593	0	180	198	164.4868851	480.9308052	1178	5.8
11100	GET a random Thread out of 100 Posts	542	0	180	193	164.2448902	419.1830158	1178	5.9
11100	GET a single Thread out of 100 Posts	738	0	180	199	165.0490761	434.2639446	1178	6
12100	GET a random Thread out of 1000 Posts	683	0	180	204	166.1930084	489.5379543	1178	5.6
12100	GET a single Thread out of 1000 Posts	1049	0	180	202	169.2481041	443.4149265	1178	6
13100	GET a random Thread out of 1000 Posts	1473	0	180	209	171.4019775	1157.299042	1178	6.1
13100	GET a single Thread out of 1000 Posts	1317	0	180	204	170.7370281	510.4939938	1178	5.8
14100	GET a random Thread out of 1000 Posts	855	0	190	209	175.9641171	468.7230587	1178	6
14100	GET a single Thread out of 1000 Posts	7557	0	190	213	173.609972	1970.304012	1178	5.1

POST:

Missing some data: After 1,000,000 posts in 24 hours, the response time remain constant.

Unfortunately locust ran into a calculation error when running a post test so there is no table data. After 1,000,000 posts in 24 hours, the response time remain constant. This was tested against https://courses-loadtest.edx.org/

/threads/{thread_id}

GET:

Waiting on Loadtest env to get meaningful results.

PATCH:

Patch events for the boolean values took more or less the same amount of time.

This was tested against a t2.large sandbox.

Type	Name	# requests	# fails	Median	Average	Min	Max	Content Size	# reqs/sec
PATCH	abuse_flagged	1814	0	300	315	96.83394432	1307.781935	2009	2.5
PATCH	following	1847	0	300	314	97.83601761	1730.396986	1939	1.9
PATCH	voted	1875	0	310	319	97.41687775	1427.90103	2104	1.3

Patch events for being a Staff vs. Non-staff user did not make a difference.

Using 2 locust user intentionally split between staff and non-staff tasks, PATCH requests were sent with a 500ms wait time. This was tested against a t2.large sandbox.

Name	reqs	#	fails	Avg	Min	Max	Median	req/s
PATCH	abuse_flag_thread	33	0(0.00%)	332	244	470	330	0
PATCH	following_thread	120	0(0.00%)	326	241	456	330	0.2
PATCH	staff_abuse_flag_thread	40	0(0.00%)	320	240	402	330	0
PATCH	staff_following_thread	110	0(0.00%)	333	239	482	330	0
PATCH	staff_vote_on_thread	1774	0(0.00%)	339	229	17607	330	1.1
PATCH	vote_on_thread	1787	0(0.00%)	331	226	14599	330	1

Different body sized edits did not seem to make a difference in response times.

Name	reqs	#	fails	Avg	Min	Max	Median	req/s
PATCH	abuse_flag_thread	16	0(0.00%)	162	117	212	150	0
PATCH	edit_thread_with_10000char	50	0(0.00%)	199	151	424	190	0.1
PATCH	edit_thread_with_1000char	61	0(0.00%)	234	138	3707	170	0.1
PATCH	edit_thread_with_250char	54	0(0.00%)	178	136	422	160	0.1
PATCH	edit_thread_with_4char	57	0(0.00%)	183	138	331	170	0.1
PATCH	edit_thread_with_5000char	57	0(0.00%)	188	141	341	180	0.1
PATCH	following_thread	42	0(0.00%)	168	130	337	160	0.2
PATCH	vote_on_thread	698	0(0.00%)	160	114	652	150	1.1

DELETE:

DELETE will be best tested with the other endpoints

For every DELETE thread, we POST a Thread and then GET a thread from the thread pool. This was tested against a t2.large.sandbox.

Name	reqs	#	fails	Avg	Min	Max	Median	req/s
DELETE	DELETE_thread	305	1(0.33%)	203	137	352	190	3
GET	GET_thread_list	306	0(0.00%)	190	154	420	170	2.9
POST	POST_thread	306	0(0.00%)	127	102	277	110	3

/comments/

GET:

mark_as_read and page did not seem to affect the response time at all.

Using 10 locust users with min/max time of 1000/2000ms, GET requests for sent for 1 of 10 threads with 100 responses in increasing increments of 100 responses, each with single comment. The page_size seemed to be the parameter that affected the response time. This was tested against a t2.large sandbox.

Type	Name	# requests	# fails	Median	Average	Min	Max	Content Size	# reqs/sec
GET	page_size=100	552	0	1200	1212	689.6290779	2079.323053	167747	0.6
GET	page_size=75	588	0	1000	1053	570.6920624	1957.83782	125858	0.7
GET	page_size=50	552	0	900	927	459.9750042	1813.49206	83958	1.1
GET	page_size=25	525	0	790	810	345.9990025	1692.13295	42058	1
GET	page_size=1	557	0	680	710	237.9820347	1673.289061	1833	0.8

None of the parameters affect the response time. As comments on a response increases, so does the response time.

Using 10 locust users with min/max time of 1000/2000ms, GET requests for sent for 1 of 10 threads with 1 response with comments increasing in increments of 50. This was tested against a t2.large sandbox.

Type	Name	# requests	# fails	Median	Average	Min	Max	Content Size	# reqs/sec
GET	comments=500	352	0	1700	1822	1361.582041	3062.257051	406832	0.1
GET	comments=450	395	0	1600	1710	1244.348049	3462.92305	366232	0.3
GET	comments=400	387	0	1400	1552	1093.059063	4270.447969	325632	0.1
GET	comments=350	394	0	1300	1394	972.3510742	2612.555027	285032	0.4
GET	comments=300	379	0	1100	1233	827.7709484	2370.13793	244432	0.2
GET	comments=250	353	0	970	1056	708.3749771	2795.8529	203832	0
GET	comments=200	352	0	830	943	583.6598873	2230.697155	163232	0.1
GET	comments=150	368	0	680	785	441.0719872	2042.181969	122632	0.2
GET	comments=100	342	0	540	658	323.1511116	1866.328001	82032	0.2
GET	comments=50	390	0	390	512	194.7860718	2818.300009	41432	0.3

POST:

POST should be similar to POSTing threads.

/comments/comment_id

PATCH:

As with PATCHing a thread, body edit size did not seem to matter. Boolean patches also seemed to be the same.

Name	reqs	#	fails	Avg	Min	Max	Median	req/s
GET	GET_comment_list	916	0(0.00%)	260	119	643	240	3
GET	GET_thread_list	916	0(0.00%)	275	187	571	260	2.9
PATCH	abuse_flag_comment	47	0(0.00%)	271	159	513	260	0.4
PATCH	edit_comment_with_10000char	60	0(0.00%)	307	205	498	300	0.2
PATCH	edit_comment_with_1000char	57	0(0.00%)	264	165	448	250	0.2
PATCH	edit_comment_with_250char	71	0(0.00%)	264	160	467	250	0.3
PATCH	edit_comment_with_4char	76	0(0.00%)	265	166	469	250	0.4
PATCH	edit_comment_with_5000char	60	0(0.00%)	293	195	597	280	0.1
PATCH	vote_on_comment	547	0(0.00%)	283	130	641	270	1.5

DELETE:

DELETE is best tested with the other endpoints. For every comment delete, we POST a thread, GET a random thread, and then DELETE that random thread.