View Source

Initial observations:

from NewRelic Data from our current Forums service:

What is HttpDispatcher?
- Seems like it gets called per other request. Is it something that connects services?
Sinatra::Application/GET api/v1/users/([^/?#]+) and Sinatra::Application/GET api/v1/threads/([^/?#]+) are called the most and account for 2/3 of the time.
- Users is called many times in conjuction with other calls. Discussion API may not have to do this.
Notifications is not called that many times but it's average response time is the slowest and accounts for about 5% of the total time.
- Talk to Kevin Falcone (Deactivated) about rabbits.
The average of most responses are 250 ms or less. Standard Deviation varies a lot.
- Given these large SDs, we can hopefully be able to replicate the behaviour when changing some of the variables such as # of threads in a course, or size of the text body.

Seeding forums data vs productions data:

	Pros	Cons
Seeding the Data	Known baseline Baseline can be configured/customized User scaling Allows projections Repeatable Open source friendly	Need write scripts to seed the data Need to analyze production data and seed data in such as way that it is realistic*
Using Production Data	Real Data Readable usable	Open source unfriendly Subject to change Need to analyze this data to determine performance factors. Non customizable

Process:

Part 1: Confirm performance factors for Discussion API (Endpoints are below)

Part 2: Define data to be used in load tests.

Leaning towards seeding the forums data.
- Write a script to seed data
Example: Measure out # of threads if threads are a factor and comparing it to what is in production and using some intuition. If a course has 1000 threads, than do some tests with 0, 100,1000,10000 threads?

Part 3: Create loadtest file and do single flow test

Create locustfile for single endpoints
Use loadtest env
Get preliminary results for any glaring issues

Part 4: Take NewRelic forums data and create+run flow test

Take NewRelic data and run a Flowtest

Endpoints:

/courses/(course_id)/
- GET depends on # of courses
/course_topics/(course_id)
- GET depends on # of topics, threads, children
/threads/
- GET parameters below
  - course_id depends on # of threads/comments
  - topic_id depends on # of threads/topics (Is this faster than a normal search?)
  - following, view, test_search should be the same, except it has a filter
  - order_by might be the same as the other fitlers, or faster
  - order_direction, page_size, page shouldn't matter
- POST parameters should not affect performance
/threads/(thread_id)
- GET Not implemented
- PATCH/DELETE Should be fast? Depends on # of threads?
/comments
- GET parameters
  - thread_id depends on # of threads
  - endorsed should be the same, except with filter
  - page_size, page, shouldn't matter
- POST parameters
  - thread_id depends on the # of threads
/comments/(comment_id)
- GET not implemented
- PATCH
- DELETE

Concerns:

If an endpoint that creates Threads/Comments varies in performance depending on the # of Threads/Comments, how would we go about this test?
- Revisit this if this actually happens. Else, don't worry about this for now.
Will existing forums data affect performance of newly seeded data? Do we need to wipe the database?
- Figure this out
- TnL testing did not test for this

Notes:

Ideally, these test would be part of the new open sourced edx load tests repo
dfriedmanR (Deactivated) and Diana Huang are doing performance work on the ruby code of the forums
Brian Beggs is upgrading the mongo database so the ruby forums will be updated in the process.
If DAPI and TnL performance tests vary too much, an LMS test may also be required.

July 28th - Aug 3rd NewRelic data for the Sinatra Ruby Forums service.

Ranked by Total Time

Action	Apdex	Count	Avg (ms)	SD (ms)	Min (ms)	Max (ms)	Total (ms)	Total (% time)	Dissat (%)
HttpDispatcher	0.98 [0.5]	3090850	109	434	1.7	21900	336000000	100	100
Sinatra::Application/GET api/v1/users/([^/?#]+)	0.98 [0.5]	1279355	99.6	430	5.8	20000	127000000	37.9	40.7
Sinatra::Application/GET api/v1/threads/([^/?#]+)	0.96 [0.5]	670427	173	447	5.9	20500	116000000	34.5	39
Sinatra::Application/GET api/v1/threads	0.99 [0.5]	225963	95.6	331	9.8	20100	21600000	6.4	3.2
Sinatra::Application/GET api/v1/([^/?#]+)/threads	0.97 [0.5]	110009	190	971	12.1	20000	20800000	6.2	4.6
Sinatra::Application/POST api/v1/notifications	0.86 [0.5]	30695	518	1110	14.3	20000	15900000	4.7	5.9
Sinatra::Application/PUT api/v1/users/([^/?#]+)	0.99 [0.5]	127381	63.9	280	6.6	11600	8140000	2.4	2.1
Sinatra::Application/GET heartbeat	1.0 [0.5]	493515	14.7	76.3	8.4	21600	7270000	2.2	0.3
Sinatra::Application/POST api/v1/threads/([^/?#]+)/comments	0.98 [0.5]	27540	166	374	65.6	10400	4580000	1.4	0.6
Sinatra::Application/POST api/v1/([^/?#]+)/threads	0.97 [0.5]	18025	139	398	26.9	10700	2510000	0.7	0.6
Sinatra::Application/GET api/v1/users/([^/?#]+)/active_threads	0.95 [0.5]	10025	214	554	15.7	13100	2150000	0.6	0.7
Sinatra::Application/POST api/v1/comments/([^/?#]+)	0.98 [0.5]	12022	170	355	70.8	6890	2050000	0.6	0.3
Sinatra::Application/GET api/v1/users/([^/?#]+)/subscribed_threads	0.94 [0.5]	7735	227	484	12	8510	1750000	0.5	0.6
Sinatra::Application/GET api/v1/comments/([^/?#]+)	0.98 [0.5]	26481	60.7	312	6.1	8140	1610000	0.5	0.8
Sinatra::Application/GET api/v1/search/threads	0.99 [0.5]	14474	98.3	305	21.7	21900	1420000	0.4	0.2
Sinatra::Application/POST api/v1/users/([^/?#]+)/subscriptions	1.0 [0.5]	20213	53.1	182	12.2	5550	1070000	0.3	0.1
Sinatra::Application/PUT api/v1/comments/([^/?#]+)	0.99 [0.5]	4231	152	308	59.6	7420	643000	0.2	0.1
Sinatra::Application/PUT api/v1/comments/([^/?#]+)/votes	0.99 [0.5]	3953	80.2	253	24.4	3750	317000	0.1	0
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/votes	0.99 [0.5]	3912	79.8	286	20.9	5710	312000	0.1	0.1
Sinatra::Application/PUT api/v1/threads/([^/?#]+)	0.99 [0.5]	1603	89.1	312	28.7	3810	143000	0	0
Sinatra::Application/DELETE api/v1/threads/([^/?#]+)	0.98 [0.5]	810	149	415	25.7	6780	120000	0	0
Sinatra::Application/PUT api/v1/comments/([^/?#]+)/abuse_flag	0.98 [0.5]	379	174	325	70.6	3780	65900	0	0
Sinatra::Application/DELETE api/v1/comments/([^/?#]+)	0.98 [0.5]	420	133	502	32.8	6620	55800	0	0
Sinatra::Application/PUT api/v1/comments/([^/?#]+)/abuse_unflag	0.98 [0.5]	205	167	370	70.1	3650	34200	0	0
Sinatra::Application/DELETE api/v1/users/([^/?#]+)/subscriptions	0.99 [0.5]	460	64.8	221	13.5	2850	29800	0	0
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/pin	0.99 [0.5]	295	76.7	276	30	3600	22600	0	0
Sinatra::Application/DELETE api/v1/threads/([^/?#]+)/votes	0.99 [0.5]	303	59.6	278	18.6	3010	18100	0	0
Sinatra::Application/DELETE api/v1/comments/([^/?#]+)/votes	1.0 [0.5]	320	33.5	34.2	21.4	463	10700	0	0
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/abuse_flag	1.0 [0.5]*	86	78.9	53.3	31.6	304	6780	0	0
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/abuse_unflag	1.0 [0.5]*	48	97.5	80	31.6	398	4680	0	0
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/unpin	1.0 [0.5]*	34	66.9	31.3	31.6	188	2270	0	0
Sinatra::Application/GET (unknown)	1.0 [0.5]*	34	2.2	0.528	1.7	2.6	74.9	0	0
Sinatra::Application/HEAD (unknown)	1.0 [0.5]*	1	2.4	0	2.4	2.4	2.4	0	0

Ranked by Average Time

Action	Apdex	Count	Avg (ms)	SD (ms)	Min (ms)	Max (ms)	Total (ms)	Total (% time)	Dissat (%)
Sinatra::Application/POST api/v1/notifications	0.86 [0.5]	30695	518	1110	14.3	20000	15900000	4.7	5.9
Sinatra::Application/GET api/v1/users/([^/?#]+)/subscribed_threads	0.94 [0.5]	7735	227	484	12	8510	1750000	0.5	0.6
Sinatra::Application/GET api/v1/users/([^/?#]+)/active_threads	0.95 [0.5]	10025	214	554	15.7	13100	2150000	0.6	0.7
Sinatra::Application/GET api/v1/([^/?#]+)/threads	0.97 [0.5]	110009	190	971	12.1	20000	20800000	6.2	4.6
Sinatra::Application/PUT api/v1/comments/([^/?#]+)/abuse_flag	0.98 [0.5]	379	174	325	70.6	3780	65900	0	0
Sinatra::Application/GET api/v1/threads/([^/?#]+)	0.96 [0.5]	670427	173	447	5.9	20500	116000000	34.5	39
Sinatra::Application/POST api/v1/comments/([^/?#]+)	0.98 [0.5]	12022	170	355	70.8	6890	2050000	0.6	0.3
Sinatra::Application/PUT api/v1/comments/([^/?#]+)/abuse_unflag	0.98 [0.5]	205	167	370	70.1	3650	34200	0	0
Sinatra::Application/POST api/v1/threads/([^/?#]+)/comments	0.98 [0.5]	27540	166	374	65.6	10400	4580000	1.4	0.6
Sinatra::Application/PUT api/v1/comments/([^/?#]+)	0.99 [0.5]	4231	152	308	59.6	7420	643000	0.2	0.1
Sinatra::Application/DELETE api/v1/threads/([^/?#]+)	0.98 [0.5]	810	149	415	25.7	6780	120000	0	0
Sinatra::Application/POST api/v1/([^/?#]+)/threads	0.97 [0.5]	18025	139	398	26.9	10700	2510000	0.7	0.6
Sinatra::Application/DELETE api/v1/comments/([^/?#]+)	0.98 [0.5]	420	133	502	32.8	6620	55800	0	0
HttpDispatcher	0.98 [0.5]	3090850	109	434	1.7	21900	336000000	100	100
Sinatra::Application/GET api/v1/users/([^/?#]+)	0.98 [0.5]	1279355	99.6	430	5.8	20000	127000000	37.9	40.7
Sinatra::Application/GET api/v1/search/threads	0.99 [0.5]	14474	98.3	305	21.7	21900	1420000	0.4	0.2
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/abuse_unflag	1.0 [0.5]*	48	97.5	80	31.6	398	4680	0	0
Sinatra::Application/GET api/v1/threads	0.99 [0.5]	225963	95.6	331	9.8	20100	21600000	6.4	3.2
Sinatra::Application/PUT api/v1/threads/([^/?#]+)	0.99 [0.5]	1603	89.1	312	28.7	3810	143000	0	0
Sinatra::Application/PUT api/v1/comments/([^/?#]+)/votes	0.99 [0.5]	3953	80.2	253	24.4	3750	317000	0.1	0
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/votes	0.99 [0.5]	3912	79.8	286	20.9	5710	312000	0.1	0.1
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/abuse_flag	1.0 [0.5]*	86	78.9	53.3	31.6	304	6780	0	0
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/pin	0.99 [0.5]	295	76.7	276	30	3600	22600	0	0
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/unpin	1.0 [0.5]*	34	66.9	31.3	31.6	188	2270	0	0
Sinatra::Application/DELETE api/v1/users/([^/?#]+)/subscriptions	0.99 [0.5]	460	64.8	221	13.5	2850	29800	0	0
Sinatra::Application/PUT api/v1/users/([^/?#]+)	0.99 [0.5]	127381	63.9	280	6.6	11600	8140000	2.4	2.1
Sinatra::Application/GET api/v1/comments/([^/?#]+)	0.98 [0.5]	26481	60.7	312	6.1	8140	1610000	0.5	0.8
Sinatra::Application/DELETE api/v1/threads/([^/?#]+)/votes	0.99 [0.5]	303	59.6	278	18.6	3010	18100	0	0
Sinatra::Application/POST api/v1/users/([^/?#]+)/subscriptions	1.0 [0.5]	20213	53.1	182	12.2	5550	1070000	0.3	0.1
Sinatra::Application/DELETE api/v1/comments/([^/?#]+)/votes	1.0 [0.5]	320	33.5	34.2	21.4	463	10700	0	0
Sinatra::Application/GET heartbeat	1.0 [0.5]	493515	14.7	76.3	8.4	21600	7270000	2.2	0.3
Sinatra::Application/HEAD (unknown)	1.0 [0.5]*	1	2.4	0	2.4	2.4	2.4	0	0
Sinatra::Application/GET (unknown)	1.0 [0.5]*	34	2.2	0.528	1.7	2.6	74.9	0	0

Ranked by Standard Deviation

Action	Apdex	Count	Avg (ms)	SD (ms)	Min (ms)	Max (ms)	Total (ms)	Total (% time)	Dissat (%)
Sinatra::Application/POST api/v1/notifications	0.86 [0.5]	30695	518	1110	14.3	20000	15900000	4.7	5.9
Sinatra::Application/GET api/v1/([^/?#]+)/threads	0.97 [0.5]	110009	190	971	12.1	20000	20800000	6.2	4.6
Sinatra::Application/GET api/v1/users/([^/?#]+)/active_threads	0.95 [0.5]	10025	214	554	15.7	13100	2150000	0.6	0.7
Sinatra::Application/DELETE api/v1/comments/([^/?#]+)	0.98 [0.5]	420	133	502	32.8	6620	55800	0	0
Sinatra::Application/GET api/v1/users/([^/?#]+)/subscribed_threads	0.94 [0.5]	7735	227	484	12	8510	1750000	0.5	0.6
Sinatra::Application/GET api/v1/threads/([^/?#]+)	0.96 [0.5]	670427	173	447	5.9	20500	116000000	34.5	39
HttpDispatcher	0.98 [0.5]	3090850	109	434	1.7	21900	336000000	100	100
Sinatra::Application/GET api/v1/users/([^/?#]+)	0.98 [0.5]	1279355	99.6	430	5.8	20000	127000000	37.9	40.7
Sinatra::Application/DELETE api/v1/threads/([^/?#]+)	0.98 [0.5]	810	149	415	25.7	6780	120000	0	0
Sinatra::Application/POST api/v1/([^/?#]+)/threads	0.97 [0.5]	18025	139	398	26.9	10700	2510000	0.7	0.6
Sinatra::Application/POST api/v1/threads/([^/?#]+)/comments	0.98 [0.5]	27540	166	374	65.6	10400	4580000	1.4	0.6
Sinatra::Application/PUT api/v1/comments/([^/?#]+)/abuse_unflag	0.98 [0.5]	205	167	370	70.1	3650	34200	0	0
Sinatra::Application/POST api/v1/comments/([^/?#]+)	0.98 [0.5]	12022	170	355	70.8	6890	2050000	0.6	0.3
Sinatra::Application/GET api/v1/threads	0.99 [0.5]	225963	95.6	331	9.8	20100	21600000	6.4	3.2
Sinatra::Application/PUT api/v1/comments/([^/?#]+)/abuse_flag	0.98 [0.5]	379	174	325	70.6	3780	65900	0	0
Sinatra::Application/GET api/v1/comments/([^/?#]+)	0.98 [0.5]	26481	60.7	312	6.1	8140	1610000	0.5	0.8
Sinatra::Application/PUT api/v1/threads/([^/?#]+)	0.99 [0.5]	1603	89.1	312	28.7	3810	143000	0	0
Sinatra::Application/PUT api/v1/comments/([^/?#]+)	0.99 [0.5]	4231	152	308	59.6	7420	643000	0.2	0.1
Sinatra::Application/GET api/v1/search/threads	0.99 [0.5]	14474	98.3	305	21.7	21900	1420000	0.4	0.2
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/votes	0.99 [0.5]	3912	79.8	286	20.9	5710	312000	0.1	0.1
Sinatra::Application/PUT api/v1/users/([^/?#]+)	0.99 [0.5]	127381	63.9	280	6.6	11600	8140000	2.4	2.1
Sinatra::Application/DELETE api/v1/threads/([^/?#]+)/votes	0.99 [0.5]	303	59.6	278	18.6	3010	18100	0	0
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/pin	0.99 [0.5]	295	76.7	276	30	3600	22600	0	0
Sinatra::Application/PUT api/v1/comments/([^/?#]+)/votes	0.99 [0.5]	3953	80.2	253	24.4	3750	317000	0.1	0
Sinatra::Application/DELETE api/v1/users/([^/?#]+)/subscriptions	0.99 [0.5]	460	64.8	221	13.5	2850	29800	0	0
Sinatra::Application/POST api/v1/users/([^/?#]+)/subscriptions	1.0 [0.5]	20213	53.1	182	12.2	5550	1070000	0.3	0.1
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/abuse_unflag	1.0 [0.5]*	48	97.5	80	31.6	398	4680	0	0
Sinatra::Application/GET heartbeat	1.0 [0.5]	493515	14.7	76.3	8.4	21600	7270000	2.2	0.3
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/abuse_flag	1.0 [0.5]*	86	78.9	53.3	31.6	304	6780	0	0
Sinatra::Application/DELETE api/v1/comments/([^/?#]+)/votes	1.0 [0.5]	320	33.5	34.2	21.4	463	10700	0	0
Sinatra::Application/PUT api/v1/threads/([^/?#]+)/unpin	1.0 [0.5]*	34	66.9	31.3	31.6	188	2270	0	0
Sinatra::Application/GET (unknown)	1.0 [0.5]*	34	2.2	0.528	1.7	2.6	74.9	0	0
Sinatra::Application/HEAD (unknown)	1.0 [0.5]*	1	2.4	0	2.4	2.4	2.4	0	0