Discussion API Performance (latest August 2016)

Background:

In reference to https://openedx.atlassian.net/browse/MA-2678; we have changed forums implementation with following details (for reference; see PR#192):

Changed post/response/comment behaviour to update post's 'last_activity_at' only at time of creation of post and creation of response/comment on a post. Previously post's 'last_activity_at' was being updated for both creation and update.
To calculate 'read' status of a post, used 'last_acitvity_at' instead of 'updated_at'.
To calculate 'unread comment count' for a post, used 'created_at' instead of 'updated_at'.

'unread comment count' was being calculate as:

unread_comment_count = Comment.collection.find(:comment_thread_id => t._id, :author_id => {"$ne" => user.id}, :updated_at => {"$gte" => read_dates[thread_key]}).count
and had a compound index against it
index({_type: 1, comment_thread_id: 1, author_id: 1, updated_at: 1})

With new implementation:

unread_comment_count = Comment.collection.find(:comment_thread_id => t._id, :author_id => {"$ne" => user.id}, :created_at => {"$gte" => read_dates[thread_key]}).count
So, we removed index
index({_type: 1, comment_thread_id: 1, author_id: 1, updated_at: 1})

and added a new one
index({comment_thread_id: 1, author_id: 1, created_at: 1})

Results:

The load tests were run on 4x c4.2xlarge instances for lms and 3x m4.large instances for forums.

The results of load tests below show differences between the old and new implementation. The two set of results looks quite similar except when it reaches "No. of clients = 336"; where there is huge difference between old and new percentiles as well as sudden rise in percentile for both old and new index with respect to "No. of clients = 224". For all the next tests (i.e. No. of clients = 460, No. of clients = 510, No. of clients = 578), the difference between old and new percentile is minimised and the new index results have lower percentile for most of the endpoints.

I have captured new relic charts too; permaLink_old_index with average rpm = 1.86k and permaLink_new_index with average rpm = 1.81k

Old Index	New Index

Old Index

New Index

No. of clients = 48

req/s = 9.4

Methods	median response time	95%	99%
DELETE_comment	250	310	1200
DELETE_thread	160	190	200
GET_comment_list	150	190	310
GET_thread	140	180	220
GET_thread_list	160	370	520
PATCH_comment	230	360	1200
PATCH_thread	190	240	370
POST_comment_comment	290	360	430
POST_comment_response	250	310	450
POST_thread	160	190	200
auto_auth	200	250	250

No. of clients = 48

req/s = 8.9

Methods	median response time	median response time	95%	95%	99%	99%
DELETE_comment	250	270	300	470	480	520
DELETE_thread	170	160	210	300	230	310
GET_comment_list	150	160	200	260	320	350
GET_thread	150	150	190	250	280	310
GET_thread_list	170	190	360	370	520	530
PATCH_comment	220	240	320	400	410	460
PATCH_thread	180	190	360	300	330	330
POST_comment_comment	290	310	340	510	490	680
POST_comment_response	250	260	300	450	390	520
POST_thread	160	160	200	250	370	340
auto_auth	220	210	230	230	230	230

No. of clients = 96

req/s = 18.5

Methods	median response time	95%	99%
DELETE_comment	270	340	390
DELETE_thread	170	230	330
GET_comment_list	170	210	330
GET_thread	160	200	250
GET_thread_list	190	440	610
PATCH_comment	240	330	360
PATCH_thread	200	260	350
POST_comment_comment	310	380	470
POST_comment_response	260	330	440
POST_thread	170	210	340
auto_auth	180	220	220

No. of clients = 96

req/s = 18.6

Methods	median response time	median response time	95%	95%	99%	99%
DELETE_comment	270	280	340	360	360	980
DELETE_thread	170	170	220	230	230	290
GET_comment_list	160	160	210	220	340	400
GET_thread	160	160	200	210	270	340
GET_thread_list	190	190	460	460	600	640
PATCH_comment	220	230	310	310	340	460
PATCH_thread	190	200	260	280	350	1000
POST_comment_comment	300	310	370	410	480	680
POST_comment_response	260	260	330	350	420	620
POST_thread	170	170	210	210	340	330
auto_auth	180	210	230	220	230	220

No. of clients = 162

req/s = 31

Methods	median response time	95%	99%
DELETE_comment	320	400	490
DELETE_thread	190	240	280
GET_comment_list	190	250	370
GET_thread	190	250	340
GET_thread_list	220	520	660
PATCH_comment	270	360	390
PATCH_thread	230	310	370
POST_comment_comment	350	450	540
POST_comment_response	300	400	490
POST_thread	190	240	340
auto_auth	200	200	200

No. of clients = 162

req/s = 31.6

Methods	median response time	median response time	95%	95%	99%	99%
DELETE_comment	310	320	410	420	590	530
DELETE_thread	190	190	240	230	250	250
GET_comment_list	190	190	250	250	380	380
GET_thread	180	180	250	250	360	340
GET_thread_list	220	210	530	520	670	660
PATCH_comment	260	250	380	360	1400	430
PATCH_thread	220	230	280	310	400	370
POST_comment_comment	340	340	450	470	540	620
POST_comment_response	290	290	400	400	510	550
POST_thread	190	190	240	230	370	360
auto_auth	210	200	220	200	220	200

No. of clients = 240

req/s = 44.6

Methods	median response time	95%	99%
DELETE_comment	400	600	700
DELETE_thread	240	320	530
GET_comment_list	240	350	480
GET_thread	240	360	470
GET_thread_list	270	630	800
PATCH_comment	330	510	600
PATCH_thread	290	420	530
POST_comment_comment	420	640	820
POST_comment_response	360	560	730
POST_thread	240	310	440
auto_auth	210	210	210

No. of clients = 240

req/s = 45.40

Methods	median response time	median response time	95%	95%	99%	99%
DELETE_comment	390	410	650	690	1300	1100
DELETE_thread	240	230	370	280	430	340
GET_comment_list	230	240	350	350	500	480
GET_thread	230	240	370	370	600	520
GET_thread_list	270	270	640	630	850	790
PATCH_comment	300	310	490	480	1200	610
PATCH_thread	280	290	460	470	640	630
POST_comment_comment	410	430	670	660	1000	970
POST_comment_response	360	360	560	600	860	850
POST_thread	230	230	300	310	500	430
auto_auth	190	210	220	220	220	220

No. of clients = 270

req/s = 51.1

Methods	median response time	95%	99%
DELETE_comment	560	7900	10000
DELETE_thread	280	620	5200
GET_comment_list	300	3100	4000
GET_thread	310	3400	5000
GET_thread_list	330	3000	4200
PATCH_comment	400	1700	6500
PATCH_thread	380	5100	6300
POST_comment_comment	560	7900	10000
POST_comment_response	470	6100	8200
POST_thread	290	2900	3900
auto_auth	320	320	320

No. of clients = 300

req/s = 48

Methods	median response time	95%	99%
DELETE_comment	1600	4600	5900
DELETE_thread	460	1600	2500
GET_comment_list	480	1300	1900
GET_thread	560	2100	2900
GET_thread_list	530	1700	2500
PATCH_comment	800	2800	3500
PATCH_thread	710	2700	3600
POST_comment_comment	1400	4300	5500
POST_comment_response	1000	3600	4800
POST_thread	430	1100	1700
auto_auth	240	250	250

No. of clients = 336

req/s = 53

Methods	median response time	95%	99%
DELETE_comment	2500	5500	6500
DELETE_thread	980	2300	2900
GET_comment_list	800	2100	2700
GET_thread	920	2700	3300
GET_thread_list	850	2400	3300
PATCH_comment	1500	3900	5000
PATCH_thread	1200	3400	4100
POST_comment_comment	2300	5400	7000
POST_comment_response	1700	4400	5900
POST_thread	740	2000	2700
auto_auth	330	1300	1300

No. of clients = 336

req/s = 54.3

Methods	median response time	median response time	95%	95%	99%	99%
DELETE_comment	1500	5300	9200	9600	12000	11000
DELETE_thread	470	2300	4100	4600	6700	5100
GET_comment_list	510	1800	5000	3700	6800	4500
GET_thread	590	2100	6500	4800	9900	5700
GET_thread_list	580	1800	4900	3900	6800	4800
PATCH_comment	810	3500	7600	6400	1100	9700
PATCH_thread	790	2900	5400	6300	7800	7300
POST_comment_comment	1300	4800	9300	9300	12000	11000
POST_comment_response	1000	3700	7700	7800	10000	9600
POST_thread	470	1700	4600	3500	6400	4000
auto_auth	320	710	320	830	320	830

No. of clients = 460

req/s = 44.5

Methods	median response time	95%	99%
DELETE_comment	5600	9800	12000
DELETE_thread	3700	6200	7900
GET_comment_list	3700	6100	8100
GET_thread	5200	9800	13000
GET_thread_list	3700	6800	8600
PATCH_comment	4800	7900	9500
PATCH_thread	4600	7300	9900
POST_comment_comment	5500	9300	11000
POST_comment_response	5000	8300	10000
POST_thread	3500	6000	9200
auto_auth	2000	2600	2600

No. of clients = 460

req/s = 49.1

Methods	median response time	95%	99%
DELETE_comment	6100	10000	11000
DELETE_thread	3500	5700	6400
GET_comment_list	3000	5000	5700
GET_thread	4200	7100	8700
GET_thread_list	3100	5300	6200
PATCH_comment	4400	7800	8600
PATCH_thread	4100	6900	7600
POST_comment_comment	5700	9900	11000
POST_comment_response	4800	8500	10000
POST_thread	3000	4800	5500
auto_auth	1700	2700	2700

No. of clients = 510

req/s = 32.4

Methods	median response time	95%	99%
DELETE_comment	7700	15000	27000
DELETE_thread	5400	12000	15000
GET_comment_list	6300	14000	31000
GET_thread	8600	19000	36000
GET_thread_list	6400	14000	32000
PATCH_comment	7000	14000	25000
PATCH_thread	6900	16000	35000
POST_comment_comment	7400	15000	36000
POST_comment_response	7000	14000	26000
POST_thread	6000	11000	32000
auto_auth	2900	4000	4400

No. of clients = 510

req/s = 33.9

Methods	median response time	95%	99%
DELETE_comment	7600	14000	19000
DELETE_thread	6700	11000	14000
GET_comment_list	6700	11000	16000
GET_thread	9400	16000	22000
GET_thread_list	7000	11000	17000
PATCH_comment	6900	12000	16000
PATCH_thread	6900	12000	21000
POST_comment_comment	7600	14000	19000
POST_comment_response	7200	13000	18000
POST_thread	6700	11000	15000
auto_auth	2400	3200	3300

No. of clients = 578

req/s = 26.70

Methods	median response time	95%	99%
DELETE_comment	9500	26000	32000
DELETE_thread	8700	25000	32000
GET_comment_list	8600	22000	31000
GET_thread	13000	27000	40000
GET_thread_list	8600	22000	32000
PATCH_comment	8800	22000	32000
PATCH_thread	8800	20000	29000
POST_comment_comment	9200	24000	31000
OST_comment_response	9300	24000	32000
POST_thread	8300	20000	32000
auto_auth	3600	5200	5500

No. of clients = 578

req/s = 40.2

Methods	median response time	95%	99%
DELETE_comment	9100	24000	32000
DELETE_thread	8500	22000	26000
GET_comment_list	8100	22000	32000
GET_thread	12000	26000	40000
GET_thread_list	8300	22000	32000
PATCH_comment	8300	23000	31000
PATCH_thread	8000	23000	28000
POST_comment_comment	9000	24000	32000
OST_comment_response	8500	23000	32000
POST_thread	8300	22000	32000
auto_auth	4100	5100	5400