Page Comparison

...

...

...

We have gathered and compared two sets of load test results. The former results (i.e. with profile_image) are run on the latest changes and the later are run on older implementation (i.e. without profile_image).

There appears to be anomaly in the results below (highlighted 99%) where the new implementation is faster than the old. Where as, for other results the percentile of new implementation increases significantly with the increase in load. For example, the first pair of results where no. of clients = 48, the difference is in tens or hundrend and for the last pair of results where no. of clients = 610, the difference is in thousands.

Logically; there should only be increase in response time for GET endpoints, but we see in results that all other endpoints show differences too. This is because all PATCH, POST and DELETE endpoints first call GET endpoint to retrieve an 'id' of thread and/or comment and then make further processing on it.

New Relic:

Here is the new relic permaLink_without_profile and permaLink_with_profile where the average rpm for former is 1.55k and for later is 1.48k and 84.5% of requests are made for AccountViewSet.list (i.e. user accounts API for multiple usernames) for later.

...

With Profile Image

...

Without Profile Image

...

No. of clients = 

req/s = 

No. of clients = 

req/s =

No. of clients = 

req/s = 

No. of clients = 

req/s = 

...

No. of clients = 

req/s = 

No. of clients = 

req/s = 

No. of clients = 

req/s = 

No. of clients = 

req/s = 

No. of clients = 192

req/s = 16.10

No. of clients = 192

req/s = 16

No. of clients = 240

req/s = 18.70

No. of clients = 240

req/s = 19.7

No. of clients = 288

req/s = 23.20

No. of clients = 288

req/s = 22.2

No. of clients = 336

req/s = 26.70

No. of clients = 336

req/s = 28.2

No. of clients = 384

req/s = 31

No. of clients = 384

req/s = 31.4

...

Background:

In reference to 

1. Changed post/response/comment behaviour to update post's 'last_activity_at' only at time of creation of post and creation of response/comment on a post. Previously post's 'last_activity_at' was being updated for both creation and update.
2. To calculate 'read' status of a post, used 'last_acitvity_at' instead of 'updated_at'.
3. To calculate 'unread comment count' for a post, used 'created_at' instead of 'updated_at'.
   
   

'unread comment count' was being calculate as:

unread_comment_count = Comment.collection.find(:comment_thread_id => t._id, :author_id => {"$ne" => user.id}, :updated_at => {"$gte" => read_dates[thread_key]}).count

and had a compound index against it
index({_type: 1, comment_thread_id: 1, author_id: 1, updated_at: 1})
 

With new implementation:

unread_comment_count = Comment.collection.find(:comment_thread_id => t._id, :author_id => {"$ne" => user.id}, :created_at => {"$gte" => read_dates[thread_key]}).count

So, we removed index
index({_type: 1, comment_thread_id: 1, author_id: 1, updated_at: 1})

and added a new one

index({comment_thread_id: 1, author_id: 1, created_at: 1})

Results:

The load tests were run on 4x c4.2xlarge instances for lms and 3x m4.large instances for forums.

The results of load tests below show differences between the old and new implementation. The two set of results looks quite similar except when it reaches "No. of clients = 336"; where there is huge difference between old and new percentiles as well as sudden rise in percentile for both old and new index with respect to "No. of clients = 224". For all the next tests (i.e. No. of clients = 460, No. of clients = 510, No. of clients = 578), the difference between old and new percentile is minimised and the new index results have lower percentile for most of the endpoints.

I have captured new relic charts too; permaLink_old_index with average rpm = 1.86k and permaLink_new_index with average rpm = 1.81k

No. of clients = 48048

req/s = 369.804

No. of clients = 48048

req/s = 388.89

No. of clients = 51096

req/s = 4118.405

No. of clients = 51096

req/s = 4018.46

No. of clients = 

162

req/s = 

31

No. of clients = 

162

req/s = 

31.

6

No. of clients = 

240

req/s = 

44.

6

No. of clients = 

240

req/s = 

45.

40

No. of clients = 610270

req/s = 3651.1

No. of clients = 610300

req/s = 48.9

UPDATE:

To narrow down the behaviour of feature endpoints on high traffic and huge data, I have conducted some more tests and here are the results.

Case 1:

I ran a few tests with a fresh new course for each run and initial data of 100 threads, 10 responses each thread and 7 comments to each response in each course. The percentile shows acceptable numbers as opposed to above results where data was increasing in a single course with each run (see above "with profile image" column). Hence we know the increasing data in any course has directly proportional effect to response percentile.

...

No. of clients = 336

req/s = 53

No. of clients

...

 = 336

req/s = 54.3

 No. of clients = 544: (error rate = 364 (1.57%) )

...

Case 2:

I had created a new fresh course populated in it a huge number of threads, responses and comments (in thousands) and then run GET thread and comment endpoints for both with profile image and without (as these are the only endpoints this changes is reflected in). 

Comparing the two set of results; we can see the difference in 99% for with profile image but I believe the numbers are acceptable, only the last two cases that I have highlighted shows anomaly.

• For profile image: when no. of clients = 510 shows greater response time than no. of clients = 544. Its reason that I could assume is the users involved in thread and comments for later test are less in number possibly.
• For without profile image: 
  • 99% for no. of clients = 544 is greater than that of with profile image which is weird.
  • there is an instant rise for clients=544 than for clients=510; to see if its a valid increase I used no. of clients somewhere between the two numbers i.e. 522, but the 99% was even higher than for 544, again weird.

No. of clients = 460

req/s = 44.5

No. of clients = 48460

req/s = 449.10

1

210290370170230260250650780autoauth210220
Methods
median response time
95%
99%
DELETE_comment
6100
10000
11000
DELETE_thread
3500
5700
6400
GET_comment_list
3000
5000
5700
GET_thread
4200
7100
8700
GET_thread_list
3100
5300
6200
PATCH_
comment
4400
7800
220
No. of clients = 48
req/s = 4
error rate = 0
480 210 220 220
Methods
median response time
95%
99%
GET_comment_list
 170
220
250
GET_thread
170 
230 
250 
GET_thread_list
180 
360 
8600
PATCH_thread
4100
6900
7600
POST_comment_comment
5700
9900
11000
POST_comment_response
4800
8500
10000
POST_thread
3000
4800
5500
auto_auth
1700
2700
2700

No. of clients = 144510

req/s = 12

32.4

No. of clients = 144

req/s = 11.4

No. of clients = 288510

req/s = 2333.9

 220310420180 240 280 260 
Methods
median response time
95%
99%
DELETE_comment
7600
14000
19000
DELETE_thread
6700
11000
14000
GET_comment_list
6700
11000
16000
GET_thread
9400
16000
22000
GET_thread_list
7000
710 
870 
auto_auth
220 
230 
230
No. of clients = 288
req/s = 23.3
error rate = 0
510 230 230 230
Methods
median response time
95%
99%
GET_comment_list
 170
230
310 
GET_thread
180 
240 
280 
GET_thread_list
190 
370 
11000
17000
PATCH_comment
6900
12000
16000
PATCH_thread
6900
12000
21000
POST_comment_comment
7600
14000
19000
POST_comment_response
7200
13000
18000
POST_thread
6700
11000
15000
auto_auth
2400
3200
3300

No. of clients = 510578

req/s = 42.3

26.70

No. of clients = 510

req/s = 40.8

No. of clients = 544578

req/s = 4440.402

No. of clients = 544

req/s = 45.4

...