Page Comparison

With the addition of new query param (i.e. requested_fields=profile_image) to discussion API GET endpoints for thread and comment; the discussion APIs got exposed to the possibility of performance degradation because this new param fetches and prepares the profile image related data of all the users that are participating in thread or comment.

We have gathered and compared two sets of load test results. The former results (i.e. with profile_image) are run on the latest changes and the later are run on older implementation (i.e. without profile_image).

There appears to be anomaly in the results below (highlighted 99%) where the new implementation is faster than the old. Where as, for other results the percentile of new implementation increases significantly with the increase in load. For example, the first pair of results where no. of clients = 48, the difference is in tens or hundrend and for the last pair of results where no. of clients = 610, the difference is in thousands.

Logically; there should only be increase in response time for GET endpoints, but we see in results that all other endpoints show differences too. This is because all PATCH, POST and DELETE endpoints first call GET endpoint to retrieve an 'id' of thread and/or comment and then make further processing on it.

New Relic:

Here is the new relic permaLink_without_profile and permaLink_with_profile where the average rpm for former is 1.55k and for later is 1.48k and 84.5% of requests are made for AccountViewSet.list (i.e. user accounts API for multiple usernames) for later.

...

Old Index

...

New Index

...

No. of clients = 48

req/s = 9.4

No. of clients = 48

req/s = 9.6

No. of clients = 96

req/s = 18.5

No. of clients = 96

req/s = 18.7

No. of clients = 162

req/s = 31

No. of clients = 162

req/s = 31.5

No. of clients = 240

req/s = 44.6

No. of clients = 240

req/s = 44.7

No. of clients = 336

req/s = 53

No. of clients = 336

req/s = 48.9

Background:

In reference to 

1. Changed post/response/comment behaviour to update post's 'last_activity_at' only at time of creation of post and creation of response/comment on a post. Previously post's 'last_activity_at' was being updated for both creation and update.
2. To calculate 'read' status of a post, used 'last_acitvity_at' instead of 'updated_at'.
3. To calculate 'unread comment count' for a post, used 'created_at' instead of 'updated_at'.
   
   

'unread comment count' was being calculate as:

unread_comment_count = Comment.collection.find(:comment_thread_id => t._id, :author_id => {"$ne" => user.id}, :updated_at => {"$gte" => read_dates[thread_key]}).count

and had a compound index against it
index({_type: 1, comment_thread_id: 1, author_id: 1, updated_at: 1})
 

With new implementation:

unread_comment_count = Comment.collection.find(:comment_thread_id => t._id, :author_id => {"$ne" => user.id}, :created_at => {"$gte" => read_dates[thread_key]}).count

So, we removed index
index({_type: 1, comment_thread_id: 1, author_id: 1, updated_at: 1})

and added a new one

index({comment_thread_id: 1, author_id: 1, created_at: 1})

Results:

The load tests were run on 4x c4.2xlarge instances for lms and 3x m4.large instances for forums.

The results of load tests below show differences between the old and new implementation. The two set of results looks quite similar except when it reaches "No. of clients = 336"; where there is huge difference between old and new percentiles as well as sudden rise in percentile for both old and new index with respect to "No. of clients = 224". For all the next tests (i.e. No. of clients = 460, No. of clients = 510, No. of clients = 578), the difference between old and new percentile is minimised and the new index results have lower percentile for most of the endpoints.

I have captured new relic charts too; permaLink_old_index with average rpm = 1.86k and permaLink_new_index with average rpm = 1.81k

No. of clients = 46048

req/s = 449.54

No. of clients = 46048

req/s = 328.49

No. of clients = 

96

req/s = 

18.

5

No. of clients = 96

req/s = 18.6

No. of clients = 162

req/s = 31

No. of clients = 162

req/s = 

31.6

No. of clients = 240

req/s = 44.6

No. of clients = 240

req/s = 45.40

No. of clients = 270

req/s = 51.1

No. of clients = 300

req/s = 48

No. of clients = 336

req/s = 53

No. of clients = 336

req/s =

 54.3

No. of clients = 460

req/s = 44.5

No. of clients = 460

req/s = 49.1

No. of clients = 510

req/s = 32.4

No. of clients = 510

req/s = 33.9

No. of clients = 578

req/s = 26.70

No. of clients = 578

req/s = 40.2