Page Comparison

Background:

In reference to 

...

unread_comment_count = Comment.collection.find(:comment_thread_id => t._id, :author_id => {"$ne" => user.id}, :created_at => {"$gte" => read_dates[thread_key]}).count

So, we removed index
index({_type: 1, comment_thread_id: 1, author_id: 1, updated_at: 1})

and added a new one

index({comment_thread_id: 1, author_id: 1, created_at: 1})

...

Results:

The load tests were run on 4x c4.2xlarge instances for lms and 3x m4.large instances for forums.

The results of load tests below show difference differences between the old and new implementation.

The load tests were run on 4x c4.2xlarge instances.

There appears to be anomaly in the results below (highlighted 99%) where the new implementation is faster than the old. Where as, for other results the percentile of new implementation increases significantly with the increase in load. For example, the first pair of results where no. of clients = 48, the difference is in tens or hundrend and for the last pair of results where no. of clients = 610, the difference is in thousands.

Logically; there should only be increase in response time for GET endpoints, but we see in results that all other endpoints show differences too. This is because all PATCH, POST and DELETE endpoints first call GET endpoint to retrieve an 'id' of thread and/or comment and then make further processing on it.

New Relic:

Here is the new relic permaLink_new_index and permaLink_old_index

where the average rpm for former is 1.55k and for later is 1.48k and 84.5% of requests are made for AccountViewSet.list (i.e. user accounts API for multiple usernames) for later.

...

Old Index

...

New Index

...

No. of clients = 48

req/s = 9.4

No. of clients = 48

req/s = 9.6

No. of clients = 96

req/s = 18.5

No. of clients = 96

req/s = 18.7

No. of clients = 162

req/s = 31

No. of clients = 162

req/s = 31.5

No. of clients = 240

req/s = 44.6

No. of clients = 240

req/s = 44.7

No. of clients = 336

req/s = 53

No. of clients = 336

req/s = 48.9

The two set of results looks quite similar except when it reaches "No. of clients = 336"; where there is huge difference between old and new percentiles as well as sudden rise in percentile for both old and new index with respect to "No. of clients = 224". For all the next tests (i.e. No. of clients = 460, No. of clients = 510, No. of clients = 578), the difference between old and new percentile is minimised and the new index results have lower percentile for most of the endpoints.

I have captured new relic charts too; permaLink_old_index with average rpm = 1.86k and permaLink_new_index with average rpm = 1.81k

No. of clients = 46048

req/s = 449.54

No. of clients = 46048

req/s = 328.49

No. of clients = 

96

req/s = 

18.

5

No. of clients = 96

req/s = 18.6

No. of clients = 162

req/s = 31

No. of clients = 162

req/s = 31.6

No. of clients = 240

req/s = 44.6

No. of clients = 240

req/s = 45.40

No. of clients = 270

req/s = 51.1

No. of clients = 300

req/s = 48

No. of clients = 336

req/s = 53

No. of clients = 336

req/s = 54.3

No. of clients = 460

req/s = 44.5

No. of clients = 460

req/s = 49.1

No. of clients = 510

req/s = 32.4

No. of clients = 510

req/s = 33.9

No. of clients = 578

req/s = 26.70

No. of clients = 578

req/s = 40.2