Addressing concerns and points to note:

It was found that locust users were not sending requests as expected. Rather than waiting for the correct wait time, after certain task, a request would be immediately sent. The locustfile has been adjusted accordingly to output the same RPS and have users that correlate to RPS. This should not invalidate any of the older tests as long as we look at the RPS and the distribution of requests is what we expected. If anything, we now have a more granular way to control RPS. As long as we can match the RPS of a previous test, it should be repeatable.
It was found in the previous test that response times over time get slower. This is something that became apparent once the forums became more stable after adding a missing index. There were many causes for this.
- The cause for this is the "asc" parameter was removed for testing because it was affecting the loadtest results. We removed it to simulate a situation where the correct indexes were in place. "asc" will be added as an index for future testing.
- GETting and PATCHing a thread became less random and started to cause certain threads to get picked more. Randomization will be redone since order was a huge factor.
- PATCH body in the script used 4, 250, 1000, 5000, 10000 character edits while the median post size is only 250 char. The 5000, and 10000 character body edits will be removed.
- When retrieving a thread list, the page_size returned more and more content as the PATCHs happened.
- PATCH will be removed from future tests.
After the indexing issue, a couple of tests were run against a few courses. It was found that course size in regards to posts was still a factor in response times where smaller courses had better response times. When testing our largest sample course BerkeleyX/ColWri2.2x/1T2014 (~25,000 threads, 30,000 comments) the indexing improved this course enough where we can now use a near worst case course as our baseline.
Unfortunately due to time constraints, the next tests will not be using seeded data. Instead we will be using BerkeleyX/ColWri2.2x/1T2014 (~25,000 threads, 30,000 comments) which is a near worst case in terms of number of posts.
View=["unread", "unanswered"] has been disabled as a search parameter. The use of these options is in question as well as the fact that they are similar 500 errors.
The 95th percentile in the previous tests have been much higher than the average/median. The reason for this is because we mixed in different page_sizes which affect the response time. The mobile app will by default request 10 items per page. The locust test were using (1, 25, 50, 75, 100) which leans towards the worst use case. To provide a more consistent bad case, a page size of 50 will be used.

Ramping:

This was a 3 hour test against a course against a seeded course (14019 Threads, 8243 comments). Users are working correctly. 200 users were spawned and the RPS is about 10-15 times that of production. There were no visible signs of ramping.

Ramping

Browser not supported