Table of Contents:

• Goals
• Testing Strategy
  • Thread and comment pool
  • Spikes in production data
  • Staff vs. Normal user
  • Pagination issue
  • Other notes
• Endpoints
• Seeding data
  • Forums Analysis
• Test Details

Anchor
goals goals

Goals:

1. Understand the load we are able to handle with the discussion API for when the mobile app is released.

   1. What can the server handle?
2. Understand the the overhead between the discussion API and the ruby forums code.
   1. Does the Discussion API perform better, worse, or on par with the browser's forums?
   2. What does the forums performance look like in general? 

Usage patterns to look out for:

• Default page size for the browser is 25 while the mobile device will be using 10. It is possible that more requests could be sent for the same amount of information.
• Push notifications
  • There is a possibility for a different usage pattern to look out for. If there is a popular thread, bursts of requests can be expected.
  • Increased forum usage as there is currently no notifications for the browser.
• The browser can display the Threads, Response, and Comments all at once. The mobile app treats all three of these as separate views. It is possible that more requests could be sent for the same amount of information.
• General Usage. Discussions on mobile could naturally increase discussion forums usage.

Endpoints:

• /courses/(course_id)/
  • GET
• /course_topics/(course_id)
  • GET
• /threads/
  • GET
  • POST
• /threads/(thread_id)
  
  • GET
  • PATCH
  • DELETE 
•  /comments
  • GET 
  • POST
•  /comments/(comment_id)
  • GET not implemented
  • PATCH 
  • DELETE
    

Testing Strategy:

Originally the plan was to isolate each endpoint and determine what kind of load it can handle, but after analysis of the data, some of these endpoints seem unnecessary to isolate for a load test. These endpoints include DELETE and PATCH which are a significantly small part of the overall load in production. For the isolated test for these endpoints, it will be paired with it's appropriate GET Thread/Comment. For example, every DELETE Thread request requires a thread_id. We obtain this thread_id by calling GET Thread List with randomize parameters, which returns a list of threads where one is then randomly selected. This selected thread is then DELETEd. Below is the chart of the additional request we make. As long the ratio of how many of these requests happen in each task is understood, we can get the desired endpoint distribution. 

*GET Thread List can always return a response (so we delete a random response), but will not always return a comment so the comment created will be the one deleted.

Thread and Comment pool:

    Various methods of select post data were considered.

...

...

Anchor
testingstrategy testingstrategy

Testing Strategy:

Originally the plan was to isolate each endpoint and determine what kind of load it can handle, but after analysis of the data, some of these endpoints seem unnecessary to isolate for a load test. These endpoints include DELETE and PATCH which are a significantly small part of the overall load in production. For the isolated test for these endpoints, it will be paired with it's appropriate GET Thread/Comment. For example, every DELETE Thread request requires a thread_id. We obtain this thread_id by calling GET Thread List with randomize parameters, which returns a list of threads where one is then randomly selected. This selected thread is then DELETEd. Below is the chart of the additional request we make. As long the ratio of how many of these requests happen in each task is understood, we can get the desired endpoint distribution. 

*GET Thread List can always return a response (so we delete a random response), but will not always return a comment so the comment created will be the one deleted.

Thread and Comment pool:

    Various methods of select post data were considered.

• Selecting threads from a smaller pool or selecting the same thread.  Rather than getting the entire list of thread_ids to send requests against, we would just store a random portion of the threads. A test was run to see if matters whether the retrieved thread was random or not, but the sandbox it was run against did not have the correct mongo indexes set up. Regardless, this strategy would not work when trying to DELETE threads as the pool of potential threads would be smaller. Additionally this relies on storing data that must be shared amongst the locust users which could lead to race conditions as a locust user could be trying to GET a thread that another locust user was in the middle of DELETEing. When dealing with much larger file IO operations, it could cause some limitations on the machine that spawns the locusts.
• Retrieving the list of thread ids when starting locust. This method was effective up until the number of threads in the data set started to increase. As the median number of posts in a course is ~2000, when trying to retrieve 20*(page size max of 100), it would take 20 queries. Additionally, as mentioned in the above strategy, storing data amongst the locust users is not a trivial task. Each locust user would try to generate it's own list of threads which is unacceptable. If a thread was POSTed or DELETEd, only that locust user would have that updated information. Attempts at using the lazy module did not work either as each list of threads was instantiated separately by each locust user. Again, even if the locust users were able to use the same global variables, there would be race conditions. 

• Calling GET thread_list per DELETE/PATCH/GET_comment. Since the ratio of GET thread_list is significantly higher than any of the other calls except for GET Thread, we can achieve the desired distribution of requests for the discussion API without having to store any of the thread_ids. The table below is a 7 day snapshot on NewRelic for the discussion forums. The only drawback is that in order to GET a single thread, we need to have a thread_id. This issue will be discussed in the next bullet.

  Action	Count		Discussion API Call
  .forum.views:single_thread	675980	4760	GET Thread
  .forum.views:forum_form_discussion	234783	1653	GET Thread List
  .forum.views:inline_discussion	155176	1093	GET Thread List
  create_thread	31176	220	POST Thread
  create_comment	27438	193	POST comment
  create_sub_comment	14345	101	POST comment
  users	13820	97	-
  .forum.views:user_profile	12336	87	-
  .forum.views:followed_threads	7698	54	GET Thread List
  vote_for_comment	6731	47	PATCH Comment
  vote_for_thread	6242	44	PATCH Thread
  upload	4208	30	-
  update_comment	3403	24	PATCH Comment
  follow_thread	3870	27	PATCH Thread
  update_thread	2827	20	PATCH Thread
  delete_thread	2091	15	DELETE Thread
  endorse_comment	1232	9	PATCH Comment
  delete_comment	770	5	DELETE Comment
  flag_abuse_for_comment	373	3	PATCH Comment
  flag_abuse_for_thread	142	1	PATCH Thread

• Using pre-stored thread_id data. Since GET Thread is called more than GET Thread List, we cannot use GET Thread List to get a thread_id. Instead, we can use a pre-defined set of thread_ids as mentioned in the first two bullets. This will allow us to be able to test GET Threads in isolation. Unfortunately the issue of trying to GET a DELETEd thread may still arise. Another option could be to have the locust user only call GET Thread List once and then run multiple GET Thread's. Again, the same issue still arises if one of those Threads happened to get DELETEd. 

Production Spikes in response time:

When running some early tests, it was found that some of the requests that were believed to be slow on production, were not appearing that way on the loadtests.

Looking at the errors that show up on GET Thread, the response time is 20s, which is the time out limit. Other factors that are involved are courses that have many posts, may take longer to GET information from. These courses, although the exception, mixed in with normal courses, could explain the spikes in the data.

 Staff vs. Normal User:

    Using users with staff access was thrown into consideration as it would be make some of the permissions a bit more difficult for some discussion forums actions such as editing the body of a thread. Some tests were ran to see if there was a difference. No difference was found the tests that were designed to check for a difference. 

Pagination:

    There were some concerns that the pagination in the forums code are not working properly. A series of tests will be run against courses of different sizes and compared. The idea is that if the pagination is working correctly, all the courses should be returning threads with similar response times. If it is not working correctly, course with more posts will take longer to return threads than a smaller course. 

Things that were left out:

Moderator actions

• Pin Thread - Not implemented
• Open/Close Thread -Not implemented
• Endorsed - Not Implemented

Course topics - This will be addressed at another time. 

...

• delete_comment	770	5	DELETE Comment
  flag_abuse_for_comment	373	3	PATCH Comment
  flag_abuse_for_thread	142	1	PATCH Thread

• Using pre-stored thread_id data. Since GET Thread is called more than GET Thread List, we cannot use GET Thread List to get a thread_id. Instead, we can use a pre-defined set of thread_ids as mentioned in the first two bullets. This will allow us to be able to test GET Threads in isolation. Unfortunately the issue of trying to GET a DELETEd thread may still arise. Another option could be to have the locust user only call GET Thread List once and then run multiple GET Thread's. Again, the same issue still arises if one of those Threads happened to get DELETEd. 

Production Spikes in response time:

When running some early tests, it was found that some of the requests that were believed to be slow on production, were not appearing that way on the loadtests.

Looking at the errors that show up on GET Thread, the response time is 20s, which is the time out limit. Other factors that are involved are courses that have many posts, may take longer to GET information from. These courses, although the exception, mixed in with normal courses, could explain the spikes in the data.

 Staff vs. Normal User:

    Using users with staff access was thrown into consideration as it would be make some of the permissions a bit more difficult for some discussion forums actions such as editing the body of a thread. Some tests were ran to see if there was a difference. No difference was found the tests that were designed to check for a difference. 

Pagination:

    There were some concerns that the pagination in the forums code are not working properly. A series of tests will be run against courses of different sizes and compared. The idea is that if the pagination is working correctly, all the courses should be returning threads with similar response times. If it is not working correctly, course with more posts will take longer to return threads than a smaller course. 

Things that were left out:

Moderator actions

• Pin Thread - Not implemented
• Open/Close Thread -Not implemented
• Endorsed - Not Implemented

Course topics - This will be addressed at another time. 

...

Endpoints: 

Anchor
endpoints endpoints

• /courses/(course_id)/
  • GET
• /course_topics/(course_id)
  • GET
• /threads/
  • GET
  • POST
• /threads/(thread_id)
  
  • GET
  • PATCH
  • DELETE 
•  /comments
  • GET 
  • POST
•  /comments/(comment_id)
  • GET not implemented
  • PATCH 
  • DELETE

Usage patterns to look out for:

• Default page size for the browser is 25 while the mobile device will be using 10. It is possible that more requests could be sent for the same amount of information.
• Push notifications
  • There is a possibility for a different usage pattern to look out for. If there is a popular thread, bursts of requests can be expected.
  • Increased forum usage as there is currently no notifications for the browser.
• The browser can display the Threads, Response, and Comments all at once. The mobile app treats all three of these as separate views. It is possible that more requests could be sent for the same amount of information.
• General Usage. Discussions on mobile could naturally increase discussion forums usage.

/threads/

GET: 

Anchor
/threads/get /threads/get

...

Anchor
seedingdata seedingdata

Seeding Data: 

Course Structure Setup:

    A tarfile with a very simple setup will be used for each load test. This course was created in studio and then exported. During the course creating when seeding data, this tarfile will be used as the skeleton.

...

...