Running LMS Forums Performance Tests

Setting up Discussions Load Testing Sandbox

Provision LMS/Discussions Sandbox

Provision sandbox using Jenkins (requires VPN)
- Set dns_name and name_tag.
- Set sandbox life of you need for longer than a week.
- Set reconfigure checkbox IF adding New Relic (costs $$$, ask first).
- Set instance_type to m4.large.
- Set enable_newrelic IF adding New Relic (costs $$$, ask first).

Manual changes to Sandbox (SSH)

Update the sandbox for auto auth:

sudo vi /edx/app/edxapp/lms.env.json
# Set AUTOMATIC_AUTH_FOR_TESTING to true

# restart application
sudo /edx/bin/supervisorctl restart edxapp:*

Update mongo indices:

mongo
> use cs_comments_service_development
> # COPY contents of edx-load-tests/discussions_api/db/create_indexes.js and PASTE in shell
> exit

Fix Forums to work with New Relic:

sudo vi /edx/app/forum/cs_comments_service/config/newrelic.yml
# Under development:
# - Set monitor_mode to true
# - Set developer_mode to false

# Restart application
sudo /edx/bin/supervisorctl restart forum

Other resources

Performance Testing for Sandboxes

Setting up a Locust box

Review How to Run Load Tests. Also see edx-load-tests repo README.

Provision Locust Box

Provision locust box using Jenkins (requires VPN)
- Set loadtester_name.
- Set locust_target_host if testing against a sandbox.
- Leave instance_type as m3.medium (when should we upgrade to m4?).
  
  It's ok if it has problems starting the locust server, because you want to run from command line anyway.

Manual changes to Locust Box (SSH)

In case locust is running:

sudo /edx/bin/supervisorctl stop locust

Update branch (if needed):

cd /edx/app/locust/load-tests
git remote -v
# if you don't see https://github.com/edx/edx-load-tests.git
git remote set-url origin https://github.com/edx/edx-load-tests.git

# run necessary git commands
sudo git pull

Setup to run locust:

cd /edx/app/locust
source venvs/locust/bin/activate
cd load-tests

Seed long thread (after setup above):

# Seed long thread (Example Sandbox):
sudo vi settings_files/lms.yml
  # Update BASIC_AUTH_USER, BASIC_AUTH_PASSWORD
  # Update COURSE_ID: course-v1:edX+DemoX+Demo_Course (or any existing course)
  # Update LOCUST_TASK_SET: SeedForumsTasks
locust --host="https://discussions.sandbox.edx.org" -f lms -c 10 -r 1 -n 4500 --no-web
  # Capture output of LARGE_TOPIC_ID=AAA LARGE_THREAD_ID=BBB
 
# Seed long thread (LoadTest Environment):
sudo vi settings_files/lms.yml
  # Comment out BASIC_AUTH_USER, BASIC_AUTH_PASSWORD
  # Update COURSE_ID: course-v1:edX+DemoX+demo-ccx-baseline (or any existing course)
  # Update LOCUST_TASK_SET: SeedForumsTasks
locust --host="https://courses-loadtest.edx.org" -f lms -c 10 -r 1 -n 4500 --no-web
  # Capture output of LARGE_TOPIC_ID=AAA LARGE_THREAD_ID=BBB

Run Forums Tests (after setup above):

# Run tests against sample sandbox (using large thread):
sudo vi settings_files/lms.yml
  # Update BASIC_AUTH_USER, BASIC_AUTH_PASSWORD
  # Update COURSE_ID: course-v1:edX+DemoX+Demo_Course (or any existing course)
  # Update LOCUST_TASK_SET: ForumsTasks
  # Update LARGE_TOPIC_ID and LARGE_THREAD_ID with output from SeedForumsTasks
locust --host="https://discussions.sandbox.edx.org" -f lms -c 10 -r 1 -n 3000 --no-web

Starting Comments Service on Devstack

If you are running the Comments Service on Devstack for the purpose of working on performance, you need to start it using the following:

API_KEY=password bundle exec rackup -o 0.0.0.0 -p 18080

Confirm that you see the message "Loading config.ru." on startup, otherwise you will not have Mongoid query caching configured.

Determining Weightings

The following can be used if and when we need to update the weightings used for the call distribution in the Locust tests.

Run script

There is a determine weightings script which can be run on a Splunk export to get information that can be used to determine the request call distribution for the lms locust tests.

This weighting script should be migrated to a Splunk dashboard and the script should be deleted at some point.

To run the script, first run the following Splunk query:

index="prod-edx" sourcetype=nginx URI="*/discussion/*" AND NOT URI="*/static/css/discussion/*"

Export the results as a CSV, and then run the following:

cd edx-load-tests/lms/scripts
./determine_call_distribution.py discussions-2016-06-13.csv

A sample file discussions-2016-06-13.csv.gz is attached to this page. You will need to unzip it.

Update Weightings

The newest weightings were taken directly from the output of the script for some calls. The percentage of heaviest read threads, which were large welcome threads on a very popular course, helped determine the ultimate read large thread distribution. Some other adjustments were made, like mobile api calls were temporarily ignored. Also, some weight was given to the original weightings.

The final weightings can be seen in the ForumsTasks for the lms locust tests.

	Original Weightings			New Weightings
Read thread count: 49088 (58%)	45	103	44%	36	93	35%
Create thread count: 2439 (2%)	4		4%	4		4%
Update thread count: 99 (0%)
Create response count: 3866 (4%)	3		3%	4		4%
Create comment count: 791 (0%)
Update comment count: 150 (0%)
Forum search count: 1522 (1%)	5		5%	4		4%
Read inline count: 7511 (8%)	15		15%	10		10%
User profile count: 1123 (1%)	1		1%	1		1%
User count (???): 644 (0%)
Forum tab count: 6169 (7%)	19		18%	10		10%
Topic filter count: 3101 (3%)
Mobile API count: 7012 (8%)
Other count: 762 (0%)
TOTAL count: 84277
Followed Threads	1		1%	1		1%
Large Thread	10		10%	23		22%