Pull Request: https://github.com/edx/edx-platform/pull/7258
DjangoKeyValueStore.set_many
was causing multiple writes whenever it saved data for a newly created XBlock. It called FieldDataCache.find_or_create
which, if the block wasn't cached, did a get_or_create
call to create it. Then DjangoKeyValueStore
immediately called save
on that block. This results in both an insert of empty state, and then an update of that record. This change means that FieldDataCache
no longer calls get_or_create
and instead just returns a new, in-memory django model object, which is then saved by DjangoKeyValueStore
.
Load Driver: https://github.com/edx/load-tests/tree/f9542ac4159928f155f27a835d3ba92783c1c15e/locust/lms
#! /bin/bash TITLE=$1 echo "STARTING LMS TEST" cd /edx/app/locust/load-tests/locust/lms locust --host http://courses-loadtest.edx.org --no-web -c 300 -r 5 -n 30000 --print-stats &> /home/cpennington/perf-results/lms.${TITLE}.$(date -Ins).txt echo "STARTING VIDEO TEST" cd /edx/app/locust/load-tests/locust/mobile_api for users in 10 20 30 40 50 100 150 200 300 400 450 460 500; do echo "STARTING ${users}" locust --host http://courses-loadtest.edx.org --no-web -c ${users} -r 5 -n ${users}00 --print-stats &> /home/cpennington/perf-results/video.${users}.${TITLE}.$(date -Ins).txt done echo "TEST COMPLETE" |
The test script was run (with a larger number of users) against revision 13ba2c4
, and then was run against the candidate revision af12b1b
, and then again against the baseline revision 13ba2c4
.
The raw results are attached to the page. An analysis comparing each of the runs against the others as attached in .ods format. Those comparisons are also reproduced below.
Summarizing, we can draw several important conclusions:
Name | # reqs | 50.00% | 66.00% | 75.00% | 80.00% | 90.00% | 95.00% | 98.00% | 99.00% | 100.00% | |
---|---|---|---|---|---|---|---|---|---|---|---|
Candidate/Re-baseline | POST handler:video:save_user_state | 5110.5 | -7.94% | -8.96% | -8.70% | -9.86% | -11.69% | -12.20% | -8.70% | 20.00% | 141.68% |
GET handler:capa:problem_get | 3437 | -16.87% | -16.67% | -15.79% | -14.14% | -10.00% | -8.33% | -7.69% | 66.67% | 175.93% | |
GET courseware:index | 3416.5 | - | 1.59% | -1.47% | -1.43% | -3.95% | -7.78% | 9.09% | - | 45.95% | |
POST handler:capa:problem_get | 2774.5 | -1.79% | -3.28% | -7.35% | -13.16% | -11.36% | -10.10% | -9.09% | -8.33% | 109.00% | |
GET handler:video:get_transcript | 2518 | - | - | - | - | -1.56% | -1.47% | -1.23% | -9.09% | 134.47% | |
GET courseware:mktg_course_about | 2235 | -1.82% | -1.69% | -1.64% | -1.61% | -1.54% | 4.29% | 32.65% | 7.69% | 88.91% | |
POST handler:goto_position | 1877.5 | -6.74% | -9.09% | - | -8.33% | -7.69% | -7.14% | - | 26.67% | 150.42% | |
GET handler:goto_position | 1805 | -10.10% | -8.33% | -8.33% | -7.69% | -7.14% | -13.33% | -12.50% | -5.88% | 98.89% | |
GET handler:capa:problem_check | 1762.5 | -13.25% | -15.22% | -13.27% | -11.00% | -11.82% | -16.67% | -21.43% | -18.75% | 54.87% | |
GET forums:single_thread | 724.5 | -4.41% | -5.48% | -5.33% | -3.90% | -2.38% | 18.18% | 12.50% | 20.00% | -4.72% | |
GET courseware:course_info | 694 | -1.05% | -1.00% | - | - | - | - | - | -57.58% | 68.25% | |
GET handler:video:save_user_state | 599 | -1.92% | - | - | - | -1.64% | -1.49% | -3.45% | -38.13% | 1.49% | |
POST handler:capa:problem_check | 468 | -3.23% | -8.00% | -9.76% | -10.47% | -12.37% | -18.18% | -23.85% | -21.43% | 404.52% | |
GET handler:capa:problem_show | 428 | -15.00% | -17.98% | -18.09% | -17.17% | -12.73% | -16.67% | - | 53.33% | 941.87% | |
GET auto_auth | 395.5 | - | - | - | 4.55% | 4.17% | 55.56% | 36.36% | 5.56% | 1.91% | |
POST enroll | 394 | -2.15% | -3.00% | - | -9.09% | - | -8.33% | - | - | 317.68% | |
GET forums:forum_form_discussion | 294.5 | - | - | -1.69% | - | 4.55% | 12.50% | 6.67% | 20.00% | 5.36% | |
GET handler:capa:problem_save | 277 | -14.81% | -15.73% | -18.75% | -19.19% | -12.73% | -16.67% | -15.38% | - | 49.35% | |
GET courseware:progress | 268 | -3.70% | -9.78% | -13.00% | -18.18% | -19.17% | -28.57% | -6.25% | -15.79% | -21.07% | |
GET forums:inline_discussion | 209.5 | -3.45% | -3.23% | - | -2.94% | -2.70% | -13.56% | -28.57% | - | -9.94% | |
GET forums:search_topic | 80 | - | -2.50% | -4.76% | -2.38% | -6.38% | -18.97% | -20.63% | -66.84% | -66.67% | |
GET courseware:about | 69 | - | - | - | - | 6.67% | 137.50% | 900.00% | 900.00% | 887.28% | |
POST handler:capa:problem_show | 27.5 | -12.68% | -8.00% | -15.85% | -16.87% | -9.78% | -10.42% | -23.08% | -23.08% | -21.97% | |
GET forums:get_user_profile | 19 | - | -4.17% | -4.17% | - | - | 83.33% | 83.33% | 83.33% | 80.78% | |
POST forums:create_comment | 17.5 | - | 3.33% | 3.33% | 3.33% | -8.57% | -2.78% | -2.78% | -2.78% | -2.24% | |
GET forums:followed_threads | 12.5 | -7.14% | -6.67% | -26.32% | -56.25% | -88.46% | -88.57% | -88.57% | -88.57% | -88.75% | |
POST handler:capa:problem_save | 6 | -24.44% | -22.22% | -29.09% | -29.09% | -9.09% | -9.09% | -9.09% | -9.09% | -3.81% | |
POST forums:create_thread | 2 | 14.29% | 14.29% | 14.29% | 14.29% | 14.29% | 14.29% | 14.29% | 14.29% | 12.15% | |
Name | # reqs | 50% | 66% | 75% | 80% | 90% | 95% | 98% | 99% | 100% | |
Candidate/Baseline | POST handler:video:save_user_state | 10986.5 | 11.54% | 12.96% | 12.50% | 12.28% | 11.48% | 12.50% | 18.31% | 39.53% | 3.54% |
GET handler:capa:problem_get | 4405.5 | - | 1.35% | 2.56% | 4.94% | 3.13% | 10.00% | 9.09% | 108.33% | 13.33% | |
GET courseware:index | 8196.5 | 6.67% | 8.47% | 6.35% | 6.15% | 7.35% | 12.16% | 30.43% | 7.69% | 4.48% | |
POST handler:capa:problem_get | 8503 | 10.00% | 3.51% | -4.55% | -4.35% | - | -1.11% | - | - | -16.83% | |
GET handler:video:get_transcript | 5316 | 20.00% | 18.75% | 20.41% | 20.00% | 18.87% | 21.82% | 31.15% | 42.86% | -6.19% | |
GET courseware:mktg_course_about | 5481 | 12.50% | 13.73% | 15.38% | 17.31% | 18.52% | 30.36% | 113.11% | 79.49% | -4.23% | |
POST handler:goto_position | 5405.5 | 9.21% | 5.26% | 10.00% | - | - | 8.33% | 7.69% | 35.71% | 1.66% | |
GET handler:goto_position | 2225.5 | 4.71% | 10.00% | - | 9.09% | 8.33% | - | - | 14.29% | 96.50% | |
GET handler:capa:problem_check | 2841.5 | 4.35% | 4.00% | 7.59% | 5.95% | 3.19% | - | - | 8.33% | -1.86% | |
GET forums:single_thread | 1838.5 | 3.17% | 4.55% | 2.90% | 4.23% | 5.13% | 38.30% | 12.50% | 14.29% | -28.79% | |
GET courseware:course_info | 1678 | 9.30% | 11.24% | 9.89% | 8.70% | 15.79% | 11.11% | 18.18% | - | -9.50% | |
GET handler:video:save_user_state | 661.5 | 15.91% | 17.02% | 18.75% | 20.83% | 20.00% | 24.53% | 42.37% | 62.30% | 12.79% | |
POST handler:capa:problem_check | 1836.5 | 5.26% | 2.99% | 5.71% | 6.94% | 3.66% | -1.10% | -1.00% | - | -10.73% | |
GET handler:capa:problem_show | 830 | 7.94% | 4.29% | 2.67% | 5.13% | 4.35% | - | 18.18% | 91.67% | 27.64% | |
GET auto_auth | 792.5 | 10.53% | 10.00% | 10.00% | 9.52% | 13.64% | 61.54% | 7.14% | 11.76% | -4.98% | |
POST enroll | 791 | 22.97% | 24.36% | 23.46% | 21.95% | 30.95% | 25.00% | 26.32% | 18.18% | -60.47% | |
GET forums:forum_form_discussion | 662 | 1.89% | 1.82% | 1.75% | 3.45% | 11.29% | 28.57% | 14.29% | 26.32% | 1.07% | |
GET handler:capa:problem_save | 548.5 | 6.15% | 5.63% | 2.63% | 1.27% | 5.49% | - | - | 27.27% | -88.92% | |
GET courseware:progress | 692 | 9.86% | 10.67% | 11.54% | 9.76% | - | -9.09% | 7.14% | 6.67% | -7.62% | |
GET forums:inline_discussion | 505 | 7.69% | 11.11% | 10.34% | 10.00% | 9.09% | 18.60% | -9.09% | - | -14.96% | |
GET forums:search_topic | 181.5 | 11.76% | 11.43% | 11.11% | 10.81% | 7.32% | 4.44% | -24.24% | -62.94% | -70.99% | |
GET courseware:about | 163.5 | 16.67% | 7.69% | 15.38% | 15.38% | 23.08% | 171.43% | 1114.29% | 962.50% | 7.56% | |
POST handler:capa:problem_show | 113.5 | 12.73% | 15.00% | 4.55% | -1.43% | 5.06% | - | 8.70% | -9.09% | -33.97% | |
GET forums:get_user_profile | 36.5 | - | - | -4.17% | - | - | 62.96% | 41.94% | 25.71% | 24.98% | |
POST forums:create_comment | 28 | -3.33% | - | -3.13% | -6.06% | -5.88% | 2.94% | -2.78% | -2.78% | -3.86% | |
GET forums:followed_threads | 30.5 | - | - | - | -6.67% | -25.00% | -23.81% | -38.46% | -38.46% | -40.68% | |
POST handler:capa:problem_save | 33.5 | 13.33% | 7.69% | 9.86% | 5.41% | 21.95% | 14.94% | 11.11% | 8.70% | 9.78% | |
POST forums:create_thread | 12 | 33.33% | 33.33% | 33.33% | 26.32% | 20.00% | 14.29% | 4.35% | 4.35% | 6.19% | |
Name | # reqs | 50% | 66%` | 75% | 80% | 90% | 95% | 98% | 99% | 100% | |
Re-baseline/Baseline | POST handler:video:save_user_state | 10792 | 21.15% | 24.07% | 23.21% | 24.56% | 26.23% | 28.13% | 29.58% | 16.28% | -57.16% |
POST handler:capa:problem_get | 8464.5 | 12.00% | 7.02% | 3.03% | 10.14% | 12.82% | 10.00% | 10.00% | 9.09% | -60.21% | |
GET courseware:index | 8426 | 6.67% | 6.78% | 7.94% | 7.69% | 11.76% | 21.62% | 19.57% | 7.69% | -28.41% | |
GET courseware:mktg_course_about | 5597 | 14.58% | 15.69% | 17.31% | 19.23% | 20.37% | 25.00% | 60.66% | 66.67% | -49.30% | |
POST handler:goto_position | 5390 | 17.11% | 15.79% | 10.00% | 9.09% | 8.33% | 16.67% | 7.69% | 7.14% | -59.40% | |
GET handler:video:get_transcript | 5257 | 20.00% | 18.75% | 20.41% | 20.00% | 20.75% | 23.64% | 32.79% | 57.14% | -59.99% | |
GET handler:capa:problem_get | 4354.5 | 20.29% | 21.62% | 21.79% | 22.22% | 14.58% | 20.00% | 18.18% | 25.00% | -58.93% | |
GET handler:capa:problem_check | 2800 | 20.29% | 22.67% | 24.05% | 19.05% | 17.02% | 20.00% | 27.27% | 33.33% | -36.63% | |
GET handler:goto_position | 2186.5 | 16.47% | 20.00% | 9.09% | 18.18% | 16.67% | 15.38% | 14.29% | 21.43% | -1.20% | |
GET forums:single_thread | 1897 | 7.94% | 10.61% | 8.70% | 8.45% | 7.69% | 17.02% | - | -4.76% | -25.26% | |
POST handler:capa:problem_check | 1833.5 | 8.77% | 11.94% | 17.14% | 19.44% | 18.29% | 20.88% | 30.00% | 27.27% | -82.31% | |
GET courseware:course_info | 1706 | 10.47% | 12.36% | 9.89% | 8.70% | 15.79% | 11.11% | 18.18% | 135.71% | -46.21% | |
GET handler:capa:problem_show | 806 | 26.98% | 27.14% | 25.33% | 26.92% | 19.57% | 20.00% | 18.18% | 25.00% | -87.75% | |
GET auto_auth | 783 | 10.53% | 10.00% | 10.00% | 4.76% | 9.09% | 3.85% | -21.43% | 5.88% | -6.76% | |
POST enroll | 782 | 25.68% | 28.21% | 23.46% | 34.15% | 30.95% | 36.36% | 26.32% | 18.18% | -90.54% | |
GET courseware:progress | 703 | 14.08% | 22.67% | 28.21% | 34.15% | 23.71% | 27.27% | 14.29% | 26.67% | 17.05% | |
GET forums:forum_form_discussion | 682.5 | 1.89% | 1.82% | 3.51% | 3.45% | 6.45% | 14.29% | 7.14% | 5.26% | -4.07% | |
GET handler:video:save_user_state | 654.5 | 18.18% | 17.02% | 18.75% | 20.83% | 22.00% | 26.42% | 47.46% | 162.30% | 11.14% | |
GET handler:capa:problem_save | 539.5 | 24.62% | 25.35% | 26.32% | 25.32% | 20.88% | 20.00% | 18.18% | 27.27% | -92.58% | |
GET forums:inline_discussion | 528.5 | 11.54% | 14.81% | 10.34% | 13.33% | 12.12% | 37.21% | 27.27% | - | -5.57% | |
GET forums:search_topic | 189.5 | 11.76% | 14.29% | 16.67% | 13.51% | 14.63% | 28.89% | -4.55% | 11.76% | -12.96% | |
GET courseware:about | 169.5 | 16.67% | 7.69% | 15.38% | 15.38% | 15.38% | 14.29% | 21.43% | 6.25% | -89.11% | |
POST handler:capa:problem_show | 113 | 29.09% | 25.00% | 24.24% | 18.57% | 16.46% | 11.63% | 41.30% | 18.18% | -15.38% | |
GET forums:get_user_profile | 38.5 | - | 4.35% | - | - | - | -11.11% | -22.58% | -31.43% | -30.87% | |
POST handler:capa:problem_save | 31.5 | 50.00% | 38.46% | 54.93% | 48.65% | 34.15% | 26.44% | 22.22% | 19.57% | 14.13% | |
GET forums:followed_threads | 31 | 7.69% | 7.14% | 35.71% | 113.33% | 550.00% | 566.67% | 438.46% | 438.46% | 427.38% | |
POST forums:create_comment | 30.5 | -3.33% | -3.23% | -6.25% | -9.09% | 2.94% | 5.88% | - | - | -1.65% | |
POST forums:create_thread | 12 | 16.67% | 16.67% | 16.67% | 10.53% | 5.00% | - | -8.70% | -8.70% | -5.31% |