...
The raw results are attached to the page. An analysis comparing each of the runs against the others as attached in .ods format. Those comparisons are also reproduced below.
Summarizing, we can draw several important conclusions:
- It's likely that the candidate revision isn't causing performance degradation.
- Candidate beat's Re-baseline by more than the difference between how much both Re-baseline and Candidate lose to Baseline
- Our test infrastructure isn't providing enough isolation to make tests repeatable.
- Re-baseline is slower than baseline, despite running the same driver and server code.
| Name | # reqs | 50.00% | 66.00% | 75.00% | 80.00% | 90.00% | 95.00% | 98.00% | 99.00% | 100.00% | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Candidate/Re-baseline | POST handler:video:save_user_state | 5110.5 | -7.94% | -8.96% | -8.70% | -9.86% | -11.69% | -12.20% | -8.70% | 20.00% | 141.68% |
| GET handler:capa:problem_get | 3437 | -16.87% | -16.67% | -15.79% | -14.14% | -10.00% | -8.33% | -7.69% | 66.67% | 175.93% | |
| GET courseware:index | 3416.5 | - | 1.59% | -1.47% | -1.43% | -3.95% | -7.78% | 9.09% | - | 45.95% | |
| POST handler:capa:problem_get | 2774.5 | -1.79% | -3.28% | -7.35% | -13.16% | -11.36% | -10.10% | -9.09% | -8.33% | 109.00% | |
| GET handler:video:get_transcript | 2518 | - | - | - | - | -1.56% | -1.47% | -1.23% | -9.09% | 134.47% | |
| GET courseware:mktg_course_about | 2235 | -1.82% | -1.69% | -1.64% | -1.61% | -1.54% | 4.29% | 32.65% | 7.69% | 88.91% | |
| POST handler:goto_position | 1877.5 | -6.74% | -9.09% | - | -8.33% | -7.69% | -7.14% | - | 26.67% | 150.42% | |
| GET handler:goto_position | 1805 | -10.10% | -8.33% | -8.33% | -7.69% | -7.14% | -13.33% | -12.50% | -5.88% | 98.89% | |
| GET handler:capa:problem_check | 1762.5 | -13.25% | -15.22% | -13.27% | -11.00% | -11.82% | -16.67% | -21.43% | -18.75% | 54.87% | |
| GET forums:single_thread | 724.5 | -4.41% | -5.48% | -5.33% | -3.90% | -2.38% | 18.18% | 12.50% | 20.00% | -4.72% | |
| GET courseware:course_info | 694 | -1.05% | -1.00% | - | - | - | - | - | -57.58% | 68.25% | |
| GET handler:video:save_user_state | 599 | -1.92% | - | - | - | -1.64% | -1.49% | -3.45% | -38.13% | 1.49% | |
| POST handler:capa:problem_check | 468 | -3.23% | -8.00% | -9.76% | -10.47% | -12.37% | -18.18% | -23.85% | -21.43% | 404.52% | |
| GET handler:capa:problem_show | 428 | -15.00% | -17.98% | -18.09% | -17.17% | -12.73% | -16.67% | - | 53.33% | 941.87% | |
| GET auto_auth | 395.5 | - | - | - | 4.55% | 4.17% | 55.56% | 36.36% | 5.56% | 1.91% | |
| POST enroll | 394 | -2.15% | -3.00% | - | -9.09% | - | -8.33% | - | - | 317.68% | |
| GET forums:forum_form_discussion | 294.5 | - | - | -1.69% | - | 4.55% | 12.50% | 6.67% | 20.00% | 5.36% | |
| GET handler:capa:problem_save | 277 | -14.81% | -15.73% | -18.75% | -19.19% | -12.73% | -16.67% | -15.38% | - | 49.35% | |
| GET courseware:progress | 268 | -3.70% | -9.78% | -13.00% | -18.18% | -19.17% | -28.57% | -6.25% | -15.79% | -21.07% | |
| GET forums:inline_discussion | 209.5 | -3.45% | -3.23% | - | -2.94% | -2.70% | -13.56% | -28.57% | - | -9.94% | |
| GET forums:search_topic | 80 | - | -2.50% | -4.76% | -2.38% | -6.38% | -18.97% | -20.63% | -66.84% | -66.67% | |
| GET courseware:about | 69 | - | - | - | - | 6.67% | 137.50% | 900.00% | 900.00% | 887.28% | |
| POST handler:capa:problem_show | 27.5 | -12.68% | -8.00% | -15.85% | -16.87% | -9.78% | -10.42% | -23.08% | -23.08% | -21.97% | |
| GET forums:get_user_profile | 19 | - | -4.17% | -4.17% | - | - | 83.33% | 83.33% | 83.33% | 80.78% | |
| POST forums:create_comment | 17.5 | - | 3.33% | 3.33% | 3.33% | -8.57% | -2.78% | -2.78% | -2.78% | -2.24% | |
| GET forums:followed_threads | 12.5 | -7.14% | -6.67% | -26.32% | -56.25% | -88.46% | -88.57% | -88.57% | -88.57% | -88.75% | |
| POST handler:capa:problem_save | 6 | -24.44% | -22.22% | -29.09% | -29.09% | -9.09% | -9.09% | -9.09% | -9.09% | -3.81% | |
| POST forums:create_thread | 2 | 14.29% | 14.29% | 14.29% | 14.29% | 14.29% | 14.29% | 14.29% | 14.29% | 12.15% | |
| Name | # reqs | 50% | 66% | 75% | 80% | 90% | 95% | 98% | 99% | 100% | |
| Candidate/Baseline | POST handler:video:save_user_state | 10986.5 | 11.54% | 12.96% | 12.50% | 12.28% | 11.48% | 12.50% | 18.31% | 39.53% | 3.54% |
| GET handler:capa:problem_get | 4405.5 | - | 1.35% | 2.56% | 4.94% | 3.13% | 10.00% | 9.09% | 108.33% | 13.33% | |
| GET courseware:index | 8196.5 | 6.67% | 8.47% | 6.35% | 6.15% | 7.35% | 12.16% | 30.43% | 7.69% | 4.48% | |
| POST handler:capa:problem_get | 8503 | 10.00% | 3.51% | -4.55% | -4.35% | - | -1.11% | - | - | -16.83% | |
| GET handler:video:get_transcript | 5316 | 20.00% | 18.75% | 20.41% | 20.00% | 18.87% | 21.82% | 31.15% | 42.86% | -6.19% | |
| GET courseware:mktg_course_about | 5481 | 12.50% | 13.73% | 15.38% | 17.31% | 18.52% | 30.36% | 113.11% | 79.49% | -4.23% | |
| POST handler:goto_position | 5405.5 | 9.21% | 5.26% | 10.00% | - | - | 8.33% | 7.69% | 35.71% | 1.66% | |
| GET handler:goto_position | 2225.5 | 4.71% | 10.00% | - | 9.09% | 8.33% | - | - | 14.29% | 96.50% | |
| GET handler:capa:problem_check | 2841.5 | 4.35% | 4.00% | 7.59% | 5.95% | 3.19% | - | - | 8.33% | -1.86% | |
| GET forums:single_thread | 1838.5 | 3.17% | 4.55% | 2.90% | 4.23% | 5.13% | 38.30% | 12.50% | 14.29% | -28.79% | |
| GET courseware:course_info | 1678 | 9.30% | 11.24% | 9.89% | 8.70% | 15.79% | 11.11% | 18.18% | - | -9.50% | |
| GET handler:video:save_user_state | 661.5 | 15.91% | 17.02% | 18.75% | 20.83% | 20.00% | 24.53% | 42.37% | 62.30% | 12.79% | |
| POST handler:capa:problem_check | 1836.5 | 5.26% | 2.99% | 5.71% | 6.94% | 3.66% | -1.10% | -1.00% | - | -10.73% | |
| GET handler:capa:problem_show | 830 | 7.94% | 4.29% | 2.67% | 5.13% | 4.35% | - | 18.18% | 91.67% | 27.64% | |
| GET auto_auth | 792.5 | 10.53% | 10.00% | 10.00% | 9.52% | 13.64% | 61.54% | 7.14% | 11.76% | -4.98% | |
| POST enroll | 791 | 22.97% | 24.36% | 23.46% | 21.95% | 30.95% | 25.00% | 26.32% | 18.18% | -60.47% | |
| GET forums:forum_form_discussion | 662 | 1.89% | 1.82% | 1.75% | 3.45% | 11.29% | 28.57% | 14.29% | 26.32% | 1.07% | |
| GET handler:capa:problem_save | 548.5 | 6.15% | 5.63% | 2.63% | 1.27% | 5.49% | - | - | 27.27% | -88.92% | |
| GET courseware:progress | 692 | 9.86% | 10.67% | 11.54% | 9.76% | - | -9.09% | 7.14% | 6.67% | -7.62% | |
| GET forums:inline_discussion | 505 | 7.69% | 11.11% | 10.34% | 10.00% | 9.09% | 18.60% | -9.09% | - | -14.96% | |
| GET forums:search_topic | 181.5 | 11.76% | 11.43% | 11.11% | 10.81% | 7.32% | 4.44% | -24.24% | -62.94% | -70.99% | |
| GET courseware:about | 163.5 | 16.67% | 7.69% | 15.38% | 15.38% | 23.08% | 171.43% | 1114.29% | 962.50% | 7.56% | |
| POST handler:capa:problem_show | 113.5 | 12.73% | 15.00% | 4.55% | -1.43% | 5.06% | - | 8.70% | -9.09% | -33.97% | |
| GET forums:get_user_profile | 36.5 | - | - | -4.17% | - | - | 62.96% | 41.94% | 25.71% | 24.98% | |
| POST forums:create_comment | 28 | -3.33% | - | -3.13% | -6.06% | -5.88% | 2.94% | -2.78% | -2.78% | -3.86% | |
| GET forums:followed_threads | 30.5 | - | - | - | -6.67% | -25.00% | -23.81% | -38.46% | -38.46% | -40.68% | |
| POST handler:capa:problem_save | 33.5 | 13.33% | 7.69% | 9.86% | 5.41% | 21.95% | 14.94% | 11.11% | 8.70% | 9.78% | |
| POST forums:create_thread | 12 | 33.33% | 33.33% | 33.33% | 26.32% | 20.00% | 14.29% | 4.35% | 4.35% | 6.19% | |
| Name | # reqs | 50% | 66%` | 75% | 80% | 90% | 95% | 98% | 99% | 100% | |
| Re-baseline/Baseline | POST handler:video:save_user_state | 10792 | 21.15% | 24.07% | 23.21% | 24.56% | 26.23% | 28.13% | 29.58% | 16.28% | -57.16% |
| POST handler:capa:problem_get | 8464.5 | 12.00% | 7.02% | 3.03% | 10.14% | 12.82% | 10.00% | 10.00% | 9.09% | -60.21% | |
| GET courseware:index | 8426 | 6.67% | 6.78% | 7.94% | 7.69% | 11.76% | 21.62% | 19.57% | 7.69% | -28.41% | |
| GET courseware:mktg_course_about | 5597 | 14.58% | 15.69% | 17.31% | 19.23% | 20.37% | 25.00% | 60.66% | 66.67% | -49.30% | |
| POST handler:goto_position | 5390 | 17.11% | 15.79% | 10.00% | 9.09% | 8.33% | 16.67% | 7.69% | 7.14% | -59.40% | |
| GET handler:video:get_transcript | 5257 | 20.00% | 18.75% | 20.41% | 20.00% | 20.75% | 23.64% | 32.79% | 57.14% | -59.99% | |
| GET handler:capa:problem_get | 4354.5 | 20.29% | 21.62% | 21.79% | 22.22% | 14.58% | 20.00% | 18.18% | 25.00% | -58.93% | |
| GET handler:capa:problem_check | 2800 | 20.29% | 22.67% | 24.05% | 19.05% | 17.02% | 20.00% | 27.27% | 33.33% | -36.63% | |
| GET handler:goto_position | 2186.5 | 16.47% | 20.00% | 9.09% | 18.18% | 16.67% | 15.38% | 14.29% | 21.43% | -1.20% | |
| GET forums:single_thread | 1897 | 7.94% | 10.61% | 8.70% | 8.45% | 7.69% | 17.02% | - | -4.76% | -25.26% | |
| POST handler:capa:problem_check | 1833.5 | 8.77% | 11.94% | 17.14% | 19.44% | 18.29% | 20.88% | 30.00% | 27.27% | -82.31% | |
| GET courseware:course_info | 1706 | 10.47% | 12.36% | 9.89% | 8.70% | 15.79% | 11.11% | 18.18% | 135.71% | -46.21% | |
| GET handler:capa:problem_show | 806 | 26.98% | 27.14% | 25.33% | 26.92% | 19.57% | 20.00% | 18.18% | 25.00% | -87.75% | |
| GET auto_auth | 783 | 10.53% | 10.00% | 10.00% | 4.76% | 9.09% | 3.85% | -21.43% | 5.88% | -6.76% | |
| POST enroll | 782 | 25.68% | 28.21% | 23.46% | 34.15% | 30.95% | 36.36% | 26.32% | 18.18% | -90.54% | |
| GET courseware:progress | 703 | 14.08% | 22.67% | 28.21% | 34.15% | 23.71% | 27.27% | 14.29% | 26.67% | 17.05% | |
| GET forums:forum_form_discussion | 682.5 | 1.89% | 1.82% | 3.51% | 3.45% | 6.45% | 14.29% | 7.14% | 5.26% | -4.07% | |
| GET handler:video:save_user_state | 654.5 | 18.18% | 17.02% | 18.75% | 20.83% | 22.00% | 26.42% | 47.46% | 162.30% | 11.14% | |
| GET handler:capa:problem_save | 539.5 | 24.62% | 25.35% | 26.32% | 25.32% | 20.88% | 20.00% | 18.18% | 27.27% | -92.58% | |
| GET forums:inline_discussion | 528.5 | 11.54% | 14.81% | 10.34% | 13.33% | 12.12% | 37.21% | 27.27% | - | -5.57% | |
| GET forums:search_topic | 189.5 | 11.76% | 14.29% | 16.67% | 13.51% | 14.63% | 28.89% | -4.55% | 11.76% | -12.96% | |
| GET courseware:about | 169.5 | 16.67% | 7.69% | 15.38% | 15.38% | 15.38% | 14.29% | 21.43% | 6.25% | -89.11% | |
| POST handler:capa:problem_show | 113 | 29.09% | 25.00% | 24.24% | 18.57% | 16.46% | 11.63% | 41.30% | 18.18% | -15.38% | |
| GET forums:get_user_profile | 38.5 | - | 4.35% | - | - | - | -11.11% | -22.58% | -31.43% | -30.87% | |
| POST handler:capa:problem_save | 31.5 | 50.00% | 38.46% | 54.93% | 48.65% | 34.15% | 26.44% | 22.22% | 19.57% | 14.13% | |
| GET forums:followed_threads | 31 | 7.69% | 7.14% | 35.71% | 113.33% | 550.00% | 566.67% | 438.46% | 438.46% | 427.38% | |
| POST forums:create_comment | 30.5 | -3.33% | -3.23% | -6.25% | -9.09% | 2.94% | 5.88% | - | - | -1.65% | |
| POST forums:create_thread | 12 | 16.67% | 16.67% | 16.67% | 10.53% | 5.00% | - | -8.70% | -8.70% | -5.31% |