Preface

Otto's basket creation and CyberSource payment endpoints will be load tested at 2x and 10x the peak production load of comparable views on the LMS. In the last month, the LMS' change_enrollment view experienced peak load of about 75 RPM in production, while the create_order view saw peak load of about 3 RPM in production. Given these stats, the new /commerce/baskets/ endpoint (on the LMS) will experience up to 750 RPM during load testing, while the Otto endpoints involved in processing CyberSource payments (/api/v2/payment/processors/, /api/v2/baskets/, /payment/cybersource/notify/, and /api/v2/baskets/:id/order/) will experience up to 30 RPM during load testing.

Execution probabilities for the two Locust tasks being used to simulate free and paid purchases are derived from a conservative estimate from marketing placing the percentage of paid enrollments at 2% of all enrollments.

Two instances of the ecommerce service and four instances of the LMS were deployed for this load test.

TL;DR

At twice of its expected peak production load (150 RPM), 95% of requests to /commerce/baskets/ complete in 500 ms. At twice of their expected peak production load (6 RPM), 95% of requests to /api/v2/payment/processors/ complete in 160 ms, 95% of requests to /api/v2/baskets/ complete in 170 ms, 95% of requests to /payment/cybersource/notify/ complete in 420 ms, and 95% of requests to /api/v2/baskets/:id/order/ complete in 130 ms.

At ten times its expected peak production load (750 RPM), 95% of requests to /commerce/baskets/ complete in 1900 ms. At ten times their expected peak production load (30 RPM), 95% of requests to /api/v2/payment/processors/ complete in 780 ms, 95% of requests to /api/v2/baskets/ complete in 590 ms, 95% of requests to /payment/cybersource/notify/ complete in 1700 ms, and 95% of requests to /api/v2/baskets/:id/order/ complete in 760 ms.

Endpoints performing fulfillment operations start throwing errors at approximately 800 RPM.

Results

Below is an overview of web transaction response time for the ecommerce service at increasing levels of load:

The performance of Python code running in process on the service and of database queries scale well, remaining roughly constant as load increases.

Errors begin to appear at approximately 800 RPM, when calls to external services (i.e., the enrollment API) made by endpoints performing fulfillment operations appear to start piling up:

Enrollment API response times appear to remain roughly constant at 100 ms, irrespective of load:

The enrollment API's performance profile is explained in large part by the fact that LMS database performance remains approximately constant at increasing levels of load:

Endpoints which don't rely on the enrollment API scale better than those which do:

Bonus: The payment flow is capable of handling load significantly higher 30 RPM. Below is an overview of the CyberSource notification view's response times at 150 RPM (50x peak production load):

Browser not supported