A course import / export performance problem on production has been reported. What do you check on production to investigate?
Step-by-step guide
Use our awesome monitoring tools!
- Use NewRelic to check on the import_handler / export_handler transactions. (Be sure you're checking the correct environment, i.e. prod vs. edge!)
- prod-edx-edxapp-cms export_handler transactions:
- https://rpm.newrelic.com/accounts/88178/applications/3345427/transactions?type=app#id=5b225765625472616e73616374696f6e2f46756e6374696f6e2f636f6e74656e7473746f72652e76696577732e696d706f72745f6578706f72743a6578706f72745f68616e646c6572222c22225d
- prod-edx-edxapp-cms import_handler transactions:
- https://rpm.newrelic.com/accounts/88178/applications/3345427/transactions#id=5b225765625472616e73616374696f6e2f46756e6374696f6e2f636f6e74656e7473746f72652e76696577732e696d706f72745f6578706f72743a696d706f72745f68616e646c6572222c22225d
- If there are transaction traces for long transactions in the "Transaction traces" section, click on one.
- The trace will show details about time spent accessing MongoDB/MySQL and various parts of the code.
- Look for:
- long amounts of time spent in something that should be short
- high call counts that should be low
- Use Splunk to see if the durations have increased over time.
- For imports:
- Start a new Splunk search and use this query:
index="prod-edx" sourcetype=nginx ("POST /import/course-v1" OR "POST /import/i4x") | rex "/import/(?<course>\S*) HTTP/1.1. (?<return_code>\S*) (?<size>\S*) (?<duration>\S*)" | timechart max(duration) by course
- Use DataDog and its captured course import durations to look at trends:
- The following timer was added to track course import durations:
courselike_import.time
- You can view data from this timer in the Metrics Explorer:
- https://app.datadoghq.com/metric/explorer?live=true&page=0&is_auto=false&from_ts=1431965716564&to_ts=1434557716564&tile_size=m&exp_metric=courselike_import.time.avg,courselike_import.time.max,courselike_import.time.count,courselike_import.time.median,courselike_import.time.95percentile&exp_scope=&exp_agg=avg&exp_row_type=metric