How to Investigate Course Import and Export Performance

A course import / export performance problem on production has been reported. What do you check on production to investigate?

Step-by-step guide

Use our awesome monitoring tools!

1. Use NewRelic to check on the import_handler / export_handler transactions. (Be sure you're checking the correct environment, i.e. prod vs. edge!)
   
   1. prod-edx-edxapp-cms export_handler transactions:
      1. https://rpm.newrelic.com/accounts/88178/applications/3345427/transactions?type=app#id=5b225765625472616e73616374696f6e2f46756e6374696f6e2f636f6e74656e7473746f72652e76696577732e696d706f72745f6578706f72743a6578706f72745f68616e646c6572222c22225d
   2. prod-edx-edxapp-cms import_handler transactions:
      1. https://rpm.newrelic.com/accounts/88178/applications/3345427/transactions#id=5b225765625472616e73616374696f6e2f46756e6374696f6e2f636f6e74656e7473746f72652e76696577732e696d706f72745f6578706f72743a696d706f72745f68616e646c6572222c22225d
   3. If there are transaction traces for long transactions in the "Transaction traces" section, click on one.
      
      1. The trace will show details about time spent accessing MongoDB/MySQL and various parts of the code.
      2. Look for:
         1. long amounts of time spent in something that should be short
         2. high call counts that should be low
2. Use Splunk to see if the durations have increased over time.
   1. For imports:
      1. Start a new Splunk search and use this query:
      2. index="prod-edx" sourcetype=nginx ("POST /import/course-v1" OR "POST /import/i4x")  | rex "/import/(?<course>\S*) HTTP/1.1. (?<return_code>\S*) (?<size>\S*) (?<duration>\S*)" |  timechart max(duration) by course
3. Use DataDog and its captured course import durations to look at trends:
   1. The following timer was added to track course import durations: courselike_import.time
   2. You can view data from this timer in the Metrics Explorer:
      1. https://app.datadoghq.com/metric/explorer?live=true&page=0&is_auto=false&from_ts=1431965716564&to_ts=1434557716564&tile_size=m&exp_metric=courselike_import.time.avg,courselike_import.time.max,courselike_import.time.count,courselike_import.time.median,courselike_import.time.95percentile&exp_scope=&exp_agg=avg&exp_row_type=metric