Slow transaction: /ccx.views:save_ccx
New Relic collected an interesting transaction trace
Timestamp: August 06, 2015 04:28
Transaction Duration: 25.226 seconds
Steps to Reproduce
You can reproduce by creating a CCX course:
Going to the Coach tab, and changing some due dates.
Note: you will need to enable CCX locally by setting FEATURES['CUSTOM_COURSES_EDX'] to True in your settings file
Reason for Variance
User Impact Summary
We are currently working on 2 approaches to reduce the time first one is on client side and other on server side, client side involves sending only those nodes to server which are changed so in this way instead of traversing whole structure only changed nodes are traverse which reduces latency and on server side we are trying to reduce the latency by bulk_operations on delete(), update() and create().Currently for deleting the ccx_override_field we first look the field in the db, fetch it and then delete in an loop and we tried to reduce this using bulk_delete where we collect the ccx_override_field ids to be deleted using hash and then delete all ccx_override_fields using single query.
, , after debugging request body of save_ccx it seems whenever we made a change (example due date) in an unit or its sub-unit the code at https://github.com/edx/edx-platform/blob/master/lms/djangoapps/ccx/views.py#L208 is traversing the whole structure of course sending from https://github.com/edx/edx-platform/blob/19604a4a6eee6f65bb048700124923972ebc1f57/lms/static/js/ccx/schedule.js#L222, an solution can be send only those nodes and their children which have changed values instead of sending the whole outline of course for traversing.
, , , that profile is really interesting. It doesn't look like it's spending too much time in "commit", as it looked like in the new relic trace.
It also looks like it could be an easy win to do the deletes all at once.
Attached a profiler dump of this operation taken from my devstack. It also points to too many MySQL operations as the main cause.