Decrease Mongo Response Timeout

Description

During a recent outage of production, connections to an un-healthy mongo server became really slow but continued to work. This tied up gunicorn workers and resulted in an LMS/CMS outage. If possible we should lower the current response timeout for responses from mongo.

More details about the incident can be found here: https://openedx.atlassian.net/wiki/display/EdxOps/2016-04-12+Mongo+Instance+Became+Unhealthy
AC:

  • Determine the current timeout for mongo responses

  • If reasonable, reduce it further so that gunicorn workers recover faster.

Steps to Reproduce

None

Current Behavior

None

Expected Behavior

None

Reason for Variance

None

Release Notes

None

User Impact Summary

None

Status

Assignee

Kevin Falcone

Reporter

Feanil Patel

Reach

None

Impact

None

Customer

None

Partner Manager

None

URL

None

Contributor Name

None

Groups with Read-Only Access

None

Actual Points

None

Category of Work

None

Stakeholders

None

Story Points

1

Sprint

None

Priority

Unset
Configure