Fix rate limiting issue in stage RCM

Description

RCM is failing in stage due to a rate limit error on an LMS endpoint:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:31 - CoursesApiDataLoader failed! 12:49:45 Traceback (most recent call last): 12:49:45 File "/edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py", line 28, in execute_loader 12:49:45 loader_class(*loader_args, **loader_kwargs).ingest() 12:49:45 File "/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/api.py", line 105, in ingest 12:49:45 response = future.result() 12:49:45 File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result 12:49:45 return self.__get_result() 12:49:45 File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result 12:49:45 raise self._exception 12:49:45 File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run 12:49:45 result = self.fn(*self.args, **self.kwargs) 12:49:45 File "/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/api.py", line 117, in _make_request 12:49:45 return self.api_client.courses().get(page=page, page_size=self.PAGE_SIZE, username=self.username) 12:49:45 File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/slumber/__init__.py", line 155, in get 12:49:45 resp = self._request("GET", params=kwargs) 12:49:45 File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/slumber/__init__.py", line 101, in _request 12:49:45 raise exception_class("Client Error %s: %s" % (resp.status_code, url), response=resp, content=resp.content) 12:49:45 slumber.exceptions.HttpClientError: Client Error 429: https://courses.stage.edx.org/api/courses/v1/courses/

Let's fix this by either adjusting the rate limit for this service account or adding in back-off logic in our code that hits this API.

ACs:

  • add exponential back-off in RCM's calls to this Studio endpoint

  • stage RCM runs stop failing on 429 responses from external APIs

Steps to Reproduce

None

Status

Story Points

1

Assignee

Unassigned

Reporter

Jason Myatt

Labels

Reach

None

Impact

None

Customer

None

Partner Manager

None

URL

None

Contributor Name

None

Groups with Read-Only Access

None

Actual Points

None

Category of Work

None

Stakeholders

None

Priority

Unset
Configure