Investigate / Fix Recent RCM issues

Description

RCM began having numerous issues and was not passing for several weeks. These were caused in part by some Ecommerce Data Loader issues where some course runs had the Empty type. These were resolved by fixing the type to what it should be based on seats, but there is still an unknown about how these course runs had the Empty type in the first place. (One hypothesis is that the course runs were created directly in Studio and then assigned the Empty type through RCM, but the PC for the org (Purdue) said the runs were created in Publisher so not sure how it happened.)

Then there is the real issue. The CoursesApiDataLoader is very consistently (1 out of every 7 or 8 seems to not run into this issue, but it happens the vast majority of the time) failing due to the following exception:

This is a new error that has been happening starting around the middle of December 2019 (sometime after 12/16/2019). The cause of the error is that the courses API is reporting it has more results than it actually has. For example, the count of results on prod is 11926 and when the page_size is set to 50, that results in 239 pages. Except starting at page 235, there are no more results (https://courses.edx.org/api/courses/v1/courses/?page=235&page_size=50 )*. According to Cale, this is the result of the count we display is based on the overall QuerySet size which we then apply some filtering to in the results. However, since RCM is running as a staff user, we should have access to all results.

In my research into this, I was able to find a handful of the ~250 course runs that are no longer showing up. They are

  • BerkeleyX/LUCS.1x/3T2015

  • BerkeleyX/LUCS.2x/3T2015

  • BerkeleyX/LUCS.3x/1T2016

  • BerkeleyX/LUCS.4x/1T2016

All of these course runs are in course-discovery, are in Studio, but are not accessible in the API. They should be at the top of page 2 at https://courses.edx.org/api/courses/v1/courses/?page=2&page_size=50 , but are clearly not and if you try to hit the detail endpoint (https://courses.edx.org/api/courses/v1/courses/BerkeleyX/LUCS.1x/3T2015), you receive a 404 with the response "developer_message":"Course not found.".

Work should be done to figure out which course runs have stopped being returned and why and fixing it so they go back to being returned. Until that happens, RCM will likely continue to fail very often and we won't be updating a subset of our courseware.

* at the time of writing, it was page 235 that began resulting in no results, but this appears to be shifting. When I first was looking into this, it was page 234, but page 234 is now returning 19 results.

Steps to Reproduce

None

Story Points

None

Assignee

Dillon Dumesnil

Reporter

Jason Myatt

Labels

Reach

None

Impact

None

Platform Area

None

Customer

None

Partner Manager

None

URL

None

Contributor Name

None

Groups with Read-Only Access

None

Actual Points

None

Category of Work

None

Platform Map Area (Levels 1 & 2)

None

Platform Map Area (Levels 3 & 4)

None

Sprint

Priority

Unset
Configure