RCM began having numerous issues and was not passing for several weeks. These were caused in part by some Ecommerce Data Loader issues where some course runs had the Empty type. These were resolved by fixing the type to what it should be based on seats, but there is still an unknown about how these course runs had the Empty type in the first place. (One hypothesis is that the course runs were created directly in Studio and then assigned the Empty type through RCM, but the PC for the org (Purdue) said the runs were created in Publisher so not sure how it happened.)
Then there is the real issue. The CoursesApiDataLoader is very consistently (1 out of every 7 or 8 seems to not run into this issue, but it happens the vast majority of the time) failing due to the following exception:
This is a new error that has been happening starting around the middle of December 2019 (sometime after 12/16/2019). The cause of the error is that the courses API is reporting it has more results than it actually has. For example, the count of results on prod is 11926 and when the page_size is set to 50, that results in 239 pages. Except starting at page 235, there are no more results (https://courses.edx.org/api/courses/v1/courses/?page=235&page_size=50 )*. According to Cale, this is the result of the count we display is based on the overall QuerySet size which we then apply some filtering to in the results. However, since RCM is running as a staff user, we should have access to all results.
In my research into this, I was able to find a handful of the ~250 course runs that are no longer showing up. They are
All of these course runs are in course-discovery, are in Studio, but are not accessible in the API. They should be at the top of page 2 at https://courses.edx.org/api/courses/v1/courses/?page=2&page_size=50 , but are clearly not and if you try to hit the detail endpoint (https://courses.edx.org/api/courses/v1/courses/BerkeleyX/LUCS.1x/3T2015), you receive a 404 with the response "developer_message":"Course not found.".
Work should be done to figure out which course runs have stopped being returned and why and fixing it so they go back to being returned. Until that happens, RCM will likely continue to fail very often and we won't be updating a subset of our courseware.
* at the time of writing, it was page 235 that began resulting in no results, but this appears to be shifting. When I first was looking into this, it was page 234, but page 234 is now returning 19 results.