Currently Bok Choy tests are running with split in Jenkins, and it is my understanding that Lettuce tests are running with draft mongo modulestore in Jenkins.
Both sets of tests are capable of running with the other modulestore.
We need an automated way for both Bok Choy and Lettuce tests to run with all supported modulstores, at least once a day. Otherwise regressions will creep in.
BTW, what prompted me to file this bug is that while attempting to write a bok choy test for TNL-399, I discovered that the behavior described in does not exist with split. Therefore I can't write a test that would currently fail, unless there is a way to run the test against draft mongo.
Right off the bat it looks like there is a problem with running the lettuce tests on split. See this gist for the command and stacktrace of the first set of errors. I had set a breakpoint at common/djangoapps/util/models.py(41)uncompress_string()
Once again, we have been bitten by this issue.
This week's release is blocked by the Problem Hints feature not working with Split courses, as the feature was tested with Mongo courses: https://openedx.atlassian.net/browse/TNL-2542
With this pr we can now use our custom/env-var-controlled bok-choy job to run bok-choy using draft. However, at this point we have many failing tests (latest run here). So, we now have capability for running it nightly; however, I don't think we'd get value out of that until those tests are fixed.
I'll add a sub-task ticket here. Any suggestions on which team would take it up?
Over time I've built in jobs and capability for running bok-choy in draft. Now, anyone can run that on the custom bok-choy job by passing in env vars. The last run showed 27 failures.
So I removed the subtask related to fixing them all...it was originally our ticket because it required manual intervention on Jenkins in order to run it. But now that is not needed, so I moved it to an issue under TNL: TNL-4058.