2017-05-23 Hangout Agenda and Notes
The Hangout agenda is open to any topics related to deploying edX and running it in production. Add your agenda topics directly to ths document.
Joining the hangout does require an invitation. If you need one, please send an email to openedx-operations-hangout@edx.org from the account that you would like us to invite.
Agenda:
- Report from Madrid
- Docker Devstack
- Rabbit Upgrade
- ElasticSearch Update
- Community Questions
Notes:
- Majority of folks are still installing Vagrant devstack, though some folks have chosen to go the docker direction
- There have been networking issues at the conference that have made installation in either case difficult..
- Question about whether the tagging strategy for our images on Docker hub
- Question about CI in edX
- Small repositories/IDA typically are testing from Travis
- Integration tests across multiple systems happen in our staging environment
- For edX platform the test suite is extremely large. Tests are run in Jenkins – hosted by us – and tests are sharded across multiple workers. Jenkins sends a hook back to github with status.
- What is the best practice for flaky tests
- Currently we decorate them with the @flaky decorator which prevents the from breaking the build
- Tests can also be re-run with comments from Github, say, "jenkins run python"
- Is the code for running the Jenkins tests public?
- Yes it is, it leverages the Jenkins Job DSL. The specific DSL for the platform tests is available here: https://github.com/edx/jenkins-job-dsl/tree/master/testeng
- There is also an Ansible role for standing up Jenkins. That role is a little crufty, but is being actively improved.
- There was a questions about how we use Vertica in our edX data warehouse
- We recently upgraded Rabbit to 3.6.9. We stood up a second cluster and used the "shovel" tool to migrate data. In general it has worked well, though we have had some performance issues.
- Persistent grading is being rolled out and will pre-compute sub-section and course grades.
- We have been back filling the persisted graded in our system which will take about a month.
- We expect that this will be turned on in Ginko by default.
- There were some bugs in Ficus that we don't think will be fixed.
- We cannot speak the the specifics off the top of our head, but if there's interest in turning persistent grades before Ginko we could help understand what is and is not fixed in Ficus.
- It's also important to know that many of the grading bugs tend to be course specific, so you may see bugs that we haven't seen before in your environment
- We have cut the forums over to 1.5 and are in process of cutting over notes and other services to 1.5.