Potential Organizational Velocity Improvements
Proposals
| Priority | Project Name | Dependencies | Description | Projects Which Benefit | Notes | Pain (0-5) | Frequency (0-5) | Cost (0-5) |
|---|---|---|---|---|---|---|---|---|
| 1 | Speed up tests locally | None | Currently, running a full test suite takes 'a long time', so people push to CI instead, which also takes a long time. If we shortened the full tests suite, or gave a good selection of fast smoke tests, devs could have a tighter turnaround time. Perhaps using per-test coverage data to do a reverse lookup based on edited lines?
| All projects in edx-platform | BJP (4/2015) - TestEng undertook an effort in summer 2014 to evaluate a tool to do this for us (i.e. auto-determine which tests to run using nose-knows ). However, we think it is worth revisiting. See TE-539 - Getting issue details... STATUS | 1 | 5 | 2 |
| 1 | Speed up tests in CI | None | For an average PR, just running the test jobs to completion takes 40 - 50 minutes, not including time in queue. That is enough time that developers context switch, which costs time in addition
| All projects in edx-platform | TestEng is actively working on this, which lowers the usefulness of Platform putting time in.
BJP (4/2015) - A truer solution for this would be a more mature build pipeline (likely coupled with the above item). That would involve testing a subset on every PR commit, and having a way to run more tests at/near merge-time. This could also imply a different branching strategy. | 1 | 5 | 2 |
| Move grade storage to submissions app | None | Dave Ormsbee can fill in more, but essentially, we can change the storage model for grades such that they are append-only, and use that to make score computation for things like the progress page more predictable and less expensive. |
| 4 | 1 | 4 | ||
| Make Runtimes per-app, rather than per-modulestore | None | Convert from Storage-centric runtimes to Application-centric runtimes | 3 | 2 | 5 | |||
| Provide access to field storage via FieldData interface, rather than via Modulestore api methods | None | We can move towards the XBlock-standard model of field storage, which should make our runtimes have to jump through fewer hoops to work correctly, and will let us re-use some building blocks across different data stores. | 3 | 1 | 5 | |||
| 2 | Kill XModule | None | Lots of technical debt here, some of which will go away with conversion to XBlock. |
| 4 | 3 | 5 | |
| 4 | Kill XModule-specific APIs/workarounds | Kill XModule | Lots more technical debt related to making two classes look like one class. Contributes many of the most brittle code pieces. |
| 5 | 1 | 5 | |
| Switch InstructorTask to generate micro-tasks and use a distributed counter (memcache) for results on particular jobs | None | Asynchronous tasks pattern gives a strategy option. This code is really hairy right now, and jumps through a lot of hoops in order to not spam the relational database. |
| 4 | 1 | 3 | ||
| 1 | Set up infrastructure for building performance tests (working with Corey, locust.io) | None | Setting up performance tests from scratch introduces risk to any project that needs to measure serverside performance. |
| 3 | 3 | 2 | |
| 3 | Move LMS to require.js | None | Integration of Require JS into the system |
| 4 | 2 | 3 | |
| 3 | Load XModule/XBlock javascript with require.js | Move LMS to require.js | Moving XModule to require.js would allow us to get rid of the large blob of all of the XModule js that's loaded on every courseware page. Moving XBlock to requirejs would allow us to stop including js snippets on the main page directly (and hopefully cache the offloaded js). |
| 4 | 2 | 4 | |
| Clean up certificates code | None | The certificates generation code is a mess |
| 5 | 1 | 4 | ||
| 1 | Pre-process and check-in Javascript and CSS | None | Currently, django collectstatic, and the requirejs optimizer, eat up lots of time any time they're invoked. If we separated the build of all of our javascript and CSS from the execution of the platform, then we'd save time on tests and on deployment, with the only cost being a test that verifies that no one forgot the asset compilation/checkin step. |
| 1 | 5 | 1 |
Definitions
Pain: How bad is this right now?
Frequency: How often do we have to experience the pain?
Cost: How much will this cost to fix?
All estimates are rough and relative only to the rest of the table, in Calen Pennington (Deactivated)'s estimation.
Input Data
Summary of Talking to Teams
I talked to most of the development teams about what they felt had made them slow historically. The two most common themes were that Jenkins and the test suite are slow to run, and that devstack setup is somewhat unstable, especially when switching between environments, or for occasional devstack users. Improving the LMS javascript architecture also came up from two different teams (T&L and destEdX).
Per-Team Notes
Analytics (Gabe)
- Static assets compilation is slow
- Paver tasks aren't well documented
- Devstack requires rebuild after an absence from platform
- issues with:
- vmware shutdown
- mongodb lock file
- issues with:
- Lack of Python IDE
- Common setup tips?
- Long test cycle
- Leads to task switching
- no time to optimize workflow
- cluster provisioning
- 2hr acceptance tests
- poor tooling for debugging
- no automated validation
- Dockerize vs long-lived cluster
- stack uncertainty
- code review tools
- outdated comments
- Support of legacy infrastructure - no tests, no non-prod environments, very brittle
- analytics-server
- analytics-exporter
- bugs with low customer impact but cause periodic manual intervention
Destination edX (Steve S)
- paver run_pylint is slow
- jenkins build times
- rerun particular shards?
- devstack reinstalls on new machines
- lack of theming/isolation
- risk of breaking whitelabel/opensource
- distractions
- interaction with delicate code
- third party auth
- certificates
- lms javascript architecture
- a/b testing adds a tax
Solutions (Steve M)
- devstack issues (~5 person-days)
- switching environments
- Code reviews are slow
- Enforcement of standards may be slowing us down without providing value to our users or velocity
- example: line length restrictions, whitespace nits
- jenkins is slow
- ramping up on additional testing requirements
- bok_choy
- building the right tests?
- pep8 cleanliness combined w/ slow jenkins
- pep8 enforcement a waste of time?
- Do we have default/shared editor configurations?
- Do we have a blessed technology list to help people make decisions about what technologies to use?
- [ cdodge ] "flakey" acceptance tests cause many manual re-triggerings of Jenkins builds which take about 30-45 minutes to complete.
Teaching and Learning (Andy)
- require.js/backbone lacking in LMS
- Lack of XBlock editing in studio
- lack of architecture docs (Daniel)
- old libraries -> unfixed bugs to work around (Daniel)
- interteam PR review process
- no code ownership
- devstack reinstalls are hard
- jenkins turnaround is too long
- quality included as part of tests results -> builds are always red until the end of a PR
Mobile (Nimisha + Chris)
- Video module fires
- delicate
- legacy code
- lack of mobile test environment
- mobile dev environments
- IDE issues
- eclipse for android is slow
- lack of standards for APIs
- hard to communicate with other teams
- hard to know who to talk to at first
- interteam projects
- no mobile escalation team
- jenkins turnaround is slow
- devstack setup
- S3 for video pipeline
- content library missing