Kyle McCormick : removing github requirements and moving them to PyPI
Working example: Blockstore. Pushed to PyPI with a GH action
In repo: 181 files
In PyPI zip: 60 files
In Olive edx-platform, we have a list of libraries in github.in where we install the entire repo (as of Olive, there are 9 of these)
As of Olive, this includes the Blockstore
Developers don’t always know the benefits of PyPI, and in the middle of an upgrade it’s probably a waste. github.in is meant for temporary things
pip install requirements/edx/base.in on a container that already has everything installed (ie a noop): 55s, the VAST majority of the time being spent on Github cloning and running setup.py
In the latest edx-platform (ie master)
only 3 GH libraries, two of which we expect to be removed shortly
pip install requirements/edx/base.in on a container that already has everything installed (ie a noop): 36s!
We may be able to get down to 25s when we remove everything
Q&A
[Becca] Q: Was the work mostly just copying over the GH action?
A: In some cases, yes. For Codejail, setup.py was missing a few modules so had to track those down
[Jeremy] Q: 1. Yay! Have we done investigation on wheels vs source distribution?
A: Not yet, but there’s a ticket for it.
[Jeremy] Note: At some point it may make a difference which PyPI mirror you’re using
[Kyle] Also, pruning edx-platform requirements will be another major time save
Cross Pollination
Tools/features/opportunities
Tips and tricks
Roadmap news
Your Name
Your Item to Share
Additional notes
Jeremy
Devstack minor fixes
Renamed dev.down make target (Diana), fixed some documentation (Diana), fixing SQL init script (Becca)
Note from Becca: by “fixing” we mean “not including the demo course.” With the split module store, it becomes too tricky to keep everything in the initial sql without having references to mongo that won’t exist on a first provision
If you have an idea for follow up actions, add it.
Or if you aren’t sure, leave it blank.
We’ll discuss all actions either way.
Your Name
Your Item of Challenge to Share
Idea for Follow Up Action
Kyle
Lot of unknowns with devx work, which makes it hard to break our work into small issues that are good for newcomers
devstack work?
looking for docker optimizations
image layering
good for people who know docker but don’t necessarily have open edx knowledge
Mafer note: Maybe it's a good idea to rethink the order of the layers in docker or perhaps have names in the docker file to be able to build a specific part.
issue reportingDifference between core developers and majority of small-service users
small-instance users would strongly prefer not to have to build as often since that takes so much time. Needs to be easier to customize without a full build for every change
Jeremy
Formulating a concise mental model of our development environment
What does our env look like? What is standard between repos? What is tied to the orchestration layer?
On Arch BOM we’re going in and out of devstack mindset, hard to dedicate time to thinking through this
Should there be a hard line between what the individual repo provides vs what the environment provides?
this is changing with Tutor
One of the goals is to smooth out the differences between repos
some Makefile targets are common across repos, some common-ish. Would be nice to have a common interface across all repos
Jeremy
Smoothing learning curves
Things like Kubernetes, docker-compose, Terraform, Ansible, etc. have a wide range of possible knowledge depth; find appropriate resources for each level
It’s hard to identify a minimum set of required knowledge