Arch Tea Time 2020-04-23 (CookieCutters, Devstack, Deployments)
Topics
Cookiecutters - does it make sense to merge the IDA and app ones, or do we need to keep them both updated?
Discussion going on about merging of IDA and app cookiecutters
Lots of commonalities – tox usage, dep handling, etc.
Benefit: less of a burden to maintain multiple cookiecutters
Minus: effort in merging
Was deferred during Dj22 upgrade because it wasn’t expected that there would be new services during upgrade… but there were Now those need to be upgraded
Why not merge?
Some differences, could be hard to put them in one repo.
Similarities > differences
Would be doable to ask questions when “cookie-cutting”, to avoid unnecessary file in generated repo.
Could both cookie-cutters inherit from a template?
Not a pattern in current cookie-cutter setup
What’s the effort of merging vs. updating both? Lots on Arch’s plate right now.
Robert: Current cookie-cutters are used as references… would merging them and adding lots of conditional logic make it less useful as a reference?
You could generate a new empty project from the cookiecutter and use that as a reference instead.
Jeremy: We have a lot of cookie-cutters (3 Python, 1 frontend, maybe more). Would be good to reduce them.
Future cookie cutters: Python project that doesn’t use Django.
Tim: It might be worth just reducing the amount of boiler-plate so that the cookie-cutters are fairly lightweight.
Front-end is gaining more cookie-cutters, but they aren’t cookie-cutters: they are just template that you copy and then modify. David’s going to run with that strategy.
Kyle: edx/configuration was filled with conditional logic and now we’re moving away from it and towards duplicating config. Is that lesson worthwhile here?
Consensus: Probably not, especially given that generating the cookie-cutter gives a readable repo.
What’s the point of a cookie-cutter?
Getting a service/app off the ground; the very beginning of the dev lifecycle
Not necessarily meant to be for maintaining that service/app. Docs and other tools might be better for that.
Dependencies in python - where to find the “binaries” and the latest versions and the best practice to ensure the latest versions are used. (Especially among various services).
How to ensure which version of a dependency we are currently using?
How to check which versions of dependencies are in my environment?
Can run “pip freeze”
“make upgrade” updates your requirements file.
“make requirements” updates your environment based on the requirements file.
Devstack
Why are we seeing so many issues recently (JWT, memory spikes, database corruption)?
In the future, what would an ideal developer stack(s) look like?
Option 1: Enhancements to current Devstack
Option 2: Services are isolated
Would this serve the purpose of local development and testing?
Can this be feasible?
What level of technical understanding of Docker/etc that we expect developers to have?
Option A: Run stubs for services that you depend on.
Option B: Explicitly run the services that you need.
Option C: Make services more independent and more tolerant of non-existence of other services.
For example, add built-in circuit breakers.
What are alternatives to Devstack?
Option 1: No centralized Devstack repo, but having each IDA have its own Docker files and Containers.
Option 2: Just use sandboxes for development and testing
Moving from today to there
Do we keep our current devstack repo or start over?
Option A: Moving to a radically different place, but do it in place without drastically disturbing development.
Option B: Each service has its own Dockerfile (which doesn’t use Ansible), publishes its own official Docker image (which can be used for either prod or dev), and perhaps has its own docker-compose.yaml for just the services it needs. The current devstack repo gradually switches to use these images, and continues to be used for testing and development tasks that span multiple services.
Provisioning databases for services
This is something that devstack currently does for us.
We can move to the Test Data OEP instead (or some iteration of it)
Uses fixtures and named data sets consistently across services.
Publishing/Deploying changes: Is it OK to make changes to repos and not publish them?
Case 1: “I want to deploy ecommerce, but there are changes here I don’t understand”
Typically, people do deploy immediately per e-commerce’s policy.
Case 2: Changes in libraries that don’t get updated until the next
make upgrade.Currently, we don’t have consistent guidance around this.
Sometimes, people deploy one IDA/MFE, but don’t upgrade all dependents.
For frontends, we automatically merge updated MFEs, but don’t automatically deploy them to prod. They are deployed to stage.
For backends, we automatically run ‘make upgrade’, but don’t automatically merge or deploy.