Page Comparison

...

• A write-up on which github.txt Python packages could move to a package cache, and which could be handled via other means is written up in

  Jira Legacy
  server System JIRA serverId 13fd1930-5608-3aac-a5dd-21b934d3a4b4 key PLAT-1907

• Notes on setting up Artifactory and some timing / test run information is here
• Notes on setting up DevPI and timing results are here

Populating Caches

One thing we will need to sort out is how we want to get out problem packages into the cache. Presumably anything in PyPI should be able to cache normally. The steps to do that being:

• Clone the correct repository
• Check out the correct commit
• Build the package
• Uploads the built package to the cache

The two ways of dealing with this (off hand) are:

Push

Triggered (either by hand or CI) when a problematic package is updated, much like pushing to PyPI. This is better for a global cache, where updates are infrequent and can be carefully controlled, however several of the repositories in question are not under edX control so there may be some confusion around who is responsible for updating the cache, and when, in these cases. It does make for some extra work when a developer is bumping the version of one of these systems and wants to test locally.

Pull

Triggered when a system (Devstack build, Jenkins build, etc) cannot find the necessary version in the cache. If it has the necessary information (repo, commit, version, write permissions to the cache) it can populate a cache itself. This makes sense for a local cache, where there is less likelihood of a conflict between automated systems updating the cache, and where permissions can be more lax.

Prepopulate & Pull

For systems like Devstack, and perhaps Jenkins testing, it might make sense to build the cache right into the Docker container when other Devstack images are built. It would make for a larger download, but potentially save a lot of testing time. If a version is bumped in development the cache would still pull it through PyPI as normal, but it would also let devstack run tests offline out of the box in most situations (I think, not sure if any non-Python dependencies are pulled at test time).

Local Caches vs. Global Cache

Another big question is whether we want to have numerous small caches or few, maybe one, big shared cache. Here are some preliminary thoughts on those options:

Local

• Fastest, closest cache
• Allows or brings us closer to offline Devstack
• Smaller (or no) DevOps burden
• / Developers would need to understand / manage the local cache if they are bumping the version of a problematic package, but would have greater control
• If we wanted to use it for Jenkins builds / tests we might still need to stand up a shared instance
  • This brings with it the overhead of somehow having to keep it populated with up-to-date dependencies
• Somewhat slower build times for Devstack either when we build the cache image or the first time a user populates the local cache

Global

• Easier to maintain one cache, less chance of it falling out of sync on a pull
• Little-to-no special handling needed for Jenkins builds / tests
• / Developer burden is lighter, but updating problematic packages may require new permissions / processes, or at least some new automation
• Slower in general use due to having to hit the internet to download packages
• Still requires Devstacks to be online
• Heavier DevOps burden - this becomes another critical system that needs to be highly available and (depending on solution chosen) may involve new clustering setups and additional database servers to maintain

DevPI vs. Artifactory

Open Questions

• How will our choices here impact the OSS / partner communities? What can we do to make sure they can benefit from any gains we see here? How can we keep from complicating things for them?