A case in support for distributed configuration (feat. django 1.11)

The common theme here is that during the django 1.11 upgrade, a not-insignificant amount of time was spent dealing with static/baked configuration.  Distributed configuration would have been nice...

...Every time we needed to bake separate AMIs for each environment

The migration bug forced us to create 1.10 AMIs to migrate databases causing us to need to bake 3 additional 1.10 AMIs during the deployment. At the time, Alton was being buggy so we ended up needing to repeatedly retry baking AMIs for about 5 hours until we got all the AMIs we needed.  Luckily we didn't need to rollback stage and rebuild everything over again.

...Every time we needed to change configuration

During loadtesting for the migration, we repeatedly swapped out the MySQL DB to have a fresh instance before each test. Each time this required pushing edx-internal, baking an AMI, and doing an asgard deployment. Some time was spent following this process.

Also, the initial banner implementation utilized configuration to hold the banner text. What we didn't seem to realize was that we'd want to change the banner wording frequently and quickly. Configuration takes two days at best to roll out to prod, so we ended up abandoning that banner implementation.