Rules of thumb (going forward)

1. Feature toggles. Use Waffle via waffle_utils for boolean flags to toggle or rollout a feature.
   
   1. Use WaffleSwitch for a simple toggle that can be dynamically changed while the server is running. 
   2. Use WaffleFlag for a staged rollout of the feature when you want to increase gradually the percentage of the population.
   3. Use CourseWaffleFlag for supporting course-level opt-in (during staged rollout) or opt-out (after full rollout) of the feature.
2. Feature settings. Use Configuration models for more complex settings and non-boolean fields to configure a feature.
   
   1. Configuration models are based on Django models.
   2. To remove a model-based config field, use a 2-phase deployment strategy where you remove use of it from the code before removing the field with a migration.  See Migration Don'ts.
3. System settings. Use Django settings for open edX instance/deployment wide configurations that do not change while the server is running.
   1. Add the setting's default value to (lms|cms)/envs/common.py (example).
   2. Override the value in other relevant (lms|cms)/envs/* files (example).
   3. Make sure to update aws.py so the setting can be configurable via JSON files (example).
      1. If the setting is security-sensitive, read its value from AUTH_TOKENS
      2. Else, read the value from ENV_TOKENS
   4. Add your setting to edX Feature Flags, along with description, notes, contact, etc. If prior discussions about the setting are on a JIRA ticket, link that in as well!

Why are

...

• To gradually include a feature in releases, without exposing unfinished work to users

...

Feature Toggles used?

Case 1: Decoupling release from deployment

When teams introduce new changes or new features in the platform and they want to:

• have control of when that feature is enabled so they can monitor it in stage/production at their own time, independent of the release cycle.
• have the ability to disable it in case things go unexpectedly in stage/production.
• be able to submit incremental changes without turning on the entire feature and preventing exposure to unfinished work.
  • Note: This last reason should be used very sparingly! Instead of having a grand toggle to unlock many changes at once, consider breaking up your feature into iterative verticals that can be released a bite at a time.  See Release Toggles Are The Last Thing You Should Do.

Best Practices

• Use WaffleSwitch.
• Tests. Run all tests with the expected future of the waffle switch being enabled.  Have at least a few tests to verify the functionality when the waffle switch is disabled.
• Removal. Once the feature is field-tested in production, the feature toggle should be removed from both (a) all places in the code and (b) from the production/stage configurations.

Case 2: Staged Rollout

When teams introduce new changes or new features in the platform and they want to control the population affected by the change or exposed to the new feature.  Here are some cases when this occurs:

• The team is concerned about possible performance/scalability degradation or functional correctness issues and they want to monitor the impact on stage/production by gradually increasing the load.
  • Use WaffleFlag.
• The team wants to Beta-test the feature with a few brave course teams that have opted-in prior to making the feature available to all courses.
  • Use CourseWaffleFlag.

Best Practices

• Tests. Run tests with both variations, on and off, to verify the functionality continues to work during the rollout transition phase.
• Removal. Once the feature is fully adopted in production, the feature toggle should be removed from both (a) all places in the code and (b) from the production/stage configurations.

Case 3: Configuration option for Open edX

When teams introduce new changes or new features that are not expected to be adopted by all open edX instances, either immediately or for a while.

• Use WaffleSwitch.
• Use Django settings.

What alternatives are there?

...

• Bok-choy tests cannot use the decorator because the server is separate from the test code.
• To override in the URL, see the External Test Suites section of the Waffle documentation. Read the following important details as well.
  • In order to override a flag, it must first exist in the database.
    • In edx-platform, you can temporarily create a record in common/test/db_fixtures/waffle_flags.json that will be loaded directly into mysql for bok-choy tests only.  Note that you can default the flag on or off depending on your needs.
    • Other teams temporarily create a migration which will create the flag in all environments, including Production. 
  • Here is some example code reloading a page with a waffle flag set to a different value.
  • Note: In edx-platform, the WAFFLE_OVERRIDE setting is already taken care of in bok-choy to enable this type of URL override.

Reference: http://martinfowler.com/bliki/FeatureToggle.html