...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

Rules of thumb (going forward)

1. Use Waffle to add flexible and temporary feature flags used to roll-out a feature.
   
2. For more permanent feature flags, prefer model-based configuration using config models.
   1. Note that the same config model is used for edx.org and white label sites.  If you need a white label aware setting, then don't use a config model.
3. When removing a model-based config field, Feature toggles. Use Waffle via waffle_utils for boolean flags to toggle or rollout a feature.
   
   1. Use WaffleSwitch for a simple toggle that can be dynamically changed while the server is running (example). 
      Toggle via Django Admin at /admin/waffle/switch/
      
      
   2. Use WaffleFlag for a staged rollout of the feature when you want to increase gradually the percentage of the population (example).
      Toggle via Django Admin at /admin/waffle/flag/
      
   3. Use CourseWaffleFlag for supporting course-level opt-in (during staged rollout) or opt-out (after full rollout) of the feature (example).
      Toggle via Django Admin at /admin/waffle_utils/waffleflagcourseoverridemodel/
      
      
4. Feature settings. Use Configuration models for more complex settings and non-boolean fields to configure a feature.
   
   1. Models.
      1. Configuration models are based on Django models.
      2. They are immutable tables where each change introduces a new row in the table.
      3. Declare your feature-specific model in your models.py file, inheriting from ConfigurationModel (example).
      4. Declare a django admin interface to your model in your admin.py (example).
   2. Site-aware. Make your config models site-aware by including a site field in the model (example).
   3. Removal. To remove a Configuration field/model, use a 2-phase deployment strategy where you remove use of it from the code before removing the field/model with a migration. See Everything About Database Migrations#Deploymentandbackward-compatiblemigrations.
      

How feature flags are currently used (simple case):

...

1. 1. 
      
2. System settings. Use Django settings for open edX instance/deployment wide configurations that do not change while the service is running. For a complete walkthrough of this method see this guide: How to add an AWS configuration variable to edX platform by example(production.py)
   
   1. Changes. Note that any changes to these settings would require a re-deployment of the service.  To be able to change these settings dynamically, see '#2 Feature settings above'.
   2. Configuration files.
      1. Add the setting's default value to (lms|cms)/envs/common.py

...

1. 1. 1. (example).
      2. Override the value in other

...

1. 1. 1. relevant (lms

...

1. 1. 1. |cms)/envs/

...

• ex: https://github.com/edx/edx-platform/blob/master/lms/djangoapps/teams/__init__.py

...

Why are feature flags used?

• In the general software development case:
  • To gradually include a feature in releases, without exposing unfinished work to users
• Additionally, for edX specifically:
  • To allow organizations running their own edX instances to selectively enable features as they desire

What alternatives are there?

• To solve the problem of gradually releasing an unfinished feature, do not be afraid to use a temporary Waffle flag.  However, make sure you have a follow-up story to clean it up.
• To solve the problem of allowing organizations to pick-and-choose functionality for their instances of the platform:
  • Think twice about whether or not the option is necessary.  There is a large cost to supporting alternatives.  Although it might seem simple to just let people have a flag and decide for themselves, it would be best to avoid options where we can.

Feature flag best practices:

• If using a flag to have a staged rollout of a feature:
  • Use Waffle.
  • Work to remove the flag ASAP after full rollout.  Track this with a JIRA ticket.
  • Be sure that all entry points are covered by a flag check.
  • Run tests in 2 ways:
    • With all flags set as expected to be in the next release
    • With all flags set to their eventual end state
• If using a flag to enable selective functionality:
  • Minimize dependence on code that is covered by the feature flag. Modularity is king.
  • Make sure all entry points are masked by a feature flag check
  • Run tests both with and without the flag

Using Waffle

Staged Rollout

You can use Waffle and the edx-platform Waffle Utils to have a staged rollout of a feature using a temporary flag. Typically, the sooner you can reach completion, the better.  Not all stages or capabilities are required.

Definitions:

• CourseWaffleFlag: An edx-platform class that supports Course Overrides of Waffle Flags (in addition to caching and name-spacing). Use this for all waffle flags inside a course. (See Waffle Utils.)
• Waffle (Flag): Refers to the out of the box Waffle capabilities.

• WaffleFlag: An edx-platform class that supports caching and name-spacing.  Use this for all waffle flags outside a course. (See Waffle Utils.)

• Waffle Utils: A set of Waffle related utilities, including CourseWaffleFlag and WaffleFlag.

...

Use a Waffle Flag to beta test with individual users in Production, both internal and partners. Implement with CourseWaffleFlag or WaffleFlag to enable future rollout capabilities.

...

1. 1. 1. * files (example).
      2. Make sure to update aws.py so the setting can be configurable via JSON files (example).
         1. If the setting is security-sensitive, read its value from AUTH_TOKENS
         2. Else, read the value from ENV_TOKENS
   2. Documentation. Add your setting to edX Feature Flags, along with description, notes, contact, etc. If prior discussions about the setting are on a JIRA ticket, link that in as well!

Why are Feature Toggles used?

Case 1: Decoupling release from deployment

When teams introduce new changes or new features in the platform and they want to:

• have control of when that feature is enabled so they can monitor it in stage/production at their own time, independent of the release cycle.
• have the ability to disable it in case things go unexpectedly in stage/production.
• be able to submit incremental changes without turning on the entire feature and preventing exposure to unfinished work.
  • Note: This last reason should be used very sparingly! Instead of having a grand toggle to unlock many changes at once, consider breaking up your feature into iterative verticals that can be released a bite at a time.  See Release Toggles Are The Last Thing You Should Do.

Best Practices

• Use WaffleSwitch.
• Gate entry points. Don't try to protect every code path with a toggle. Focus on checking for the toggle on just entry points that would lead users to the change/feature.
• Tests. Run all tests with the expected future of the waffle switch being enabled.  Have at least a few tests to verify the functionality when the waffle switch is disabled.
• Removal. Once the feature is field-tested in production, the feature toggle should be removed from both (a) all places in the code and (b) from the production/stage configurations. Plan ahead by creating a clean up story ahead of time.

Case 2: Staged Rollout

When teams introduce new changes or new features in the platform and they want to control the population affected by the change or exposed to the new feature.  Here are some cases when this occurs:

• The team is concerned about possible performance/scalability degradation or functional correctness issues and they want to monitor the impact on stage/production by gradually increasing the load.
  • Use WaffleFlag.
• The team wants to Beta-test the feature in Production with individuals on the development team.
  • Use WaffleFlag.
• The team wants to Beta-test the feature in Production with a few brave course teams that have opted-in prior to making the feature available to all courses.
  • Use CourseWaffleFlag.

Best Practices

• Gate entry points. Don't try to protect every code path with a toggle. Focus on checking for the toggle on just entry points that would lead users to the change/feature.
• Tests. Run tests with both variations, on and off, to verify the functionality continues to work during the rollout transition phase.
• Removal. Once the feature is fully adopted in production, the feature toggle should be removed from both (a) all places in the code and (b) from the production/stage configurations. Plan ahead by creating a clean up story ahead of time.

Open edX releases*: The aim is for these waffle flags to be short-lived.  However, due to timing or complicated rollouts, you may need to consider how to handle the flag across different Open edX releases as described in the table above.

Testing Waffle Flags

Python Unit Tests:

...

Case 3: Configuration option for Open edX

When teams introduce new changes or new features that are not expected to be adopted by all open edX instances.  This case may co-exist with Case #1 or Case #2 above.  So use the guidelines from the above cases to decide which Waffle technology to use.

Best Practices

• Think twice about whether or not this option is necessary. There is a large cost to supporting long-term toggles. Although you can avoid hard decisions by just providing a toggle and letting others decide for themselves, code complexity is reduced with unnecessary alternative paths.
• Consider alternative design patterns such as plug-in architectures and other extensible mechanisms that don't require code-level toggles. 

Audit Trail of Changes

Waffle Changes

The django_admin_log table keeps a history and audit trail of all WaffleFlag and WaffleSwitch made via Django Admin.

WaffleSwitch changes can be queried via

select * from django_admin_log where content_type_id=104 order by action_time desc; 

WaffleFlag changes can be queried via

select * from django_admin_log where content_type_id=103 order by action_time desc; 

CourseWaffleFlag changes can be queried via

select * from waffle_utils_waffleflagcourseoverridemodel where waffle_flag = <YOUR_NAMESPACED_WAFFLE_FLAG_NAME>; 

Configuration Model Changes

As mentioned above, Configuration models are append only and so changes to the settings via Django Admin will be maintained in its own table, with a new row added for each change.

Coding Tips

Testing with Waffle

Python Unit Tests

• For WaffleFlag and CourseWaffleFlag, use the override_waffle_flag decorator implemented in this testutils.py file.

Bok-choy Testing:

...

Bokchoy Tests

• Bokchoy tests cannot use the decorator because the server is separate from the test code.
• To override in the URL, see the External Test Suites section of the Waffle documentation. Read the following important details as well.
  • In order to override a flag, it must first exist in the database.
    • In edx-platform, you can temporarily create a record in common/test/db_fixtures/waffle_flags.json that will be loaded directly into mysql for bok-choy tests only.  Note that you can default the flag on or off depending on your needs.
    • Other teams temporarily create a migration which will create the flag in all environments, including Production. 
  • Here is some example code reloading a page with a waffle flag set to a different value.
  • Note: In edx-platform, the WAFFLE_OVERRIDE setting is already taken care of in bok-choy bokchoy to enable this type of URL override.

Current edX Feature Flag usage, a step-by-step guide:

• For a long-lived feature flag, make super sure that the team wants to use a feature flag. Ask many people for their opinions, and explore other options first if they’re available.
• Add your feature flag to the doc page edX Feature Flags, along with description, notes, contact, etc. If prior discussions about the flag are on a JIRA ticket, link that in as well!
• Modify lms/envs/common.py (or cms/envs/common.py) to include your feature flag (in ALL_CAPS), along with its default value.
• At the points where you’ll be “gating” access, get the boolean value like this: settings.FEATURES.get('ENABLE_TEAMS', False). This will check for the feature flag in common.py and use its value if present, or default to false.

...

Pending Infrastructural Work (up for grabs)

1. Extract the waffle_utils djangoapp out of edx-platform into its own repo so it can be used by other IDAs.
2. Figure out how to enable a waffle switch/flag for ALL unit tests in the platform - without using a context manager.
3. Enhance waffle_utils to be site-aware so toggles can be overridden for White Label sites.
4. Implement a CourseWaffleSwitch equivalent of CourseWaffleFlag when the need arises.
5. Better tools for (1) duplicating settings from one system to another and (2) initializing settings in a system.

Reference

• http://martinfowler.com/bliki/FeatureToggle.html

Glossary

• waffle_utils: A django app (currently within edx-platform) that provides a set of Waffle related utilities, including CourseWaffleFlag and WaffleFlag.  It is also an explicit abstraction layer for toggling of features.  Use the interfaces provided in this library instead of Waffle directly.

• WaffleFlag: A class within waffle_utils that supports request-caching and name-spacing.  Use this for all waffle flags outside a course.

• CourseWaffleFlag: A class that supports Course-level overrides of WaffleFlags.