Rules of thumb (going forward)

1. Feature toggles. Use Waffle via waffle_utils for boolean flags to toggle or rollout a feature.
   
   1. Use WaffleSwitch for a simple toggle that can be dynamically changed while the server is running (example). 
      Toggle via Django Admin at /admin/waffle/switch/
      
      
   2. Use WaffleFlag for a staged rollout of the feature when you want to increase gradually the percentage of the population (example).
      Toggle via Django Admin at /admin/waffle/flag/
      
   3. Use CourseWaffleFlag for supporting course-level opt-in (during staged rollout) or opt-out (after full rollout) of the feature (example).
      Toggle via Django Admin at /admin/waffle_utils/waffleflagcourseoverridemodel/
      
      
2. Feature settings. Use Configuration models for more complex settings and non-boolean fields to configure a feature.
   
   1. Models.
      1. Configuration models are based on Django models.
      2. They are immutable tables where each change introduces a new row in the table.
      3. Declare your feature-specific model in your models.py file, inheriting from ConfigurationModel (example).
      4. Declare a django admin interface to your model in your admin.py (example).
   2. Site-aware. Make your config models site-aware by including a site field in the model (example).
   3. Removal. To remove a Configuration field/model, use a 2-phase deployment strategy where you remove use of it from the code before removing the field/model with a migration. See Migration Dos and Don'ts.
      
      
3. System settings. Use Django settings for open edX instance/deployment wide configurations that do not change while the service is running. For a complete walkthrough of this method see this guide: How to add an AWS configuration variable to edX platform by example(aws.py)
   
   1. Changes. Note that any changes to these settings would require a re-deployment of the service.  To be able to change these settings dynamically, see '#2 Feature settings above'.
   2. Configuration files.
      1. Add the setting's default value to (lms|cms)/envs/common.py (example).
      2. Override the value in other relevant (lms|cms)/envs/* files (example).
      3. Make sure to update aws.py so the setting can be configurable via JSON files (example).
         1. If the setting is security-sensitive, read its value from AUTH_TOKENS
         2. Else, read the value from ENV_TOKENS
   3. Documentation. Add your setting to edX Feature Flags, along with description, notes, contact, etc. If prior discussions about the setting are on a JIRA ticket, link that in as well!

...

Best Practices

• Use WaffleSwitch.
• Gate entry points. Don't try to protect every code path with a toggle. Focus on checking for the toggle on just entry points that would lead users to the change/feature.
• Tests. Run all tests with the expected future of the waffle switch being enabled.  Have at least a few tests to verify the functionality when the waffle switch is disabled.
• Removal. Once the feature is field-tested in production, the feature toggle should be removed from both (a) all places in the code and (b) from the production/stage configurations. Plan ahead by creating a clean up story ahead of time.

Case 2: Staged Rollout

When teams introduce new changes or new features in the platform and they want to control the population affected by the change or exposed to the new feature.  Here are some cases when this occurs:

• The team is concerned about possible performance/scalability degradation or functional correctness issues and they want to monitor the impact on stage/production by gradually increasing the load.
  • Use WaffleFlag.
• The team wants to Beta-test the feature in Production with individuals on the development team.
  • Use WaffleFlag.
• The team wants to Beta-test the feature in Production with a few brave course teams that have opted-in prior to making the feature available to all courses.
  • Use CourseWaffleFlag.

Best Practices

• Gate entry points. Don't try to protect every code path with a toggle. Focus on checking for the toggle on just entry points that would lead users to the change/feature.
• Tests. Run tests with both variations, on and off, to verify the functionality continues to work during the rollout transition phase.
• Removal. Once the feature is fully adopted in production, the feature toggle should be removed from both (a) all places in the code and (b) from the production/stage configurations. Plan ahead by creating a clean up story ahead of time.

...

Audit Trail of Changes

Waffle Changes

The django_admin_log table keeps a history and audit trail of all WaffleFlag and WaffleSwitch made via Django Admin.

WaffleSwitch changes can be queried via

select * from django_admin_log where content_type_id=104 order by action_time desc; 

WaffleFlag changes can be queried via

select * from django_admin_log where content_type_id=103 order by action_time desc; 

CourseWaffleFlag changes can be queried via

select * from waffle_utils_waffleflagcourseoverridemodel where waffle_flag = <YOUR_NAMESPACED_WAFFLE_FLAG_NAME>; 

Configuration Model Changes

As mentioned above, Configuration models are append only and so changes to the settings via Django Admin will be maintained in its own table, with a new row added for each change.

Coding Tips

Testing with Waffle

...

• Bokchoy tests cannot use the decorator because the server is separate from the test code.
• To override in the URL, see the External Test Suites section of the Waffle documentation. Read the following important details as well.
  • In order to override a flag, it must first exist in the database.
    • In edx-platform, you can temporarily create a record in common/test/db_fixtures/waffle_flags.json that will be loaded directly into mysql for bok-choy tests only.  Note that you can default the flag on or off depending on your needs.
    • Other teams temporarily create a migration which will create the flag in all environments, including Production. 
  • Here is some example code reloading a page with a waffle flag set to a different value.
  • Note: In edx-platform, the WAFFLE_OVERRIDE setting is already taken care of in bokchoy to enable this type of URL override.

Pending Infrastructural Work (up for grabs)

...

• waffle_utils: A django app (currently within edx-platform) that provides a set of Waffle related utilities, including CourseWaffleFlag and WaffleFlag.  It is also an explicit abstraction layer for toggling of features.  Use the interfaces provided in this library instead of Waffle directly.

• WaffleFlag: A class within waffle_utils that supports request-caching and name-spacing.  Use this for all waffle flags outside a course.

• CourseWaffleFlag: A class that supports Course-level overrides of WaffleFlags.