Using Feature Flags on edx-platform

Rules of thumb

1. Use Waffle to add flexible and temporary feature flags used to roll-out a feature.
   
2. For more permanent feature flags, prefer model-based configuration using config models.
   1. Note that the same config model is used for edx.org and white label sites.  If you need a white label aware setting, then don't use a config model.
3. When removing a model-based config field, remove use of it from the code before removing the field with a migration.

How feature flags are currently used (simple case):

• As a global boolean, defined in (lms|cms)/envs/common.py
  • some hacks put this value in other places too
  • ex: (lms/cms)/envs/bok_choy.py for ENABLE_TEAMS
• When the flag is needed to “gate” access to the feature, we just ask the global boolean to decide.
  • ex: https://github.com/edx/edx-platform/blob/master/lms/djangoapps/teams/__init__.py
• This is specifically for edx-platform, and may not be comprehensive.
• See /wiki/spaces/LEARNER/pages/26183719 for an example of how ECOM handles this problem

Why are feature flags used?

• In the general software development case:
  • To gradually include a feature in releases, without exposing unfinished work to users
• Additionally, for edX specifically:
  • To allow organizations running their own edX instances to selectively enable features as they desire

What alternatives are there?

• To solve the problem of gradually releasing an unfinished feature, do not be afraid to use a temporary Waffle flag.  However, make sure you have a follow-up story to clean it up.
• To solve the problem of allowing organizations to pick-and-choose functionality for their instances of the platform:
  • Think twice about whether or not the option is necessary.  There is a large cost to supporting alternatives.  Although it might seem simple to just let people have a flag and decide for themselves, it would be best to avoid options where we can.

Feature flag best practices:

• If using a flag to have a staged rollout of a feature:
  • Use Waffle.
  • Work to remove the flag ASAP after full rollout.  Track this with a JIRA ticket.
  • Be sure that all entry points are covered by a flag check.
  • Run tests in 2 ways:
    • With all flags set as expected to be in the next release
    • With all flags set to their eventual end state
• If using a flag to enable selective functionality:
  • Minimize dependence on code that is covered by the feature flag. Modularity is king.
  • Make sure all entry points are masked by a feature flag check
  • Run tests both with and without the flag

Using Waffle

Staged Rollout

You can use Waffle and the edx-platform Waffle Utils to have a staged rollout of a feature using a temporary flag. Typically, the sooner you can reach completion, the better.  Not all stages or capabilities are required.

Definitions:

• CourseWaffleFlag: An edx-platform class that supports Course Overrides of Waffle Flags (in addition to caching and name-spacing). Use this for all waffle flags inside a course. (See Waffle Utils.)
• Waffle (Flag): Refers to the out of the box Waffle capabilities.

• WaffleFlag: An edx-platform class that supports caching and name-spacing.  Use this for all waffle flags outside a course. (See Waffle Utils.)

• Waffle Utils: A set of Waffle related utilities, including CourseWaffleFlag and WaffleFlag.

Open edX releases*: The aim is for these waffle flags to be short-lived.  However, due to timing or complicated rollouts, you may need to consider how to handle the flag across different Open edX releases as described in the table above.

Testing Waffle Flags

Python Unit Tests:

• For Waffle, see the waffle docs for unit testing.  
• In edx-platform, if you are using the WaffleFlag or CourseWaffleFlag, you can instead use the override_waffle_flag decorator implemented in this testutils.py file.

Bok-choy Testing:

• Bok-choy tests cannot use the decorator because the server is separate from the test code.
• To override in the URL, see the External Test Suites section of the Waffle documentation. Read the following important details as well.
  • In order to override a flag, it must first exist in the database.
    • In edx-platform, you can temporarily create a record in common/test/db_fixtures/waffle_flags.json that will be loaded directly into mysql for bok-choy tests only.  Note that you can default the flag on or off depending on your needs.
    • Other teams temporarily create a migration which will create the flag in all environments, including Production. 
  • Here is some example code reloading a page with a waffle flag set to a different value.
  • Note: In edx-platform, the WAFFLE_OVERRIDE setting is already taken care of in bok-choy to enable this type of URL override.

Current edX Feature Flag usage, a step-by-step guide:

• For a long-lived feature flag, make super sure that the team wants to use a feature flag. Ask many people for their opinions, and explore other options first if they’re available.
• Add your feature flag to the doc page edX Feature Flags, along with description, notes, contact, etc. If prior discussions about the flag are on a JIRA ticket, link that in as well!
• Modify lms/envs/common.py (or cms/envs/common.py) to include your feature flag (in ALL_CAPS), along with its default value.
• At the points where you’ll be “gating” access, get the boolean value like this: settings.FEATURES.get('ENABLE_TEAMS', False). This will check for the feature flag in common.py and use its value if present, or default to false.

Reference: http://martinfowler.com/bliki/FeatureToggle.html