If you are running an experiment and want to bucket users into different experiences and monitor metrics based on that, there are some utility classes to help.

ExperimentWaffleFlag

ExperimentWaffleFlag is a drop-in replacement for a WaffleFlag or CourseWaffleFlag, that adds bucketing support and some other experiment niceties.

Using it in code

Definition

from lms.djangoapps.experiments.flags import ExperimentWaffleFlag
from openedx.core.djangoapps.waffle_utils import WaffleFlagNamespace

EXAMPLE_FLAG_NAMESPACE = WaffleFlagNamespace(name='example_namespace')

# Convention is to use a ticket number for experiments, but this can be any string
EXAMPLE_FLAG = ExperimentWaffleFlag(EXAMPLE_FLAG_NAMESPACE, 'JIRA-123')

This should look familiar. It’s the same pattern that edx-platform uses for waffle flags.

There are two new optional experiment-specific kwargs, though:

num_buckets: defaults to 2, specify how many buckets you want users to be broken into.
experiment_id: defaults to None, specify a unique int for your experiment on your installation of Open edX – this is only necessary to define if you want to use the enrollment_start feature below.
use_course_aware_bucketing: defaults to True. If True, then a given user can hash into different buckets for different courses. If False, then any given user will hash to the same default bucket across all course-runs.

Choosing a unique experiment ID

Again, this is only necessary if you use some of the more advanced features like enrollment_start. But if you want to ensure you are picking an experiment ID that is not taken yet, run the following command in mysql:

select distinct experiment_id from experiments_experimentdata
union
select distinct experiment_id from experiments_experimentkeyvalue
order by experiment_id;

That will tell you all the currently-in-use experiment IDs.

Usage

bucket = EXAMPLE_FLAG.get_bucket(course_key)
if bucket == 1:
    # variant 1 path
elif bucket == 2:
    # variant 2 path
else:
    # control path

Always treat the zero bucket as the control code path. If ExperimentWaffleFlag decides the user is not in the experiment at all (let alone bucketed), it will return 0.

You can also use the more familiar is_enabled(course_key=None) method, which simply returns True if the bucket is not zero. Which is particularly helpful for two-bucket experiments.

If you pass the optional course_key to either get_bucket or is_enabled, then:

Any applicable WaffleFlagCourseOverrideModels you create on the experiment flag or sub-flags will take effect, and
An analytics event will be emitted per-course-per-session (rather than just per-session), and
If use_course_aware_bucketing is True, then you will potentially get a different bucket per course for the same user.

Conversely, if you omit the course_key argument, none of the above will be true.

When to Check

In general, check whether the flag is enabled as late as possible. This is purely for analytics reasons, because we’ll send a segment event once the flag is checked (if a user is actually bucketed). So, for example:

That code will always send a segment event, for both instructor paced and self paced courses, even when your experiment is only interested in self paced courses. You can always do a filter on your analytics to cut them out. But it’s easier if you just reorder it:

You’ll only get segment events for self paced courses and your analytics are easier to slice.

Note that we send a segment event only if you are in the experiment. If the first main waffle flag isn’t enabled, we don’t send. Or if the enrollment didn’t meet our deadline. We do still send the segment event if you were forced into a bucket. (See below for what those different paths mean.)

Django configuration

You’ll set up one waffle flag to gate access to the experiment at all. If the user does not pass this first waffle flag check, no analytics event will be sent and they will get the default control experience.

You can also optionally set up sub-waffle-flags that will whitelist certain users, groups, or courses into a specific bucket (once they do get into the experiment). Commonly, you might want to whitelist staff users into the new code flow rather than into the control bucket.

And lastly, you can optionally set up a threshold date for course enrollments. What that means is that if you pass a course key to get_bucket or is_enabled above, users will only actually be bucketed and analyzed if their course enrollment started after your threshold date. This is a way to avoid suddenly changing the course experience for users in the middle of an enrollment.

Permissions

Make sure that you or someone on your team has the right permissions in Django to edit waffle flags and experiment data.

Main Waffle Flag

In the example above, your main waffle flag name will be example_namespace.example_flag. So you’ll want to make that flag in Django admin:

Click Add and then you’ll see a screen like the following. Enter the name and set Everyone to No to begin with. You can change that later, but let’s start with the experiment disabled.

Then click Save at the bottom.

Forcing a particular bucket

Now, a common desire is to bucket all staff or even all users into the new experience. Add another flag like you did above, but with a new name (you’ll add a .1 to the end of your waffle flag name – this means this is a bucket override whitelist for bucket 1).

Note that you’ll still need to open up access in the above main flag in order for staff to even attempt to get bucketed. The main waffle flag being turned on for everyone corresponds to everyone being bucketed, not everyone receiving the treatment. In order to force everyone into treatment, you will need to have the main waffle flag turned on for everyone as well as the example_namespace.example_flag.1(or whatever bucket treatment you want to force) set to everyone.

You could also make a .0 waffle flag to force, say, a course into the control experience, as a way to opt them out. Or for testing, you could whitelist your own user into the control group. If you try to force-bucket the same user into multiple different buckets, the lower bucket number will take priority.

Enrollment Cutoff

If you are bucketing per-course and want to only consider courses after your experiment actually begins, you can add a new experiment key value for your experiment.

Add a new Experiment Key-Value Pair and set the field enrollment_start for the correct experiment id (set when the experiment waffle was defined above, if you pass the waffle experiment_id=xx).

The value is parsed by dateutil.parser.parse.

With this set, you will only see analytics and bucketing happen for course enrollments that start after this date.

Analytics

For everyone that is bucketed, we’ll send a segment track event once per session (or once per course per session, if you are bucketing per session).

Here is the information that is sent:

Architecture and Engineering

Bucketing users for an experiment