Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

There are two new optional experiment-specific kwargs, though:

  • num_buckets: defaults to 2, specify how many buckets you want users to be broken into.

  • experiment_id: defaults to None, specify a unique int for your experiment on your installation of Open edX – this is only necessary to define if you want to use the enrollment_start feature below.

  • use_course_aware_bucketing: defaults to True. If True, then a given user can hash into different buckets for different courses. If False, then any given user will hash to the same default bucket across all course-runs.

Choosing a unique experiment ID

Again, this is only necessary if you use some of the more advanced features like enrollment_start. But if you want to ensure you are picking an experiment ID that is not taken yet, run the following command in mysql:

Code Block
languagesql
select distinct experiment_id from experiments_experimentdata
union
select distinct experiment_id from experiments_experimentkeyvalue
order by experiment_id;

That will tell you all the currently-in-use experiment IDs.

Usage

Code Block
languagepy
bucket = EXAMPLE_FLAG.get_bucket(course_key)
if bucket == 1:
    # variant 1 path
elif bucket == 2:
    # variant 2 path
else:
    # control path

...

If you pass the optional course_key to either get_bucket or is_enabled, you will potentially get a different bucket per course for the same user. And will emit an analytics event then:

  1. Any applicable WaffleFlagCourseOverrideModels you create on the experiment flag or sub-flags will take effect, and

  2. An analytics event will be emitted per-course-per-session (rather than just per-session), and

  3. If use_course_aware_bucketing is True, then you will potentially get a different bucket per course for the same user.

Conversely, if you omit the course_key argument, none of the above will be true.

When to Check

In general, check whether the flag is enabled as late as possible. This is purely for analytics reasons, because we’ll send a segment event once the flag is checked (if a user is actually bucketed). So, for example:

...

And lastly, you can optionally set up a threshold date for course enrollments. What that means is that if you are slicing users into buckets based off which course they are in (i.e. passing pass a course key to get_bucket or is_enabled above), they users will only actually be bucketed and analyzed if their course enrollment started after your threshold date. This is a way to avoid suddenly changing the course experience for users in the middle of an enrollment.

...

Then click Save at the bottom.

...

Forcing a particular bucket

Now, a common desire is to bucket all staff or even all users into the new experience. Add another flag like you did above, but with a new name (you’ll add a .1 to the end of your waffle flag name – this means this is a bucket override whitelist for bucket 1).

Note that you’ll still need to open up access in the above main flag in order for staff to even attempt to get bucketed. The main waffle flag being turned on for everyone corresponds to everyone being bucketed, not everyone receiving the treatment. In order to force everyone into treatment, you will need to have the main waffle flag turned on for everyone as well as the example_namespace.example_flag.1(or whatever bucket treatment you want to force) set to everyone.

...

You could also make a .0 waffle flag to force, say, a course into the control experience, as a way to opt them out. Or for testing, you could whitelist your own user into the control group. If you try to force-bucket the same user into multiple different buckets, the lower bucket number will take priority.

...