Toggles and Settings Documentation

Overview

The following toggles and settings tooling potential tasks around toggles and settings tooling, was generated while working on [BD-21] Toggles/Settings Documentation, Toggles and Settings Doc-a-thon 2021, and earlier work from the Arch-BOM team.

Next Steps

• Write announcement for value created by BD-21.

• Cost of Delay / Prioritization of future work, when we decide to continue.

• Potential for surveys or interviews as input to prioritization.

• Discuss continuation of blended for this work.

Potential Tasks

Linting

• Need to add deprecation ticket to a separate annotation for convenient filtering out during future reviews

• (HIGH) Add annotation for toggle_target_value (get name reviewed) to document whether to keep the True or False case once removed.

  • Required for temporary toggles. Is it useful to have this in the code, or is some of this DEPR metadata better kept in the DEPR ticket alone?

• (HIGH) Add annotation for toggle_depr_ticket to make it simpler to know if a toggle has an associated DEPR ticket.

  • Required for temporary toggles? Would it be better to create the DEPR ticket when creating the toggle?

• (HIGH) Ensure undefined annotations are not allowed.

  • For example, if toggle_warnings were used instead of toggle_warning, linting would complain.

  • Without this, the annotation would be visible in code, but it wouldn’t make it to readthedocs and you wouldn’t know it without checking.

  • Removing the unnecessary ability to add comments in-between annotations would simplify the regex update required for this.

  • Note: I think this may be causing us not to catch other linting issues with bad indents.

• (MEDIUM) Possible bug where required fields for temporary use case might be missed when multiple use cases are listed.

  • Note: This is not high priority, because we should prioritize switch to new toggle_life_expectancy annotation, which will make this go away.

• (MEDIUM) Ensure optional annotations are not set to None.

  • Applies to specific annotations where None is not a valid value (e.g. toggle_warning).

  • See example of where None is copied often and leads to a lot of extra code and useless docs.

• [RG]* (HIGH) Ensure a properly formatted date for toggle_target_removal_date annotation.

  • Examples with toggle_target_removal_date of None, even though it is required.

    • Many of the examples also added unnecessary text to toggle_warnings that toggle_target_removal_date was not set.

      • Possibly authors documented during docathon, and were afraid to set a date.

      • How-to was updated with proposal for how to choose a date. Linting failure help text could point to these docs.

• Check toggle_implementation annotation matches actual implementation class used when possible.

  • Note: examples of manually corrected errors.

• No linting exists to ensure settings are annotated.

  • Needs brainstorming if we think this would be important.

• No linting exists to ensure ConfigModels with booleans are annotated.

Improvements to published docs and reporting

• (HIGH) Add additional existing annotations (e.g. toggle_implementation) to published docs

  • See https://discuss.openedx.org/t/which-toggle-annotations-should-be-published/4690

  • AC:

    • Create ADR

    • Update sphinx plugin to add new annotations.

• (HIGH) Include annotations from dependencies in the automated docs:

  • Sphinx docs for toggles

  • Sphinx docs for settings (see open questions)

  • Toggle state report jobs (on-hold until we see this used?)

• (HIGH) Complete implementation of toggle state report outside of edx-platform

  • See how-to report on new IDA.

  • Apply to credentials, which was already mocked.

    • Replace credentials mock toggle state endpoint with a real endpoint.

    • Update the rest of the config according to the how-to.

  • Update the how-to to provide a simpler example than edx-platform (which overrides the report).

  • Add to cookiecutter-django-ida in https://github.com/edx/edx-cookiecutters/?

• Include more annotated settings in RTD documentation.

  • Context:

    • For settings toggles we currently only capture annotations if they are in lms/envs/common.py or cms/envs/common.py

    • We have settings annotations where the default and annotation live outside of those two files within edx-platform. See an example annotation.

    • We will have annotations where the default live in an external library (edx-when, or other edx owned libraries)

    • See some discussion and disagreements in Slack in #external-openedx-toggles-and-settings.

  • Potential Tickets

    • Discovery: How can we capture settings toggles not documented in lms/envs/common.py or cms/envs/common.py but are still in the edx-platform repo

      • AC:

        • Either a POC that captures settings in other places or an ADR to guide us on where these docs should live.

    • Discovery: How do we collect annotations from libraries installed in edx-platform?

• Allow for formatting in annotations, rather than having everything be plain-text.

  • Consider allowing arbitrary formatted rST. 

• Remove pylint disable pragmas from annotation output.

  • All, or just line-too-long?

  • See examples (including amnesty).

• RTD developer documentation single book for edx-platform

  • The edx-platform toggles and settings documentation was published as a separate book as a starting point, because the rest of the edx-platform docs weren’t published.

  • Would be nice to fix before too many references to docs that might be moving.

• CI testing for docs

  • edx-platform doesn’t test make technical-docs (current toggles and settings docs)

  • Spotty support for testing make docs for other repos.

• Reporting on suspicious toggle state data

  • Examples:

    • Bad data like waffle names with leading or trailing spaces in the database.

    • Potentially orphaned data (data with no annotation and no toggle definition)

  • Toggle State Report may be simplest location, using a new column

Update Toggle and Setting Annotation Definition

• [RG]* (HIGH) Implement ADR 0005-toggle-life-expectancy.rst.

  • See ADR Consequences for details.

Toggle Enhancements

• (HIGH) Enhance BaseToggle with additional methods.

  • Add is_disabled (not is_enabled), is_on (is_enabled) and is_off (not is_enabled)

    • Question: Would is_true() and is_false() be better than is_on/is_off?

      • [Robert Raposa] I like is_true/is_false better. Someone else should second my opinion and we can update this proposal.

  • Makes a variety of conditions more readable when ENABLE_ or DISABLE_ are in the toggle name.

  • For example:

    • not DISABLE_SOME_FEATURE.is_enabled() # BEFORE; brain twister

    • DISABLE_SOME_FEATURE.is_off() # AFTER

      • Question: Is this simpler to understand?:

        • DISABLE_SOME_FEATURE.is_false() # AFTER

  • This problem was discussed on Slack.

• Replace or improve ExperimentWaffleFlag

  • See Learning MFE toggle feedback