Page Comparison

...

Step 1: File a flaky bug ticket in Github:

1. Check if someone is already following this flaky test process at the same time:

   1. Search for an existing Github ticket about this issue using the flaky-test label.

      1. Github search for flaky tests.

   2. Search on Github to see if the test was already deleted.

2. Create (or update) a Github ticket with the following informationissue:

   1. Create a ticket new issue in the corresponding repo .

      1. [idea] Consider creating an openedx Issue template for flaky tests.

   2. Title: Something like "<testclass testcase> fails intermittently"

   3. Labels: At a minimum flaky-test label (this is needed for searches)

   4. Description: Include the following:

   5. A link to a failed build in GitHub Actions.

   6. A link to a passing run for the same commit.

   7. The Error Message and Stacktrace for a failed test.

      1. You should include this text in the Github ticket so it is searchable to others.

      2. Wrap each of these using code-formatting in the text.

   Putting it all together you'll be entering something like this

   containing the test code

   8. Title: [Flaky test] `path/to/test.py::class::method`

   9. Labels: flaky-test

   10. Description: Paste the following template:

       Code Block
       This test fails intermittently and will

       has

       be

       been

       deleted

       removed

       according

       from

       to the

       codebase.

       Flaky Test

       For

       Process

       triaging

       in

       this bug: - Until proven one way or the other, it could be either the code that is flaky or the test that is flaky. - The test has been removed from the codebase, and thus the functionality is no longer covered by CI tests. . - Evaluate whether or not it is or can be covered elsewhere and thus the test was unnecessary, or if there is now risk that a bug in the code could escape and thus the test

       <https://github.com/openedx/edx-platform/pull/___PENDING___> See docs on [how to address flaky tests](https://openedx.atlassian.net/wiki/spaces/AC/pages/4306337795/Flaky+Test+Process#Handling-a-flaky-test-Github-issue) for why this should be fixed and

       re-enabled.

       how to

       TestClass:test_case failed in [this build on GitHub |

       go about it. - Failing CI run: https://...

       ]and

       -

       passed

       Subsequent

       in

       Passing

       the

       CI

       [subsequent build | {link}]

       run: https://...

       Error

       Failure

       Message

       output: ```

       Whatever the error message was for the testcase. ``` Stacktrace ``` The stacktrace from the test result. ```

       ...test failure... ```

       …and fill out these placeholders:

       • URL of failed test run in GitHub Actions.

       • URL of passing run for the same commit.

       • Output from the failing test, including error message, stack trace, and anything else needed to recognize the same failure again. Build logs aren’t kept forever, and this also makes the text searchable.

3. Now we can remove the test!

Step 2: Make a PR deleting the flaky test(s):

1. Delete the flaky test(s) in a single commit in a new PR (so that it is easily cherry-picked by others.)

   1. Include In the Github link in your commit message, link to the GitHub issue you just created, e.g.:

      Code Block
      test: Delete flaky test TEST::METHOD Deleted according to flaky test process: https://openedx.atlassian.net/wiki/spaces/AC/pages/4306337795/Flaky+Test+Process Flaky test ticket: - https://github.com/openedx/edx-platform/issues/11111_____

   2. Note that temp: could be used in place of test: for the commit type if you assume it will be fixed and restored.

   3. If you have any helpful thoughts to add, feel free to comment on the PR.

2. Place a Update your Github issue to link to the PR back in your Github issue:

   Code BlockThe test was [deleted in this PR|https://github.com/openedx/edx-platform/pull/99999]

   (replace the ___PENDING___ placeholder in the first line).

3. Get a passing build and at least 1 review before merging to master.

   • Note: It would be best if the reviewer has some idea about the relative importance of the test and could help prioritize the ticket to fix the test.

Handling a flaky test Github issue

...

• Is the test necessary? For example, is it a cypress test that makes more sense in a lower part of the pyramid (e.g. python or javascript unit tests)?

• Is the test covering a longer flow where only one part is flaky? Maybe you can reduce the amount of the test that needs to be deleted.

• Is it only the test that is flaky, or is the code under test itself non-deterministic?

• If the test is of debatable usefulness, consider time-boxing the effort to fix and closing the ticket if it takes too long.