Something just happened on my edx-platform PR:
The spigot was off and I pushed a new commit
Tests on the previous commit finished and triggered the notifier
The notifier commented that all tests have finished
We should add some logic to prevent this. We should probably check if the commit from the build that triggered the notifier is the tip of the branch, and if it isnt don't have it notify anything.