...
IV. What didn’t we do so well, and what could we have done to do better ❌
Testing
[Jenna]: How can we tighten the lead time between the code cutoff and having the testing sandboxes ready? For Redwood, there was about a week in between. How can product help in defining which features should be enabled, etc?
[Jenna]: Only having 3 weeks for testing ended up feeling tight. What what would it look like to try a 6 week testing period between code cut and release, especially if we can shorten the lead time on the sandboxes?
[Jenna]: Redwood was feature-packed. How can we scale back and still have a meaningful set of capabilities/features in Sumac, without over-committing?[Peter]: There was a lot uncertainty and confusion with the security issue, particularly around what and with whom details could be discussed. It would be good to have a documented and, ideally, more transparent process before this happens againChelsea]: We could potentially firm up exactly how the testing sandbox should be configured and make sure that’s communicated well with the team setting up the testing sandbox (Example: it was a miss on my part to make sure Aspects was enabled from the start).
[Peter]: Some of the regression tests are poorly defined and therefore difficult to test confidently. For the next release, we should try to define the legacy tests as well as the new tests were defined.
[Chelsea]: I had claimed/planned to do a test that only once I sat down to do the test did I realize how technical the tester needed to be to test the functionality. It made me wonder if our test sheet should have a designated label for tests that need to be done by an engineer or can be done by anyone.
[Chelsea]: We could potentially firm up exactly how the testing sandbox should be configured and make sure that’s communicated well with the team setting up the testing sandbox (Example: it was a miss on my part to make sure Aspects was enabled from the start).
[Feanil]: Even if it’s not officially supported, should we allow the community to provide backport fixes to older releases at the release managers discretion? I feel like people are making these on their local forks and we could just let them upstream them if they can be validated easily.
[Sarina] how can we “fast fail” in the monthly release meetings run by product (that is, “kick” something out of the release as soon as it’s slipping, and divert resources to getting the other things done)?
[Sarina] consider having release meetings every month, always looking forward to the next release’s cut. Starting meetings in August for an October cut seems a bit late.
Bug triage and fixing
[Maria] Although 3 weeks were enough to complete a high percentage of tests, it wasn’t enough time for bug triages to intervene and help maintainers solve high-priority issues. We currently have 9 high-priority issues on the board that haven’t been solved (see both Release Testing and Test failures).
[Maria] We need more maintainers' and community participation to solve issues reported on the board, even after the Redwood cut-off. How do we increase participation so the boards don’t overflow with unsolved issues from past releases? How can we more effectively draw maintainers' attention?
[Maria] We need better documentation to classify security issues and report them effectively. [Maria] We need a better way of organizing issues from other repositories that directly affect the release. This time, I created two boards: Release testing, which tracks issues created outside the BTR repo, and Testing failures, which tracks test failure reports. However, now I find having both views a bit confusing. Maybe this could be fixed by having more explicit names or having a single view for both and differentiating them using labels.
[Maria] As bug triagers, our main focus should be triaging and solving high-priority issues and blockers in a timely manner. This helps to avoid accumulating issues over time and avoids solving blockers in a rush if we don’t get maintainers' attention in time.
[Maria] We need to automatize getting issues labeled as “release testing” from other repos into the BTR repo for better visibility.
Security
[Peter]: There was a lot uncertainty and confusion with the security issue, particularly around what and with whom details could be discussed. It would be good to have a documented and, ideally, more transparent process before this happens again.
[Maria] We need better documentation to classify security issues and report them effectively.
Product
[Jenna]: Redwood was feature-packed. How can we scale back and still have a meaningful set of capabilities/features in Sumac, without over-committing?
[Sarina] how can we “fast fail” in the monthly release meetings run by product (that is, “kick” something out of the release as soon as it’s slipping, and divert resources to getting the other things done)?
[Sarina] consider having release meetings every month, always looking forward to the next release’s cut. Starting meetings in August for an October cut seems a bit late.
Release notes
[Maria] We need to automatize getting new settings/feature flags/waffles flags for the release notes. I tried it but didn’t have enough capacity to finish the script.
Backports
[Feanil]: Even if it’s not officially supported, should we allow the community to provide backport fixes to older releases at the release managers discretion? I feel like people are making these on their local forks and we could just let them upstream them if they can be validated easily.
[Maria] Is the BTR issue for tracking backports enough? I don’t think it’s being used. Also, how are we testing them? Is it reported anywhere? I think Max did this in some capacity, I’m not sure though.
...