Ulmo.1 Retrospective 🌳⏪

How to contribute 📝

1️⃣ Review the template and add your thoughts under the relevant sections.
2️⃣ Be specific! Share what worked well, what didn’t, and any actionable suggestions for improvement.
3️⃣ Tag yourself in your contributions (e.g., [Your Name]: Your feedback here).
4️⃣ Keep it constructive. The goal is to learn and refine our process.

Where to focus your feedback 🔎

✅ Successes & achievements – What went well? What should we continue doing?
❌ Challenges & improvements – What issues did we face? How could we handle them better next time?
📈 Action items for Ulmo – What concrete steps should we take to improve?

Some key areas to consider:

• Product 🏗️

• Testing process 🔍

• Bug triage & fixing 🐛

• Security 🔐

• Backports ⏪

• Release notes 📄

I. Context, data, and highlights 📈

Ulmo took longer than originally planned. The release was initially expected around December but was finally shipped in January after several delays.

Some of the factors that influenced the timeline were:

• Large feature work (RBAC and Content Libraries roles) landing close to the release window.

• Testing starting later than expected due to sandbox stabilization issues.

• Several release blockers discovered late in the cycle, including a security issue, content library blockers, and a grading race condition bug.

• Additional validation required in external environments (MIT infrastructure) before confirming fixes for the concurrency issue.

• A high volume of testing coordinated across the community, including multiple testathons to increase coverage.

Despite the delays, the release successfully shipped thanks to strong collaboration across multiple working groups. Thank you all!

II. Successes and achievements 🏅

• The changes to release toggle/waffle flag documentation automation @Sarina Canelake helped with made that part of Ulmo easier than falling off a log.

• Strong collaboration during testing, including several testathons that helped increase coverage.

• Fast response to blockers once issues were identified.

• Improvements to the testing sheet that made tracking progress easier.

• Automation improvements to toggle and waffle flag documentation.

• Good cross team coordination when debugging complex issues.

III. We should keep ✅

• Testathons and cross working group testing sessions.

• Clear communication in Slack and BTR meetings (1 message summary after each meeting).

• The testing sheet as the main coordination tool for testing.

• Collaborative debugging across teams when issues appear.

IV. What didn’t we do so well, and what could we have done to do better ❌

Product

• Large features such as RBAC and Content Libraries were still evolving close to the release window. This created uncertainty about whether they could safely be included and contributed to delays in the release timeline.

Testing

• Testing began later than expected because the sandbox environment required time to stabilize. Plugins were enabled incrementally and some services such as MinIO and Discovery caused initial instability.

• Several tests related to RBAC and Content Libraries had to remain hidden until backports were completed, which delayed testing of those areas.

Bug triage and fixing

• The issue backlog continues to grow across releases. Some issues remain open across multiple release cycles and it can be difficult to prioritize which ones should be addressed.

Security

• A security issue was discovered relatively late during testing. While the issue was addressed quickly, it still contributed to delays in the release schedule.

Backports

• Backports were required late in the cycle for several fixes and features, including RBAC related changes. This delayed testing in some areas and added complexity to the stabilization phase.

VI. Action Items for Verawood release 📈

Release

• I think Farhan and Kyle’s proposal to update the release process is excellent and we should consider adopting it for Verawood because it brings that process more in line with current reality on the ground.

Product

• Encourage large initiatives to define earlier milestones and checkpoints so that readiness can be evaluated sooner in the cycle.

• Introduce earlier readiness checks for major projects in Release Planning meetings so potential delays or scope adjustments can be identified before the release window.

Testing

• Prioritize testing of high impact and core platform functionality earlier in the testing cycle to surface potential release blockers sooner.

• Work with sandbox maintainers to ensure the testing environment reaches a stable baseline earlier in the release cycle.

• Document the sandbox rebuild schedule and deployment process so testers know when fixes will be available for retesting.

Bug triage and fixing

• Continue improving issue triage practices and encourage contributors to pick up bug fixes, especially for high priority issues that persist across multiple releases. More frequent releases may also help reduce backlog pressure.

Security

• Continue encouraging security awareness during testing and consider whether additional checks earlier in the cycle could help surface issues sooner.

Backports

• Clarify expectations around backporting fixes and features so contributors understand when backports are encouraged and how they affect testing timelines.

Release notes