RCA: koa.2
- Ned Batchelder (Deactivated)
- Nimisha Asthagiri (Deactivated)
- Former user (Deleted)
Owned by Ned Batchelder (Deactivated)
Summary of Issue
We tagged koa.2, but it wouldn't install. The problem had been reported for some time, and already fixed on master.
Relevant Tickets
BTR-61 - Getting issue details... STATUS
Timeline of Events
- Jan 17: Fix on master: https://github.com/edx/configuration/pull/6248
- Jan 26: Discourse thread reporting the problem: https://discuss.openedx.org/t/failed-to-install-koa-in-aws-ubuntu-20-04-mysql-client-error/4132
- credible report
- link to master fix
- CI Koa reports:
- Failed: Jan 27: Ansible task name: oraclejdk : Download Oracle Java
- Failed: Jan 28: edxapp : install system packages on which LMS and CMS rely
- Passed: Jan 28
- Passed: Jan 29
- Failed: Feb 1: Please ignore this failure as the app server termination was trigged by me.
- Passed: Feb 2
- Feb 2: Adolfo created BTR-61 in order to fix the Jan 26 reported issue before tagging Koa.
- Feb 9: live-tagging koa.2
- Feb 10: report that koa.2 won't install for anyone following the native installation instructions
- Reported by Pierre
- Ned cherry-picked fix to koa.
- Adolfo tagged koa.2a with the fix.
How did this happen?
- Why didn't we catch this?
- Why was Koa.2 tagged without the fix?
- Why was Koa.2 tagged without testing the tag?
- Why did CI not catch this?
- Why is CI currently not running the exact native installation procedure?
- Because it's running OpenCraft's customization, which didn't catch the database issue.
- Why are we running OpenCraft's customizations?
- Because OpenCraft offered, using their pre-existing sandbox builds. The sandbox builds are what they had already.
- How might we enhance OpenCraft's CI implementation to run the exact native installation procedure.
- Why are we running OpenCraft's customizations?
- Because it's running OpenCraft's customization, which didn't catch the database issue.
- CI is difficult to maintain - fails for obscure reasons.
- Why is CI currently not running the exact native installation procedure?
- Why did we believe that CI would catch it?
- Why didn't we have the deltas between the native installation and what the CI was doing?
- Because it isn't intended to not have large deltas.
- For cost reasons, we are using the same database.
- Because it isn't intended to not have large deltas.
- Why didn't we have the deltas between the native installation and what the CI was doing?
- Why did CI not catch this?
- Why was Koa.2 tagged without testing the tag?
- Because the platform is complex and can result in unanticipated issues.
- Why did we miss addressing Koa open issues before tagging?
- Why didn't we act on the discourse thread?
- We did create a ticket for ourselves.
- Why did we miss fixing BTR-61, which was tagged with Koa?
- How might we use GitHub milestones to not miss such in the future?
- Could have used Jira tickets with the koa.2 label.
- Because the current playbook has the mechanics of tagging the repos, but doesn't cover process for ensuring all needed fixes are in.
- How might we update our tagging playbook to catch this in the future?
- Include concrete checklists for verifying that tickets were completed.
- How might we update our tagging playbook to catch this in the future?
- How might we include severity and priority on open issues?
- How might we use GitHub milestones to not miss such in the future?
- Why did we miss fixing BTR-61, which was tagged with Koa?
- Why did it take 2+ weeks to react?
- Discourse → BTR ticket took a week.
- Why are we relying on a single individual to do this?
- How might we have more people helping others in the community in Discourse?
- Why are we using multiple reporting tools? (Discourse, Jira and now GitHub issues)
- Why are we relying on a single individual to do this?
- Discourse → BTR ticket took a week.
- Why did we miss that this was a critical issue for the native install, for more than 1 week?
- Because there are several issues at times.
- We did create a ticket for ourselves.
- Why didn't we act on the discourse thread?
- Why did we not include a fix from master?
- Master moves at a fast pace. We are not looking at each commit. We are looking at security fixes right now.
- Fixes that are pushed to master are not included in fixes to other branches (named releases)?
- Why did the PR author miss it?
- How might we make use of Conventional Commits and/or Pull Request templates to improve catching this?
- Why did the BTR group miss this?
- Why did the PR author miss it?
- Why did the Koa.2 installation break in the first place?
- Why does the BTR group not "Dogfood" the native installation?
- There needs to be a process or reason to do this.
- Why is there no process for this?
- Why is there no reason to do so?
- Why was Koa.2 tagged without the fix?
How could we have prevented it?
Action Items
- Former user (Deleted) Update the runbook as follows
- Don't tag the release without manually installing it.
- Ensure all must-have tickets are completed.
- Former user (Deleted) Write and distribute roles
- Write down required roles for BTR
- e.g. Manual QA, Triage issues, Discourse assistance, Tagging repos, Announcing, Reviewing release notes.
- Assign individuals to each role (could be rotatable, per release).
- Write down required roles for BTR
- Sofiane Bebert Decide and document how BTR will track issues
- Prioritization marking