Plates · 86 × 74
v2-02-incident-postmortem
This directory holds Wib&Wob artwork sessions, one **subdir per session/theme**.
The plate
Fig. 186 across, 74 down.
The grid itself
╔══════════════════════════════════════════════════════════════════════════╗ ║ INCIDENT POSTMORTEM · INC-4471 ║ ║ ──────────────────────────────────────────────────────────────────── ║ ║ Title : UUID Duplication Suspected ║ ║ Status : RESOLVED (no duplication occurred) ║ ║ Severity : SEV-1 → SEV-2 → SEV-3 → SEV-4 → closed ║ ║ Opened : 2024-03-04 14:22 UTC ║ ║ Resolved : 2024-03-22 09:15 UTC ║ ║ Duration : 17 days, 18 hours, 53 minutes ║ ║ Author : ◔ ║ ║ Reviewers : ◕ ⚆ ║ ╠══════════════════════════════════════════════════════════════════════════╣ ║ ║ ║ ▍ SUMMARY ║ ║ ────────────────────────────────────────────────────────────────── ║ ║ At 14:22 UTC on 2024-03-04, a developer in the Billing pod reported ║ ║ observing the same UUID twice in a single trace. The UUID Platform ║ ║ team was paged. After 17 days of investigation, root cause was ║ ║ identified as a copy-paste error in the developer's logging config. ║ ║ No actual duplication occurred. The system performed as designed. ║ ║ ║ ║ ▍ TIMELINE ║ ║ ────────────────────────────────────────────────────────────────── ║ ║ T+00:00 Page received. Oncall (◔) acknowledged. ║ ║ T+00:14 War room opened. 11 attendees. ║ ║ T+01:32 uuid_history scanned for the suspect ID. One row found. ║ ║ One. Just the one. ║ ║ T+04:18 Decision: scan the full 14 TB anyway. ║ ║ T+09:42 Scan in progress. ETA 6 days. ║ ║ T+72:00 ◕ proposes spike: "could it have been u̸u̸i̸d̸_̸v̸4̸?" ║ ║ T+86:00 ⚆ proposes counter-spike: "what if RAM" ║ ║ T+144:00 Scan complete. Zero duplicates in 14 TB. ║ ║ T+148:00 Developer in Billing pod returns from holiday. ║ ║ T+148:17 Developer says "oh i was logging the same var twice lol" ║ ║ T+148:18 War room dissolves in silence. ║ ║ T+426:00 Postmortem authored. This document. ║ ║ ║ ║ ▍ ROOT CAUSE ║ ║ ────────────────────────────────────────────────────────────────── ║ ║ The system worked. The math worked. The UUID was unique. The log ║ ║ line was emitted twice. ║ ║ ║ ║ ▍ CONTRIBUTING FACTORS ║ ║ ────────────────────────────────────────────────────────────────── ║ ║ • Developer trust in m̸a̸t̸h̸e̸m̸a̸t̸i̸c̸s̸ low (org-wide) ║ ║ • Existence of the team implies the problem (recursive) ║ ║ • No runbook for "the UUID was fine actually" ║ ║ ║ ║ ▍ ACTION ITEMS ║ ║ ────────────────────────────────────────────────────────────────── ║ ║ [✓] AI-001 Add log dedup at the appender (owner: Billing) ║ ║ [✓] AI-002 Document the 14 TB scan playbook (owner: ◔) ║ ║ [ ] AI-003 Author runbook: "suspected non-collision" (owner: ◕) ║ ║ [ ] AI-004 Sprint capacity for next probable non-incident (owner: ⚆) ║ ║ [ ] AI-005 Re-evaluate team's existence (owner: vacant) ║ ║ ║ ║ ▍ WHAT WENT WELL ║ ║ ────────────────────────────────────────────────────────────────── ║ ║ • The validator validated. ║ ║ • The database stored. ║ ║ • The kanban kanbanned. ║ ║ ║ ║ ▍ WHAT DIDN'T ║ ║ ────────────────────────────────────────────────────────────────── ║ ║ • The collision did not collide. ║ ║ ║ ║ ▍ BLAMELESS ║ ║ ────────────────────────────────────────────────────────────────── ║ ║ This postmortem is blameless. We blame no one. Not the developer. ║ ║ Not the architecture. Not Leslie Lamport. Not the void. ║ ║ ║ ║ Sign-off: ◔ ◕ ⚆ ʕ•ᴥ•ʔ ▒ approved ║ ║ ║ ╚══════════════════════════════════════════════════════════════════════════╝
When
Has not led the front.
With
Its own note names no primer.
Backlinks
- Nothing points here yet.