Postmortems — Blameless Culture & Root Cause Analysis
Tại sao postmortem quan trọng
Một incident xảy ra. Team cứu được customers. Everyone is tired.
Natural tendency: move on, forget about it, hope it doesn't happen again.
Problem: Nếu không analyze incident, root cause stays. It happens again. Maybe even worse lần tới.
Natural tendency #2: "Person X made a mistake, let's fire them."
Problem: You lose good engineer, root cause stays, và lesson lost.
Postmortem breaks this cycle. It says: "We had incident. What can we learn? How do we prevent it next time?"
This is not about blame. It's about system improvement.
Internal Model: Blameless postmortem
What "blameless" means
"Blameless" does NOT mean:
- No one did anything wrong
- Mistakes weren't made
- We should ignore human error
"Blameless" DOES mean:
- We focus on system factors, not individual failings
- We ask "why did system allow this mistake?" not "why did person make mistake?"
- We assume people act rationally given information they had
Blameless example
Scenario: Engineer deployed code that crashed production database. Entire service down for 30 minutes.
Blame approach: "Engineer didn't test properly. Need to fire/discipline them or require more code review."
Result:
- Engineer feels targeted, leaves company
- Team becomes risk-averse
- Root cause (no pre-prod database testing) not fixed
- Similar incident happens again with different engineer
Blameless approach: "Engineer deployed change. In hindsight, it was bad. Why did system allow this?"
Analysis:
- Code review didn't catch the issue (reviewers not familiar with database internals?)
- No pre-production database environment to test against
- No canary deployment (change went to 100% immediately)
- Alert for database health was too slow (took 3 minutes to notice)
Fixes:
- Add pre-prod database environment
- Implement canary deployment (1% → 10% → 50% → 100%)
- Improve database health alerts to < 1 minute
Result:
- Engineer learns from experience, stays
- System gets better
- Next risky change is caught by improved process
The difference: blame focuses on person, blameless focuses on system.
The "Swiss cheese" model
Systems have multiple layers of defense:
Layer 1: Code review
(should catch issues)
↓ [reviewer missed issue]
Layer 2: Pre-prod environment
(should surface issues)
↓ [no pre-prod env exists]
Layer 3: Canary deployment
(should limit blast radius)
↓ [no canary, went to 100%]
Layer 4: Database alerts
(should detect problem)
↓ [alert too slow]
Layer 5: Disaster recovery
(should restore service)
↓ [RTO took 30 minutes]
Result: SERVICE DOWN FOR 30 MINBlameless postmortem finds holes in layers, not blames the engineer.
Blame postmortem finds the engineer at the layer they touched.
Which is more useful?
Internal Model: Root cause analysis — 5 Whys
What is root cause?
There's often immediate cause and root cause:
Immediate cause: Engineer deployed bad code
Root cause: No pre-production database environment for testing
Immediate cause: Database query was slow
Root cause: Query not optimized; no test for query performance
Immediate cause: Alert didn't fire
Root cause: Alert threshold too loose; no one tested alert in stagingConfusing immediate cause with root cause leads to shallow fixes:
Shallow fix: "Fire engineer" (prevents that person from deploying again)
Deep fix: "Add pre-prod environment" (prevents anyone from deploying untested DB changes)5 Whys technique
Ask "why?" repeatedly until you reach root cause:
Incident: Database crashed
Q1: Why did database crash?
A: Query consumed 100% CPU
Q2: Why did query consume 100% CPU?
A: Query was unoptimized (missing index)
Q3: Why was query missing index?
A: No automated test for query performance
Q4: Why no automated test?
A: Database team doesn't have resources to add performance tests
Q5: Why no resources?
A: Company didn't prioritize database testing infrastructure
Root cause: Organizational priority, not engineer error
Fix: Allocate resources to database test infrastructureOther root cause analysis techniques
Fishbone diagram (Ishikawa):
┌─ Code review process (missed issue)
├─ Testing (no pre-prod env)
DATABASE─→ ├─ Deployment (no canary)
INCIDENT ├─ Monitoring (alert too slow)
└─ Communication (took time to discover)Timeline analysis:
13:45 - Deployment started
13:46 - Database queries started failing (first sign)
13:49 - First alert fired (3 min delay)
13:52 - On-call paged
13:54 - On-call logged in
14:00 - Incident response started
14:05 - Root cause found (database query issue)
14:30 - Fix deployed
Total: 45 minutes from issue start to alert fire
15 minutes from alert to response
25 minutes from response to fixQuestion each gap: "Why 3 minutes for alert?" "Why 2 minutes for page?" etc.
Conducting postmortem — Process
Timing
Schedule immediately while memory fresh:
- Level 1 incident → postmortem same day (if after hours) or next morning
- Level 2 incident → postmortem within 24 hours
- Level 3 incident → postmortem within 1 week
Why immediately? Memory fades. Details lost. Team gets busy with next thing.
Who attends?
- Incident responders (IC, SMEs) — they know what happened
- IC's manager — optional, for visibility
- Postmortem facilitator (trained moderator) — neutral party
- NOT: People not involved (they distract, take bandwidth)
- NOT: VP / executives (makes people nervous about blame)
Usually 5–8 people max.
Postmortem structure
Duration: 60–90 minutes for typical incident
Timeline (example for 60 min postmortem):
1. Facilitator intro (5 min)
- State goal: "We're here to learn, not blame"
- Facilitate, don't judge
- Confidentiality (what's said stays here)
2. Incident narrative (15 min)
- IC describes timeline (what happened, when)
- "At 13:45, deployment started. At 14:00, alerts fired."
- Focus on facts, not judgments
3. Root cause discussion (20 min)
- Facilitator asks: "Why did this happen?"
- Use 5 Whys or fishbone
- Explore multiple contributing factors
- "It wasn't just the query. It was also no alerting."
4. Contributing factors (10 min)
- List all factors that contributed
- "Query unoptimized, no pre-prod env, alert threshold high"
- Distinguish from root cause
5. Action items (10 min)
- What will we fix?
- Each action item: "who, what, when?"
- Action items address root cause + contributing factors
6. Summary (5 min)
- Facilitator summarizes lessons
- "Key lesson: test in pre-prod before deploying DB changes"Asking the right questions (facilitator role)
Questions that blame:
- "Why did you deploy without testing?" (accusatory)
Questions that learn:
- "Was there a pre-production database environment available?" (system question)
- "What tooling exists to test database changes?" (improvement question)
Writing postmortem document
Standard format:
POSTMORTEM: Database Outage on 2025-06-24
Incident Duration: 13:45 - 14:30 (45 minutes)
Severity: Level 1
Incident Commander: Alex Chen
Facilitator: Bao Pham
Timeline:
13:45 - Deployment of feature X started
13:46 - Database queries began returning timeouts
13:49 - Alert "Database CPU > 90%" fired
13:52 - On-call engineer Alex paged
13:54 - Alex logged in, entered war room
14:00 - Root cause identified: missing index on new query
14:05 - Hotfix deployed (added index)
14:30 - Service fully recovered
Impact:
- 30,000 users affected
- 45 minutes of elevated error rate
- 15% of requests failed
- SLO breached (availability 98.5% vs 99.9% target)
Root Cause:
Database query added in feature X was not optimized. Query used full table scan instead of index, consuming 100% database CPU and blocking other queries.
Contributing Factors:
1. No pre-production database environment to test against
2. Code review process didn't catch unoptimized query
(reviewers not familiar with database query patterns)
3. Database performance tests not automated
4. Alert for database health had 3-minute delay
5. Deployment went to 100% immediately (no canary)
Lessons Learned:
1. Database changes require pre-production testing
2. Code review process should include database SME or automated query performance check
3. Alerts should fire within 1 minute of issue
4. Risky changes should use canary deployment
Action Items:
1. [P0] Set up pre-production database environment for testing
Owner: Database team | Due: 1 week
2. [P0] Add automated database query performance tests
Owner: Database + Backend team | Due: 2 weeks
3. [P1] Require database SME review for all DB queries
Owner: Engineering leads | Due: Immediate (process change)
4. [P1] Reduce database health alert latency from 3 min to < 1 min
Owner: Monitoring team | Due: 1 week
5. [P2] Implement canary deployment (0% → 10% → 50% → 100%)
Owner: DevOps | Due: 3 weeks
Timeline for fixes:
- P0 items: Week 1
- P1 items: Week 2
- P2 items: Week 3–4Action items — making them stick
Common problem: Action items written but never done.
Good action item:
Add automated database performance tests
Owner: Database team (specific person: Carmen)
Due: July 1, 2025
Success criteria:
- All new queries require performance test
- Tests run on every deploy
- Builds fail if query performance regresses > 20%Bad action item:
Improve database processes
Owner: Team
Due: Soon
Success criteria: Make sure queries are fasterThe difference:
- Good: specific, measurable, assigned to person, has deadline
- Bad: vague, subjective, no owner, no deadline
Follow-up: Every week, review action items. Track progress. If action item slipping, escalate.
Building blameless culture
Culture != policy
You can't enforce blameless culture with policy. You build it through:
Visible leadership commitment
- When senior engineer has incident, postmortem is just as thorough
- Leader doesn't say "oh, skip the postmortem for this one"
Rapid response to blame culture
- If someone blames individual in postmortem, facilitator immediately corrects
- "We focus on system factors here, not person blame"
Showing improvements happen
- Do the action items from postmortems
- Show team: "Last month's incident led to this improvement"
- If postmortems are ignored, team learns: "postmortems don't matter, it's just theater"
Safe feedback on process
- After postmortem, ask: "Was this helpful? What could be better?"
- Example: "Meeting felt rushed, people didn't speak up"
- Fix: Next postmortem, take more time for root cause discussion
Anti-pattern: Blame masquerading as blameless
"We're blameless!" team says, but in postmortem:
Facilitator: "Why did this happen?"
Engineer: "Well... I made a mistake..."
Manager: [silent glare at engineer]
Others: [uncomfortable silence]
Facilitator: "Anything else?"
Engineer: [shaken] "No, I think that's it."This is blame culture with blameless label. The glare communicated the real message.
Real blameless:
Facilitator: "Why did this happen?"
Engineer: "Deployed without testing because deadline was tight"
Manager: [nods supportively]
Facilitator: "That makes sense. Was testing infrastructure available?"
Engineer: "Pre-prod database would have caught it, but we don't have one"
Facilitator: "So system didn't provide the tooling needed. What would help?"
Engineer: "Pre-prod environment would let us test safely"
Facilitator: "Good insight. Let's make that an action item."
Manager: [supportive] "And let's also look at timeline pressure"The difference: focus on system, not person.
Measuring postmortem effectiveness
Metric 1: Action items completion
Q2 Summary:
- Action items created: 24
- Action items completed: 18 (75%)
- Action items still open: 6
Trend: 60% → 70% → 75% (improving)If completion rate < 50%, postmortems are not driving improvement.
Metric 2: Incident repeat rate
Incident: "Unoptimized database query causes outage"
- Q1: happened 2 times
- Q2: happened 1 time
- Q3: happened 0 times (postmortem action items working)Same incident happening again = postmortem didn't work.
Metric 3: Mean time to detect (MTTD)
After alerting improvement action item:
- Before: Alert fired 3 minutes after issue
- After: Alert fires < 1 minutePostmortem improvements should show measurable change.
Common postmortem mistakes
Mistake 1: Postmortem too soon
Team exhausted from incident. People haven't slept. Postmortem happening at midnight.
Result: Shallow analysis, people just want to go home.
Fix: Wait until team is rested. 24 hours later is usually better than same evening.
Mistake 2: Postmortem with too many people
VP attends. Customer representative attends. Marketing director attends.
Result: People nervous, don't speak freely, analysis is defensive.
Fix: Keep to incident responders only. Share summary with leadership after.
Mistake 3: Blame language disguised as blameless
"Why did engineer deploy without adequate testing?" (implies engineer's job)
vs.
"What testing infrastructure existed?" (focuses on system)
First is blame, second is blameless. Watch for subtle language.
Mistake 4: Postmortem for minor incidents
Every incident gets 60-minute postmortem. Level 3 incident (no customer impact) gets same treatment as Level 1.
Result: Postmortem fatigue, team hates postmortems.
Fix: Level 3 → async brief post (15 min, 1 person). Level 2 → 30 min meeting. Level 1 → 60 min meeting.
Summary
Postmortems are the process that turns failures into improvements:
□ Blameless means focusing on system, not person
□ Root cause analysis (5 Whys, fishbone) digs to real cause
□ Timeline analysis identifies process delays
□ Action items should be specific, assigned, and measurable
□ Postmortem culture built through leadership commitment and visible improvement
□ Measure effectiveness through completion rate and incident recurrence