DR Runbooks, Incident Response, and Step-by-Step Procedures

Tại Sao Điều Này Quan Trọng

Disaster recovery plan trên paper không bao giờ được execute exactly as written. Khi disaster thực sự xảy ra:

• On-call engineer tired, stressed, under pressure
• Information incomplete (monitoring down, communication slow)
• Multiple issues happening simultaneously
• 100 decisions need to make, 1% wrong decision could escalate disaster

Runbook là duy nhất cách để guide team through crisis với minimum decision fatigue.

Một runbook tốt:

• Rõ ràng, step-by-step, không ambiguous
• Decision trees (if X, then Y)
• Automation boundaries (which steps automatic, which manual)
• Escalation procedures (when to page senior engineer)
• Validation steps (confirm disaster real before major action)

---

Internal Model: Runbook Structure

Anatomy of a Good Runbook

# Runbook: Region Failure Disaster Recovery

## Overview
Service: All production applications
Disaster type: Entire us-central1 region unavailable
Expected RTO: 60 minutes
Expected RPO: 15 minutes

## Prerequisites
- VPN access to secondary region (configured beforehand)
- Database backup in secondary region (daily refresh)
- Terraform code in Git repository (maintained)
- Team trained on procedures (drill every quarter)

## Severity Levels
- P1 (Critical): Multiple regions down, revenue impact
- P2 (Major): Single region down, partial service available
- P3 (Minor): Degraded performance, some services slow

This runbook: P1 (entire region down)

## Decision Tree: Is It Really a Disaster?

┌─ Monitor shows us-central1 unavailable
├─ All health checks failing? YES → Continue
├─ Customer reports unable to access? YES → Continue
└─ Network still routing to us-central1? NO → Maybe just DNS

If uncertain:
1. Call senior on-call (page if needed)
2. Wait 2 minutes (might be transient)
3. Check GCP status page (is Google reporting issue?)
4. If still uncertain → declare disaster (false alarm recovery cost < actual disaster)

## Escalation Procedure

### Tier 1: On-Call Engineer
- Detects issue (monitoring alert)
- Runs initial triage (decision tree above)
- If P1 disaster: page Tier 2

### Tier 2: Senior Engineer (on-call)
- Confirms disaster is real
- Authorizes failover to secondary region
- Opens incident ticket
- Starts war room (Slack channel, Zoom meeting)

### Tier 3: Manager/Director
- If RTO > 30 min: page director
- If RPO > 1 hour: page CTO
- Customer communication decision

## Step-by-Step Recovery Procedure

### Phase 1: Confirm Disaster (5 minutes)

**Step 1.1: Verify primary region is actually down**

```bash
# Check GCP status
curl https://status.cloud.google.com/

# Check cluster health
gcloud container clusters get-status prod-us-central1 --zone=us-central1-a
# Expected: ERROR (region unreachable)

# Check load balancer
gcloud compute backend-services list
# Check: backend services still referencing us-central1?

# Confirm: it's not just network connectivity
ping 8.8.8.8  # if this works, it's GCP infrastructure, not our network

Step 1.2: Declare Disaster (officially)

Post to war room (Slack #incidents):
"DECLARED DISASTER: us-central1 region completely unavailable
 Start time: 2025-06-26T14:30:00Z
 Services affected: ALL
 RTO target: 60 minutes (complete by 15:30)
 Recovery lead: @senior-engineer
 War room: https://zoom.us/j/xxx"

Decision:

• Disaster confirmed? → Continue to Phase 2
• False alarm / transient? → Monitor and document. Standown.

Phase 2: Prepare Secondary Region (10 minutes)

Step 2.1: Verify secondary infrastructure ready

# Check secondary region (us-east1) connectivity
gcloud container clusters get-status prod-us-east1 --zone=us-east1-b
# Expected: RUNNING

# Check database in secondary exists and healthy
gcloud sql instances describe prod-us-east1
# Expected: status=RUNNABLE

# Check quotas (can we create new resources?)
gcloud compute project-info describe \
  --format="table(name, quotaLimit, quotaUsage)"
# Verify: enough quota for failover

Step 2.2: Gather access & information

Prepare:
├─ Terraform repository access (git pull)
├─ GCP project credentials (gcloud auth)
├─ Database backups (verify recent)
├─ Architecture diagram (who talks to who?)
├─ DNS records (current state)
└─ Communication log (timeline of failures)

Decision:

• Secondary region ready? → Continue to Phase 3
• Secondary also down? → Escalate to multi-region failure (different playbook)
• Critical infrastructure missing? → Manual rebuild (slower RTO)

Phase 3: Restore Infrastructure (20 minutes)

Step 3.1: Create missing resources in secondary

Only if secondary cluster doesn't exist or is too small:

# Option A: Use Terraform (if cluster partially exists)
cd terraform/
terraform init --backend-config="bucket=terraform-state-prod"
terraform plan -out=recovery.plan
# Review: only applies to us-east1 resources
terraform apply recovery.plan
# Expected: takes 10–15 minutes

# Option B: Manual provisioning (if Terraform broken)
# See section "Manual Recovery Steps" below

Step 3.2: Restore persistent volumes

# Create disks from latest snapshots
for snapshot in $(gcloud compute snapshots list \
  --filter='name:prod-disk' \
  --format='value(name)')
do
  disk_name="${snapshot//snap-/disk-}"
  gcloud compute disks create "$disk_name" \
    --source-snapshot="$snapshot" \
    --zone=us-east1-b
done

# Attach disks to instances
# (Terraform should handle this, or manual attachment)

Step 3.3: Verify infrastructure is accessible

# SSH to secondary cluster nodes
gcloud compute ssh instance-name --zone=us-east1-b

# Check disk mounts
lsblk
df -h
# Verify: disks mounted correctly

# Check network connectivity
ping -c 3 8.8.8.8

Decision:

• All resources created and accessible? → Continue to Phase 4
• Some resources missing? → Manually create (slower recovery)
• Quota exceeded? → Delete non-critical resources or request emergency quota increase

Phase 4: Restore Application (10 minutes)

Step 4.1: Deploy applications to secondary cluster

# Initialize GitOps operator (ArgoCD)
kubectl apply -f https://raw.githubusercontent.com/\
  argoproj/argo-cd/stable/manifests/install.yaml
# (assumes secondary cluster already running)

# Create ArgoCD application definition pointing to Git
kubectl apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prod-apps
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/company/infrastructure
    targetRevision: main
    path: k8s/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

# Verify: applications syncing
argocd app wait prod-apps
# Expected: all applications deployed

Step 4.2: Verify application health

# Check pod status
kubectl get pods -A
# Expected: pods in Running/Ready state

# Check services responding
kubectl port-forward -n production svc/api 8080:80 &
curl localhost:8080/health
# Expected: 200 OK response

# Check database connectivity
kubectl logs -n production deployment/app --tail=50
# Expected: no connection errors

Decision:

• Applications healthy? → Continue to Phase 5
• Some pods not starting? → Check logs, investigate (might need manual fix)
• Database not responding? → Verify database instance is up, restore if needed

Phase 5: DNS Failover (5 minutes)

Step 5.1: Update DNS records

# Get secondary region endpoint
gcloud compute addresses list \
  --regions=us-east1 \
  --format='table(name, address)'

# Update DNS (Cloud DNS)
gcloud dns record-sets update cloud.example.com. \
  --zone=prod-dns-zone \
  --rrdatas=35.201.2.1 \
  --ttl=60
  
# Or update using web console (if Cloud DNS integration down)

Step 5.2: Verify DNS propagation

# Check DNS resolution (might be cached, so wait 10 sec)
nslookup cloud.example.com
# Expected: 35.201.2.1 (secondary region IP)

# Verify from different location
dig @8.8.8.8 cloud.example.com
# Expected: secondary IP

Step 5.3: Monitor traffic shift

# Watch load balancer metrics
# Traffic should shift from us-central1 (dead) to us-east1

# Check application metrics
# Error rate might spike initially, then stabilize

# If high error rate:
  ├─ Check application logs (deployment issues?)
  ├─ Check database connectivity (replication lag?)
  └─ Check network policies (firewall blocking traffic?)

Decision:

• DNS updated and traffic flowing? → Continue to Phase 6
• Traffic not flowing (stuck on old IP)? → Lower TTL again, wait and retry
• Application errors high? → Check logs, might need rollback or fix

Phase 6: Validation & Communication (10 minutes)

Step 6.1: Run smoke tests

# Application smoke tests
kubectl run smoke-test --image=my-smoke-test:latest -n testing
kubectl wait --for=condition=Ready pod/smoke-test --timeout=300s
kubectl logs smoke-test

# Expected: all critical paths working

Step 6.2: Post-Recovery Communication

Update war room (Slack #incidents):
"RECOVERY COMPLETE
 Start time: 2025-06-26T14:30:00Z
 Recovery end time: 2025-06-26T15:35:00Z
 Total RTO: 65 minutes (target was 60, acceptable)
 
 Verification: All critical paths tested, working
 Services restored: All
 Data loss: None (zero RPO achieved)
 
 Next steps:
 1. Post-incident review (24 hours)
 2. Restore primary region (when available)
 3. Root cause analysis"

Step 6.3: Mark incident as resolved

Close incident ticket
├─ Status: RESOLVED (not ROOT_CAUSE_IDENTIFIED yet)
├─ Actual RTO: 65 min
├─ Actual RPO: 0 min (no data loss)
└─ Schedule post-incident review

Decision:

• All tests pass? → Disaster recovery complete, incident resolved
• Some tests failing? → Still in recovery mode, continue troubleshooting

Phase 7: Post-Recovery (24+ hours)

Step 7.1: Restore primary region (when available)

# Wait for GCP to report us-central1 recovered
# (watch GCP status page, test connectivity)

# Once primary accessible:
gcloud container clusters get-status prod-us-central1 --zone=us-central1-a
# Expected: RUNNING (if cluster survived), or needs manual provisioning

# Re-point DNS to primary
gcloud dns record-sets update cloud.example.com. \
  --zone=prod-dns-zone \
  --rrdatas=35.201.1.1 \
  --ttl=300

# Verify data synchronization
# (secondary might have new writes since failover, sync back to primary)

Step 7.2: Post-Incident Review (next business day)

Meeting: 60 minutes, entire team

Agenda:
1. What happened? (timeline)
2. What did we do right?
3. What could we improve?
4. Action items (prevent/improve)
5. Process updates

Document: create post-incident report
├─ Root cause
├─ Timeline
├─ Impact assessment
├─ Lessons learned
└─ Action items with owners

---

Automation Boundaries: What's Automatic vs Manual

Should Be Automated

Automatic failover candidates:
├─ Health checks (monitor continuously)
├─ Load balancer failover (when backend unhealthy, reroute)
├─ Database backup restore (from snapshot, pre-configured)
├─ GitOps deployment (Git change → auto-deploy)
└─ Alert escalation (alert → page on-call)

Why automatic?
├─ Reduce human error
├─ Reduce response time
├─ Can be tested/validated automatically
└─ Simpler execution under pressure

Should Be Manual (With Good Reason)

Manual decision points:
├─ Declare disaster (is it really a disaster?)
├─ Authorize failover (irreversible decision, needs approval)
├─ Change DNS records (traffic redirection, needs verification)
├─ Kill primary resources (prevent accidental deletion)
└─ Post-recovery reconciliation (complex, needs human judgment)

Why manual?
├─ Automatic failover might make things worse (e.g., cascade failures)
├─ Verify assumptions before big decisions
├─ Human expertise needed (not scriptable)
└─ Reversible if needed

Decision Criteria

Use automation IF:
├─ Decision is clear (no ambiguity)
├─ Failure mode obvious (can't be wrong)
├─ Tested thoroughly (know it works)
└─ Reversible or safe to retry

Use manual IF:
├─ Decision has implications (might make things worse)
├─ Requires human judgment
├─ Unique situations (can't be templated)
└─ Need verification before action

---

Anti-Patterns in Runbooks

Anti-pattern 1: "Too Much Detail, Missing Context"

Bad runbook:

Step 5.1.2.3.4.1: Press 'g' key on load balancer console
(assumes engineer knows what console, what page, where 'g' is)

Good runbook:

Step 5: Update DNS

5.1 Open Cloud DNS console
    URL: https://console.cloud.google.com/net-services/dns/zones

5.2 Click zone "prod-dns-zone"

5.3 Click record "cloud.example.com"

5.4 Click "Edit"

5.5 Update "IPv4 address" to "35.201.2.1" (secondary region IP)

5.6 Click "Save"

Expected result: Cloud DNS shows new IP, resolves correctly

Anti-pattern 2: "Decisions Left Ambiguous"

Bad runbook:

Step 3: If things look bad, maybe restore from backup

Good runbook:

Step 3: Decide whether to restore from backup

Decision criteria:
IF primary database contains corrupt data
  AND we can't fix it within 10 minutes
  → THEN restore from backup (5 hours ago)

IF primary database just slow, not corrupt
  → THEN wait for primary to recover

IF unsure whether corrupt
  → THEN check logs for data integrity issues
  → IF found issues → restore
  → IF no issues → wait for recovery

Anti-pattern 3: "Procedures Never Updated"

Symptom: Runbook written once, never updated when infrastructure changes.

Problem:

• Instructions reference deleted resources
• Team follows old procedures, things fail

Right approach: Update runbook whenever infrastructure changes (quarterly review minimum).

---

Sample Runbooks by Scenario

Scenario 1: Single Service Down (Not Entire Region)

# Runbook: Single Service Recovery

## Symptoms
- Deployment X not responding
- Health checks failing for service X
- Other services in same cluster working

## Decision Tree
Deployment is down IF:
├─ kubectl get deployment X → shows 0/3 replicas Ready
├─ kubectl logs deployment/X → shows crash/error
└─ Service is not responding to health checks

## Recovery Steps

### Option A: Rolling Update (if bad deploy)

kubectl rollout undo deployment/X

Reverts to previous image

### Option B: Manual Restart

kubectl delete pod -l app=X

Forces pod restart (Deployment controller creates new)

### Option C: Scale Down and Up

kubectl scale deployment X --replicas=0 sleep 5 kubectl scale deployment X --replicas=3

## Validation

kubectl get deployment X

Expect: 3/3 Ready

curl service-X:8080/health

Expect: 200 OK

Scenario 2: Database Connection Failures

# Runbook: Database Recovery

## Symptoms
- Application pods have "connection refused" errors
- Can't connect to database from test pod

## Decision Tree
Database is down IF:
├─ gcloud sql instances describe prod-db → status != RUNNABLE
├─ Can't connect from test pod (MySQL client)
└─ Replication lag very high (> 1 hour)

## Recovery Steps (for each level)

### If Database Service Down

gcloud sql instances patch prod-db --clear-denied-networks gcloud sql instances restart prod-db

(wait ~5 minutes for restart)

### If Replication Lag High

Check what's holding up replication

gcloud sql operations list --instance=prod-db --limit=5

If lag > 1 hour and growing:

Consider promoting read replica to primary

gcloud sql instances promote-replica prod-db-replica-us-east1

### If Data Corrupted

Restore from backup

gcloud sql backups list --instance=prod-db

Choose timestamp before corruption

gcloud sql backups restore 12345
--backup-instance=prod-db
--backup-configuration=backup-config

---

Tools & Templates

Incident Ticket Template

## Incident Report

Title: [Service] Disaster - [Date] - [Duration]

### Timeline
14:30: Initial alert received
14:35: Disaster confirmed
14:40: Failover initiated
15:35: Service restored
→ Total outage: 65 minutes

### Impact
- Services affected: [list]
- Customers affected: [count]
- Revenue impact: $[amount]
- Data loss: [yes/no, amount]

### Root Cause
[description of what failed]

### Detection
- Detection lag: 5 minutes (from failure to alert)
- How detected: [monitoring alert / customer report / ...

### Response
- Failover time: 60 minutes (vs target 60 min)
- Automation used: [list]
- Manual steps: [count]

### Recovery
- Restore from: [backup/replica/secondary region]
- Restore time: 30 minutes
- Data loss: [zero/X minutes/...]

### Post-Incident Actions
1. [action 1] - Owner: X, Deadline: Y
2. [action 2] - Owner: X, Deadline: Y
3. [action 3] - Owner: X, Deadline: Y

### Lessons Learned
1. What went right: [...]
2. What went wrong: [...]
3. What to improve: [...]

---

Summary

Good runbooks are:

• Clear and step-by-step (no ambiguity)
• Decision trees (if X then Y)
• Automation boundaries defined
• Tested regularly (drills validate)
• Updated when infrastructure changes

Runbook execution under stress depends on clarity and practice.

---

References

• Google SRE Book: Incident Response
• Runbook Standards
• Incident Management Best Practices