DR Testing, Chaos Engineering, and Failure Validation

Tại Sao Điều Này Quan Trọng

DR plan trên paper ≠ DR plan works in practice. Vô số disaster recovery plans được viết nhưng thất bại khi actually needed because:

Nobody tested them
Assumptions bị sai (backup mất, script broken, permission denied)
Procedures mất ngày (expected 1 hour, took 6 hours)
Infrastructure không sẵn sàng (secondary region không có quota)

Testing là duy nhất cách để validate DR recovery thực sự hoạt động.

Internal Model: Disaster Recovery Verification

Three Levels of DR Validation

Level 1: Configuration Audit

Check (automated):
├─ Backups exist and are recent
├─ Cross-region replicas configured
├─ Snapshots scheduled and running
├─ Failover DNS records exist
└─ Runbooks documented

Cost: Low (automated checks)
Frequency: Weekly
Catches: 70% of obvious errors

Level 2: Partial Recovery Test

Test individual components (monthly):
├─ Restore database from backup (test, then delete)
├─ Restore GKE cluster from backup
├─ Verify snapshot accessibility
├─ Test DNS failover (manually update record)
└─ Verify secondary resources accessible

Cost: Medium (manual testing, infrastructure)
Frequency: Monthly
Catches: 90% of realistic errors

Level 3: Full Disaster Recovery Drill

Complete end-to-end recovery (quarterly):
├─ Simulate region failure
├─ Execute full recovery procedure
├─ Application fully functional in secondary region
├─ Run smoke tests
├─ Measure actual RTO

Cost: High (full infrastructure, team time, downtime)
Frequency: Quarterly (production: yearly, staging: quarterly)
Catches: 95%+ of real-world problems

Configuration Audit: Automated Checks

Backup Validation

python

# Cloud Function: daily backup validation

from google.cloud import backup_dr_v1
from google.cloud import monitoring_v3

def validate_backups(request):
    client = backup_dr_v1.BackupVaultClient()
    
    # Check all backup vaults
    vaults = client.list_backup_vaults(parent=f"projects/{PROJECT_ID}")
    
    results = []
    for vault in vaults:
        # List backups in vault
        backups = client.list_backups(parent=vault.name)
        
        latest_backup = None
        for backup in backups:
            if backup.create_time > (latest_backup.create_time if latest_backup else 0):
                latest_backup = backup
        
        # Check: backup exists and recent
        if not latest_backup:
            results.append({
                'vault': vault.name,
                'status': 'FAIL',
                'reason': 'No backups found'
            })
        else:
            age_hours = (datetime.now() - latest_backup.create_time).total_seconds() / 3600
            if age_hours > 24:
                results.append({
                    'vault': vault.name,
                    'status': 'FAIL',
                    'reason': f'Latest backup {age_hours:.1f} hours old'
                })
            else:
                results.append({
                    'vault': vault.name,
                    'status': 'PASS',
                    'age_hours': age_hours
                })
    
    # Report to monitoring
    report_to_monitoring(results)
    return results

# Schedule: Cloud Scheduler (daily 6 AM)

Replica Lag Monitoring

Continuous monitoring (per database):

Cloud SQL:
├─ Check replica lag < 60 seconds
├─ Alert if lag > 300 seconds
└─ Metric: cloudSQL/replication/replica_lag

Cloud Spanner:
├─ Check multi-region consistency
├─ Alert if consistency lag > 100ms
└─ Metric: spanner/cpu/smooth_utilization (proxy)

Snapshot Accessibility Test

python

# Monthly: verify snapshot can be restored

from google.cloud import compute_v1

def test_snapshot_restore():
    compute_client = compute_v1.DisksClient()
    
    # Pick latest snapshot
    snapshots = compute_client.list_snapshots(project=PROJECT_ID)
    latest_snap = list(snapshots)[-1]
    
    # Try to create disk from snapshot
    new_disk = compute.Disk()
    new_disk.name = f"test-restore-{uuid.uuid4()}"
    new_disk.source_snapshot = latest_snap.self_link
    
    try:
        operation = compute_client.create(
            project=PROJECT_ID,
            zone=TEST_ZONE,
            resource=new_disk
        )
        
        # Wait for completion
        while not operation.done:
            time.sleep(1)
        
        # Cleanup
        compute_client.delete(project=PROJECT_ID, resource=new_disk.name)
        
        return {'status': 'PASS', 'message': 'Snapshot restore successful'}
    except Exception as e:
        return {'status': 'FAIL', 'message': str(e)}

Partial Recovery Tests: Monthly Component Tests

Test 1: Database Restore

Frequency: Monthly

Procedure:

1. Identify latest backup (from Cloud SQL backup list)
2. Create new test instance from backup
3. Verify:
   ├─ Instance size matches production
   ├─ Databases restored
   ├─ Tables accessible
   └─ Row count matches expected
4. Run sample queries
5. Delete test instance

Time: 30–45 minutes
Cost: ~$5 (test instance + backup restore)

Example (Cloud SQL):

bash

# List backups
gcloud sql backups list --instance=prod-mysql

# Restore from backup
gcloud sql backups restore [BACKUP_ID] \
  --backup-configuration=backup-config \
  --backup-instance=prod-mysql

# Create test instance from backup point-in-time
gcloud sql instances clone prod-mysql test-mysql-temp \
  --point-in-time=2025-06-26T10:00:00Z

Test 2: GKE Backup Restore

Frequency: Monthly

Procedure:

1. Pick latest GKE backup
2. Create test cluster (same size as primary)
3. Restore backup to test cluster
4. Verify:
   ├─ Deployments created
   ├─ Pods running
   ├─ Services accessible
   ├─ Storage volumes mounted
   └─ Readiness probes passing
5. Run application smoke tests (basic queries)
6. Delete test cluster

Time: 1–2 hours
Cost: ~$50 (test cluster)

Test 3: Snapshot Restore and Mount

Frequency: Monthly

Procedure:

1. Pick latest PD snapshot
2. Create disk from snapshot
3. Attach to test instance
4. Mount and verify filesystem
5. Sample data integrity check (file hashes, row counts)
6. Cleanup

Time: 20 minutes
Cost: ~$2 (disk, attach)

Full Disaster Recovery Drill: Quarterly

Drill Scenario: Entire Region Down

Objective: Recover entire application stack to secondary region in RTO target.

Prerequisites:

Secondary region already has quota
Terraform code ready (infrastructure-as-code)
Backup/snapshots replicated to secondary region
Team trained on runbooks

Drill Steps:

Phase 1: Declare Disaster (T=0)

Assume: us-central1 completely unavailable
├─ All GKE clusters down
├─ All databases inaccessible
├─ All storage in region inaccessible
└─ Recovery to us-east1 required

Phase 2: Infrastructure Recovery (T=0–30 min)

1. Verify secondary infrastructure ready:
   ├─ VPC exists in us-east1
   ├─ Subnets created
   ├─ Security groups configured
   └─ Quotas available

2. Create compute resources (via Terraform):
   ├─ GKE cluster us-east1 (3 nodes)
   └─ Database instance us-east1 (from backup)

3. Restore persistent volumes:
   ├─ Create disks from snapshots
   ├─ Attach to instances
   └─ Mount

Phase 3: Application Recovery (T=30–60 min)

1. Deploy applications (via GitOps):
   ├─ Deploy GitOps operator to new cluster
   ├─ GitOps pulls manifests from Git
   ├─ Workloads automatically deployed

2. Restore data:
   ├─ Volumes already mounted
   ├─ Database restored

3. Verify application health:
   ├─ Pods running (kubectl get pods)
   ├─ Services responding (curl service)
   ├─ Database queries work (SELECT test)

Phase 4: Traffic Failover (T=60–75 min)

1. Update DNS:
   ├─ Point cloud.example.com to secondary IP
   ├─ Lower TTL to 60 seconds

2. Verify clients routed correctly:
   ├─ DNS responds with secondary IP
   ├─ Clients connect to secondary service
   ├─ Application responses normal

3. Monitor secondary infrastructure:
   ├─ CPU, memory, network usage
   ├─ Error rate
   ├─ Latency

Phase 5: Validation (T=75–90 min)

1. Run smoke tests:
   ├─ Create test object, verify retrieves
   ├─ Run sample transactions
   ├─ Check business metrics

2. Measure RTO:
   ├─ Actual time from T=0 to fully operational = 90 min
   ├─ Expected RTO = 60 min
   ├─ Actual vs Expected: 90 min > 60 min (need improvement)

3. Identify bottlenecks:
   ├─ Terraform apply took 25 min (need optimization)
   ├─ Database restore took 20 min (snapshot size issue)
   ├─ GitOps deployment took 10 min (ok)

Phase 6: Post-Drill (T=90+)

1. Cleanup:
   ├─ Delete test infrastructure (cluster, database)
   ├─ Restore DNS to primary
   └─ Verify primary still working

2. Documentation:
   ├─ Actual RTO: 90 minutes (vs target 60 minutes)
   ├─ Issues found:
   │  ├─ Terraform apply slow, needs parallelization
   │  ├─ Database restore needed tuning
   │  └─ Team unfamiliar with runbook (needs training)
   ├─ Action items assigned to owners
   └─ Scheduled for next drill (3 months)

Chaos Engineering: Controlled Failure Injection

Level 1: Chaos at Resource Level

python

# Chaos Monkey: randomly kill pods in GKE

import random
import subprocess

def chaos_kill_random_pod(namespace='production'):
    # Get list of pods
    pods = subprocess.check_output(
        ['kubectl', 'get', 'pods', '-n', namespace, '-o', 'jsonpath={.items[*].metadata.name}']
    ).decode().split()
    
    # Kill random pod
    victim = random.choice(pods)
    print(f"Killing pod: {victim}")
    subprocess.run(['kubectl', 'delete', 'pod', victim, '-n', namespace])
    
    # Monitor recovery
    # Check: is pod restarted within 30 seconds?
    # Check: is application still responding?

# Schedule: hourly during business hours

Level 2: Chaos at Service Level

python

# Chaos: simulate database failover

def chaos_database_failover():
    # Simulate primary database down
    # 1. Close all connections to primary
    # 2. Switch application to read replica
    # 3. Monitor:
    #    ├─ Are queries still working?
    #    ├─ Is data consistency maintained?
    #    └─ What is user-facing impact?
    
    pass

Level 3: Chaos at Region Level

python

# Chaos: simulate entire region failure

def chaos_region_failure():
    # 1. Block all traffic to primary region
    # 2. Failover to secondary region (via DNS or load balancer)
    # 3. Monitor:
    #    ├─ Did failover happen automatically?
    #    ├─ How long did it take (RTO measurement)?
    #    └─ Did any data get lost (RPO measurement)?
    
    pass

Metrics: What to Measure

Recovery Metrics

RTO (Recovery Time Objective):

Definition: Time from disaster declaration to service fully operational
Measurement:
├─ T=0: Disaster declared (all primary infrastructure down)
├─ T=X: Service responding correctly from secondary region
├─ RTO = X minutes

Target vs Actual:
├─ Target: 60 minutes
├─ Actual (drill result): 90 minutes
├─ Action: optimize to meet target

RPO (Recovery Point Objective):

Definition: Amount of data lost
Measurement:
├─ T=0: Disaster occurs
├─ Latest backup: T=-30 min (30 minutes before disaster)
├─ Data lost: transactions from T=-30 to T=0 (30 minutes of transactions)
├─ RPO = 30 minutes

Verify:
├─ Check: last backup timestamp
├─ Check: replication lag at time of failure
└─ Calculate: data loss window

Infrastructure Metrics

Recovery speed by component:

Benchmark:
├─ Infrastructure creation (Terraform apply): < 10 min
├─ Database restore from snapshot: < 15 min
├─ GKE cluster creation: < 10 min
├─ Application deployment (GitOps): < 5 min
└─ Total: ~40 minutes (leaves buffer for issues)

Actual (from drill):
├─ Terraform: 25 min (5× slower, investigate)
├─ Database: 20 min (ok)
├─ GKE: 12 min (ok)
├─ GitOps: 8 min (ok)
└─ Total: 90 minutes (2.25× slower than target)

Common Failure Modes to Test

Failure Mode 1: Backup Doesn't Exist

Scenario: Attempt to restore from backup, but backup file corrupted/missing

Test:
1. Deliberately delete backup file
2. Attempt restore
3. Verify: error message clear
4. Verify: fallback procedure defined (manual recovery from logs?)

Mitigation:
├─ Backup validation (hourly: test snapshot accessibility)
├─ Multiple backup copies (never single copy)
└─ Alerts if backup not created

Failure Mode 2: Secondary Region Quota Exceeded

Scenario: Attempt to create resources in secondary region, but quota limit

Test:
1. Check current quota usage in secondary region
2. Assume worst case (fail to create 1000 machines)
3. Verify: quota increased or auto-approved
4. Verify: team aware of quota limitation

Mitigation:
├─ Pre-increase quota in secondary region (before needed)
├─ Auto-scale quota (if available)
└─ Monitor quota usage

Failure Mode 3: Configuration Drift

Scenario: Git repository has old configuration, actual service differs

Test:
1. Manually modify Kubernetes deployment in primary cluster
2. Trigger DR procedure
3. Restore to secondary from Git
4. Verify: secondary matches Git (not primary's manual changes)

Mitigation:
├─ Enforce GitOps (block manual changes)
├─ Regular drift detection (diff Git vs actual)
└─ Auto-sync to force convergence

Failure Mode 4: Secrets Missing

Scenario: Recover to secondary region, but secrets not accessible

Test:
1. Remove secret access from secondary region
2. Attempt application startup
3. Verify: clear error message
4. Verify: manual secret recovery procedure defined

Mitigation:
├─ Cross-region Secret Manager replication
├─ Test secret access in secondary region (monthly)
└─ Pre-stage secrets in secondary

Running Drills: Practical Execution

Pre-Drill Checklist (1 week before)

Communication:
├─ Announce drill to all teams
├─ Set expected downtime window (if applicable)
└─ Disable automatic alerts (prevent unnecessary paging)

Environment:
├─ Verify secondary infrastructure available
├─ Verify team access to runbooks
├─ Verify tools/scripts ready
└─ Verify backup/snapshot availability

Team:
├─ Designate drill commander (makes decisions)
├─ Brief team on objectives
├─ Review runbook (1–2 hours before)
└─ Assign roles (infrastructure, application, monitoring)

During Drill

Documentation:
├─ Start timer (T=0)
├─ Record every action taken
├─ Record every issue/blocker
├─ Record actual times for each phase

Monitoring:
├─ Watch secondary infrastructure provisioning
├─ Watch application coming online
├─ Watch error rates (should be high initially, then drop)
└─ Keep communication channel open (Slack/Zoom)

Decision making:
├─ Drill commander approves major decisions
├─ No shortcuts (execute as-is, find issues)
├─ Pause if actual issue found (ask: should we fix or document?)

Post-Drill Retrospective (within 1 day)

Review:
├─ RTO achieved: X minutes (vs target Y minutes)
├─ RPO achieved: X hours (vs target Y hours)
├─ Issues found: list of blockers/slowdowns
├─ Root causes: why was RTO not met?

Action items:
├─ Fix each issue (assign owner, deadline)
├─ Update runbook based on lessons learned
├─ Schedule next drill
└─ Share results with stakeholders

Anti-Patterns

Anti-pattern 1: "We Have a Disaster Recovery Plan, No Need to Test"

Symptom: DR documentation exists but untested.

Problem:

Procedures broken
Assumptions wrong
Team unfamiliar
Discover failures when actually needed

Right approach: Test quarterly, measure RTO, iterate.

Anti-pattern 2: "Drill Too Expensive, Do It Rarely"

Symptom: Drill happens once per year.

Problem:

Configuration changes between drills (broken by then)
Team forgets procedures
Issues discovered too late

Right approach: Partial tests monthly (backups, snapshots), full drill quarterly.

Anti-pattern 3: "Drill in Production at Peak Hours"

Symptom: Schedule drill during business hours without customer communication.

Problem:

Customers see failover (unexpected)
Support team overwhelmed
Actual incidents masked by drill traffic

Right approach: Drill in staging (realistic without affecting prod), or announce prod drill and monitor carefully.

Summary

DR testing validates that disaster recovery actually works. Three levels:

Configuration audit (weekly): Automated checks
Partial tests (monthly): Individual component restore
Full drill (quarterly): End-to-end recovery with RTO measurement

Measure RTO/RPO, identify bottlenecks, iterate.

DR Testing, Chaos Engineering, and Failure Validation ​

Tại Sao Điều Này Quan Trọng ​

Internal Model: Disaster Recovery Verification ​

Three Levels of DR Validation ​

Configuration Audit: Automated Checks ​

Backup Validation ​

Replica Lag Monitoring ​

Snapshot Accessibility Test ​

Partial Recovery Tests: Monthly Component Tests ​

Test 1: Database Restore ​

Test 2: GKE Backup Restore ​

Test 3: Snapshot Restore and Mount ​

Full Disaster Recovery Drill: Quarterly ​

Drill Scenario: Entire Region Down ​

Chaos Engineering: Controlled Failure Injection ​

Level 1: Chaos at Resource Level ​

Level 2: Chaos at Service Level ​

Level 3: Chaos at Region Level ​

Metrics: What to Measure ​

Recovery Metrics ​

Infrastructure Metrics ​

Common Failure Modes to Test ​

Failure Mode 1: Backup Doesn't Exist ​

Failure Mode 2: Secondary Region Quota Exceeded ​

Failure Mode 3: Configuration Drift ​

Failure Mode 4: Secrets Missing ​

Running Drills: Practical Execution ​

Pre-Drill Checklist (1 week before) ​

During Drill ​

Post-Drill Retrospective (within 1 day) ​

Anti-Patterns ​

Anti-pattern 1: "We Have a Disaster Recovery Plan, No Need to Test" ​

Anti-pattern 2: "Drill Too Expensive, Do It Rarely" ​

Anti-pattern 3: "Drill in Production at Peak Hours" ​

Summary ​

References ​

DR Testing, Chaos Engineering, and Failure Validation

Tại Sao Điều Này Quan Trọng

Internal Model: Disaster Recovery Verification

Three Levels of DR Validation

Configuration Audit: Automated Checks

Backup Validation

Replica Lag Monitoring

Snapshot Accessibility Test

Partial Recovery Tests: Monthly Component Tests

Test 1: Database Restore

Test 2: GKE Backup Restore

Test 3: Snapshot Restore and Mount

Full Disaster Recovery Drill: Quarterly

Drill Scenario: Entire Region Down

Chaos Engineering: Controlled Failure Injection

Level 1: Chaos at Resource Level

Level 2: Chaos at Service Level

Level 3: Chaos at Region Level

Metrics: What to Measure

Recovery Metrics

Infrastructure Metrics

Common Failure Modes to Test

Failure Mode 1: Backup Doesn't Exist

Failure Mode 2: Secondary Region Quota Exceeded

Failure Mode 3: Configuration Drift

Failure Mode 4: Secrets Missing

Running Drills: Practical Execution

Pre-Drill Checklist (1 week before)

During Drill

Post-Drill Retrospective (within 1 day)

Anti-Patterns

Anti-pattern 1: "We Have a Disaster Recovery Plan, No Need to Test"

Anti-pattern 2: "Drill Too Expensive, Do It Rarely"

Anti-pattern 3: "Drill in Production at Peak Hours"

Summary

References