My Awesome Project

Appearance

Backup Strategies for GKE: Concepts, Architecture, Restore Workflows ​

Tại Sao Điều Này Quan Trọng ​

GKE cluster chứa hai loại state hoàn toàn khác nhau: Kubernetes manifests (deployments, services, configmaps) và persistent data (databases, volumes). Disaster recovery phải bảo vệ cả hai, nhưng chúng cần những cơ chế khác nhau.

Backup for GKE giải quyết vấn đề: nó nắm bắt toàn bộ state của cluster (kubernetes resources + persistent volumes) tại một thời điểm, cho phép restore lên cluster khác hoặc recreate từ đầu. Không có proper backup strategy = nếu cluster bị hủy, mất tất cả workloads + data.

Internal Model: Backup for GKE Architecture ​

Components ​

BackupPlan: Template định nghĩa backup configuration.

  • Source cluster: cluster nào sẽ được backup
  • Scope: workloads nào backup (namespace selectors, label selectors)
  • Backup frequency: manual hay cron schedule
  • Retention policy: giữ bao nhiêu backups
  • Storage location: GCS bucket nào lưu backup

Backup: Actual snapshot tại một thời điểm.

  • Resource manifest (deployments, services, configmaps, secrets)
  • Persistent disk snapshots (mỗi PVC được snapshot)
  • Metadata: timestamp, size, completion status

RestorePlan: Template định nghĩa restore configuration.

  • Source backup plan: lấy từ backup nào
  • Target cluster: restore tới cluster nào
  • Namespace mapping: source namespace → target namespace (có thể khác)
  • Conflict handling: nếu resource đã tồn tại, thay thế hay skip
  • Transform rules: modify manifest trước khi restore (e.g., change image registry)

Restore: Execution của restore process.

  • Creates new resources từ backup manifests
  • Restores PVC data từ snapshots
  • Waits for workloads to become healthy

Backup Data Capture ​

Backup captures hai loại data:

1. Kubernetes Resources (YAML manifests)

Deployment
├─ spec (replicas, selector, template)
├─ status (current replicas, ready replicas)
└─ annotations (backup metadata)

PersistentVolumeClaim
├─ spec (size, storage class, access mode)
└─ status (phase, bound to PVC)

ConfigMap, Secret, Service, Ingress...

Backup save resources mỗi 5 phút (không phải real-time). Nếu resource deleted/modified giữa backup, changes sẽ bị mất.

2. Persistent Volume Data

Backup automatically tạo snapshots của mỗi PVC được mounted bởi workloads:

Workload: StatefulSet mysql-0
├─ mounted volume: pvc-mysql-data
│  └─ Backup tạo PD snapshot: backup-mysql-data-t1234567890

Workload: Deployment app
├─ mounted volume: pvc-app-logs
│  └─ Backup tạo PD snapshot: backup-app-logs-t1234567890

Snapshot tạo asynchronously — backup có thể complete trước snapshot hoàn tất. Backup status sẽ indicate waiting for snapshots.

Incremental vs Full Backups ​

Backup for GKE typically save incremental snapshots:

First backup:

  • Copy toàn bộ PVC data (100 GB) → single snapshot

Subsequent backups:

  • Copy chỉ changed blocks (vd, 5 GB mới) → incremental snapshot
  • Snapshot chain: backup2 depends on backup1 (depends on original)
T0: Full snapshot (100 GB data)
    └─ pvc-data-full-20250601T000000

T1: Incremental (+5 GB changed)
    └─ pvc-data-incr-20250601T010000
       (references backup-full-20250601T000000)

T2: Incremental (+3 GB changed)
    └─ pvc-data-incr-20250601T020000
       (references backup-incr-20250601T010000)

To restore: need all three snapshots in chain.
If delete backup-full or backup-incr-20250601T010000
    → cannot restore backup-incr-20250601T020000

Implication: Cannot delete old backups arbitrarily — must follow chain (delete newest first).

Cross-Region Backup Replication ​

By default, backups stored in GCS bucket (regional or multi-region). To achieve cross-region DR:

Option 1: Multi-region GCS bucket

  • Backup automatically replicated to multiple regions
  • But asynchronous (RPO = replication lag, typically 1 hour)

Option 2: Cross-region backup copy

  • Use Storage Transfer Service to copy backups to another region
  • Manual/scheduled copy (daily, weekly)

Option 3: Backup in cross-project

  • Enable cross-project backup (beta feature)
  • Backup created in separate project/region automatically
  • Provides isolation (production outage can't affect backup project)

Restore Workflow: Multi-Stage Process ​

Stage 1: Resource Manifest Restore ​

RestorePlan configured:
├─ Source backup: backup-20250601T100000
├─ Target cluster: us-east1-gke-cluster
├─ Namespace mapping: production → production-restored
└─ Conflict strategy: replace

Restore execution:
1. Read backup manifests from GCS
2. Apply namespace transformation
3. Apply conflict resolution
4. Create/update resources in target cluster
   ├─ Deployments
   ├─ ConfigMaps, Secrets
   ├─ Services
   └─ StorageClasses, PVCs (empty, waiting for volume restore)

Duration: Typically 2–10 minutes (depends on resource count).

Post-stage-1 state: Pods are pending (PVCs not bound to data yet).

Stage 2: Volume Snapshot Restore ​

For each PVC in backup:
1. Check snapshot existence (snapshot might not exist if backup incomplete)
2. Create new PD from snapshot:
   └─ new-pvc-data-disk (from snapshot-pvc-data-t1234567890)
3. Attach to new PVC (created in stage 1)
4. Kubernetes attaches disk to target node
5. Workload mounts volume

Duration: 5–30 minutes (depends on snapshot size).

Risk: If snapshot doesn't exist or is corrupted → restore fails, workload stuck pending.

Stage 3: Workload Stability ​

After volumes attached:
1. Init containers run (if defined)
2. Application starts
3. Readiness probes execute
4. Once healthy:
   └─ Pod marked as ready, load balancer includes in pool

If pod crashes or doesn't become ready:
├─ Kubectl shows error (CrashLoopBackOff, Pending, etc.)
└─ Manual investigation needed

Duration: 1–10 minutes (depends on app startup time).

After completion: Restore is complete, workload fully functional.

Full Restore Timeline Example ​

T0:00     Restore triggered
T0:05     Manifests applied, pods created (pending)
T0:15     Snapshots restoring (25% complete)
T0:20     Snapshots restoring (50% complete)
T0:30     Snapshots done, workload starting
T0:35     App initialization running
T0:42     Pod healthy, restore complete

Total restore time: 42 minutes
RTO achieved: 42 minutes (if backup plan prepared beforehand)

Backup Strategy Considerations ​

Scope: What to Include? ​

Include:

  • Namespace with critical workloads
  • Persistent volumes (especially stateful sets)
  • ConfigMaps (non-sensitive configuration)

Exclude:

  • Secrets (should backup separately with different isolation)
  • Transient pods (jobs, temporary debugging pods)
  • High-churn workloads (avoid cluttering backup)

Configuration:

yaml
backupPlan:
  backupConfig:
    includeSecrets: false  # backup secrets separately
    includedNamespaces:
    - production
    - legacy-app
    includedResources:
    - deployments
    - statefulsets
    - services
    - configmaps
    - pvcs

Frequency: Backup Schedule ​

Trade-off:

  • Frequent backups (every 1 hour) = RPO = 1 hour, but storage cost high, backup job creates load
  • Infrequent backups (daily) = RPO = 24 hours, low cost, but more data loss

Recommendation:

  • Critical stateful services: every 2–4 hours
  • Stateless services: daily
  • High-churn services: daily (incremental snapshots help)

Retention: How Long to Keep ​

Rule of thumb:

  • Keep 7–14 days for quick recovery from accidental deletion
  • Keep 1–2 backups/month for long-term archival (compliance)
Backup schedule: daily
├─ Keep 7 daily (production recovery)
├─ Keep 4 weekly (1 per week, ~4 weeks back)
└─ Keep 12 monthly (1 per month, 1 year)

Total storage: ~25 days worth of snapshots
Cost: ~100 GB (depends on PVC size)

Common Restore Scenarios ​

Scenario 1: Accidental Deletion of Workload ​

Situation: Engineer kubectl delete deployment --all in production (oops).

Recovery:

  1. Identify latest backup before deletion (< 1 hour ago if hourly backups)
  2. Create RestorePlan to restore to same cluster
  3. Use namespace mapping: production → production-temp-restore
  4. Restore manifests only (don't restore volumes, avoid rewriting PVCs)
  5. Once verified, merge resources back to production namespace
  6. Delete temp restore

Time: 15–30 minutes (manual work to verify/merge).

Scenario 2: Data Corruption in Stateful Service ​

Situation: Bug in application corrupted database, PVC data is corrupt.

Recovery:

  1. Identify last-known-good backup (before corruption started)
  2. Restore to new cluster in same region
  3. Point application to restored database (update connection string)
  4. Verify data integrity
  5. Keep original cluster for comparison (post-mortem)

Time: 45 minutes — 2 hours (depending on volume size, verification effort).

Scenario 3: Complete Region Failure ​

Situation: us-central1 region completely down (network, infrastructure failed).

Recovery:

  1. Wait for alert that region is down (2–5 min)
  2. Initiate restore to secondary region (us-east1)
  3. Restore includes manifests + snapshots (cross-region copy already done)
  4. Point external DNS to secondary cluster
  5. Verify all workloads healthy in secondary

Time: RTO = detection (5 min) + restore execution (45 min) = 50 min total.

Prerequisites:

  • Backup must be stored in multi-region bucket or copied to secondary region daily
  • Secondary region must have quota (GKE cluster created, resource limits available)
  • DNS must point to secondary cluster (prepared beforehand)

Performance Impact: Cost of Backup ​

Snapshot Creation Overhead ​

During backup:

  • Snapshot creation on PD = network bandwidth used for copying changed blocks
  • Large PVC (500 GB) = 500 GB × 2 copy (local + snapshot) = 1 TB network I/O
  • Typical snapshot duration: 30–60 minutes for large volumes

Impact on production:

  • If workload writing heavily during backup = increased I/O latency
  • Network throughput shared = other traffic might slow down

Mitigation:

  • Schedule backups during low-traffic hours
  • Use incremental snapshots (only changed blocks)
  • Distribute backups across cluster (don't backup all workloads simultaneously)

Storage Cost ​

Per backup:

  • Manifests: negligible (few MB)
  • Snapshots: proportional to PVC size
    • Full snapshot: 100 GB PVC = 100 GB storage cost
    • Incremental: only changed blocks (depends on workload)

Example:

  • 10 PVCs, each 100 GB, daily backups
  • Day 1: full snapshot = 1 TB
  • Day 2–30: incremental = 50 GB/day average (changed data)
  • 30 days total: 1 TB + (29 × 50 GB) = 2.45 TB
  • Cost: 2.45 TB × $0.026/GB/month = ~$60/month

Performance Tuning ​

Backup schedule optimization:

Stateless services: daily backup (low cost)
├─ Deployments, services, configmaps (few GB manifests)

Stateful services: every 4 hours (catch data loss early)
├─ Snapshots incremental (only changed data)

High-churn services (logging, metrics): daily
├─ Snapshots might be large if heavy write workload
├─ Consider separate retention (delete after 7 days)

Disaster Scenario: Multi-Component Failure ​

Situation: Cluster fails + backup region fails + secrets not replicated.

Failure chain:

  1. Primary cluster infrastructure failure (nodes dead, etcd corrupted)
  2. Attempt restore from backup in secondary region
  3. Restore succeeds, but database service can't authenticate (secrets missing)
  4. Application can't start (environment variables missing)

Why it failed:

  • Backup excluded secrets (security concern, secrets should backup separately)
  • Secondary region backup of secrets not done
  • No process to manage secrets in restore scenario

Prevention:

  • Backup secrets to separate, isolated location (Secret Manager cross-project)
  • Document secret recovery procedure separately from backup restore
  • Test restore with full secret replication (quarterly drill)

Validation: Ensuring Backups Work ​

Test 1: Backup Completion ​

Every backup, check:
1. Backup status = "Succeeded"
2. Resource count matches expected
3. Snapshot count matches PVC count
4. Backup size reasonable (not too small, not too large)

Alert if any check fails.

Test 2: Snapshot Accessibility ​

Once per week:
1. Pick random snapshot from backup
2. Try to create test PD from snapshot
3. Attach to test node, mount, verify data readable
4. Delete test PD
5. If fails → backup is corrupted

Test 3: Restore Drill (Quarterly) ​

Once per quarter (or after major cluster changes):
1. Pick latest backup
2. Create test cluster in secondary region
3. Restore full backup (manifests + volumes)
4. Verify all workloads become healthy
5. Run smoke tests
6. Delete test cluster

Measure: actual RTO achieved vs target RTO.
If actual > target → need optimization.

Anti-Patterns ​

Anti-pattern 1: "Backup Everything, No Cleanup" ​

Symptom: Backup all namespaces, all resources, never delete old backups.

Problem:

  • Storage cost balloons (100+ GB/month for large cluster)
  • Backup windows get longer (more to backup)
  • Snapshot chain becomes complex (can't delete old snapshots safely)
  • Search/find backup from specific date becomes difficult

Right approach: Define clear backup scope, retention policy, and cleanup schedule.

Anti-pattern 2: "We Have Backups, We Don't Need Snapshots" ​

Symptom: Team creates daily backups but skips PVC snapshots (snapshot disabled in backup plan).

Problem:

  • Restore rebuilds PVCs empty
  • Application data lost
  • "Backup" is only manifest, not data

Right approach: Backup must include both manifests AND snapshots. Verify by testing restore.

Anti-pattern 3: "Restore Test Not Needed, It Will Work" ​

Symptom: Team creates backup plan, assumes restore will work, never test.

Problem:

  • When disaster strikes, restore fails (snapshot corrupted, permission issue, quota exceeded)
  • Team discovers backup useless under pressure
  • Application down longer than necessary

Right approach: Test restore quarterly, at minimum. Update runbook with actual timings from test.

Summary ​

Backup for GKE protects both Kubernetes manifests and persistent volumes. Restoration is multi-stage (manifests → volumes → application startup) and takes 30–60 minutes typically.

Key decisions:

  • Backup scope: What resources, what namespaces?
  • Backup frequency: Based on RPO tolerance
  • Backup retention: Balance cost vs compliance requirement
  • Cross-region: Where to store backup for DR?

Must validate with regular restore drills.

References ​