Persistent Disk Snapshots & Cross-Region Replication

Tại Sao Điều Này Quan Trọng

Persistent Disk (PD) là storage block-level trên GCP, cung cấp durability (11 nines) nhưng không cross-region theo mặc định. Toàn bộ PD data nằm trong một region. Nếu region down, PD inaccessible. Snapshots là cách duy nhất để backup PD data để cross-region recovery.

Hiểu snapshot mechanics quan trọng vì nó quyết định:

RPO (replication lag nếu async)
RTO (time to restore từ snapshot)
Storage cost (incremental vs full)
Snapshot scheduling (when to take, how often)

Internal Model: How Snapshots Work

Copy-on-Write Mechanism

Snapshot không copy toàn bộ 500 GB data ngay. Nó sử dụng copy-on-write (CoW):

Initial state:
┌──────────────────────────┐
│   Persistent Disk        │
│   500 GB of data blocks  │
│   (all in region A)      │
└──────────────────────────┘

T0: Create snapshot
Snapshot creation:
┌─────────────────────────────┐
│ Snapshot 1 (reference only) │
│ "Points to" original blocks │
│ No data copied yet          │
└─────────────────────────────┘

T1: Application writes to block 10
Before write:
┌──────────────────┐    ┌──────────────────┐
│ Original blocks  │←───│ Snapshot 1 still │
│ (all 500 GB)     │    │ points here      │
└──────────────────┘    └──────────────────┘

CoW triggered:
1. Copy block 10 to snapshot storage
2. Update block 10 on PD (original)
3. Snapshot still references old block 10 copy

Result:
┌─────────────────────────────┐
│ Original PD (block 10 new)  │
└─────────────────────────────┘
               ↓ (points to old state)
┌─────────────────────────────┐
│ Snapshot 1 (block 10 old)   │
└─────────────────────────────┘

Advantage: No immediate copy of 500 GB; snapshot creation is instant.

Cost: Only pay for changed blocks. If 5% of data changed = 25 GB snapshot storage.

Snapshot Chains and Incremental Snapshots

Multiple snapshots from same PD create a chain:

Full snapshot (T0):
└─ Snapshot-1-full: all 500 GB (100% of data)

Incremental snapshot (T1, 1 hour later):
└─ Snapshot-2-incremental:
   References Snapshot-1-full
   Only 20 GB changed data

Incremental snapshot (T2, 2 hours later):
└─ Snapshot-3-incremental:
   References Snapshot-2-incremental
   Only 15 GB changed data

To restore Snapshot-3, need entire chain:
Snapshot-1-full → Snapshot-2-incremental → Snapshot-3-incremental
(if delete Snapshot-1 or Snapshot-2, can't restore Snapshot-3)

Data layout:

Snapshot-1: 500 GB (full)
Snapshot-2: 20 GB (incremental, only new/changed)
Snapshot-3: 15 GB (incremental, only new/changed)
Total storage: 535 GB (not 1500 GB)

Deletion rules:

Can delete Snapshot-3 immediately (newest)
Cannot delete Snapshot-2 if Snapshot-3 exists (would break chain)
Must delete in reverse order (newest first)

Snapshot Restoration

Create disk from snapshot:

gcloud compute disks create restored-disk \
  --source-snapshot=snapshot-3-incremental \
  --zone=us-east1-b

Process:

Allocate new disk (500 GB)
Start restoring from snapshot (CoW reversed)
Blocks fetched on-demand from snapshot (lazy restore)

Duration: ~5–15 minutes for blocks to be accessible (not all blocks restored yet).

Full restore: Blocks restore on-demand (first read of block causes fetch). Can take hours if all data accessed.

Performance: First hour, I/O latency high (blocks being fetched). After warm-up, normal.

Cross-Region Snapshot Replication

Async Replication: One-Way Copy

By default, snapshots are regional (stored in one region only).

To protect against region failure, manually copy snapshots:

Region A (us-central1)          Region B (us-east1)
┌──────────────────────┐       ┌──────────────────────┐
│ Persistent Disk      │       │ Snapshot copy        │
│ 500 GB               │       │ (from region A)      │
└──────────────────────┘       └──────────────────────┘
         ↓
   Take snapshot
   Snapshot-1
         ↓
   Copy to Region B
   (asynchronous, 5-30 min)
         ↓
   Snapshot-1-copy arrives in B
         ↓
   Can restore in region B

RPO: Replication lag between regions:

Same continent (us-central1 to us-east1): 5–10 min
Cross-continent (us to asia): 30–60 min

Methods to replicate:

Manual script:

bash

# Take snapshot in region A
gcloud compute disks-resources snapshots create snap-$DATE \
  --source-disk=prod-disk \
  --zone=us-central1-a

# Copy to region B (GCS intermediate)
gcloud compute snapshots export gs://backup-bucket/snap-$DATE \
  --snapshot=snap-$DATE

gcloud compute snapshots import snap-$DATE-restored \
  --source-file=gs://backup-bucket/snap-$DATE

Duration: 10–30 min per snapshot.

Persistent Disk Asynchronous Replication: Available for Premium Persistent Disks only.

gcloud compute disks create prod-disk-async-replica \
  --source-disk=prod-disk \
  --replication-type=async-replica \
  --replica-zones=us-east1-b

Mechanism:

Primary PD in us-central1 (accepts writes)
Replica PD in us-east1 (async, ~1 min replication lag)
RPO: ~1 minute target
Can promote replica to primary if region fails

Cost: 1.2× storage cost (paying for both primary + replica).

Limitation: Only works for Premium Persistent Disks (not Standard).

GCS-Based Backup (Most Common)

Store snapshots in GCS bucket (multi-region if needed):

Region A (us-central1)
┌──────────────────────┐
│ Persistent Disk      │
└──────────────────────┘
       ↓
   Create snapshot
   Snapshot-1
       ↓
   Export to GCS (snapshot → image format)
   gs://dr-backup-bucket/disk-image-20250601.tar.gz
       ↓
   (if GCS is multi-region, auto-replicated)
       ↓
   In Region B or C:
   Import from GCS
   └─ Create disk from image

Duration: 15–30 min (depends on disk size, network bandwidth).

Storage: GCS multi-region cheaper than PD snapshots (long-term archival).

Trade-off: Slower restore (import process) vs cheaper storage.

Scheduling Snapshots for DR

Decision: Full vs Incremental

Full snapshot (weekly):

Copy 100% of data
Slow but independent (not dependent on previous snapshots)
Duration: 30–60 min
Use case: baseline backup, disaster recovery archival

Incremental (daily or more frequent):

Copy only changed blocks
Fast and cheap
Duration: 5–15 min
Use case: recovery from recent data loss

Recommended schedule:

Weekly (Sundays 2 AM):
└─ Full snapshot (independent baseline)

Daily (2 AM—6 AM window):
└─ Incremental snapshots (Monday—Saturday)

Result:
  Week 1:
  ├─ Full snapshot (all 500 GB)
  ├─ Incremental Day 1 (20 GB changed)
  ├─ Incremental Day 2 (18 GB changed)
  ├─ Incremental Day 3 (22 GB changed)
  ├─ Incremental Day 4 (15 GB changed)
  ├─ Incremental Day 5 (19 GB changed)
  └─ Incremental Day 6 (21 GB changed)

Total storage: 500 + (20+18+22+15+19+21) = 615 GB per week
Cost: ~$15/month for full week of backups

Snapshot Retention Policy

Rule of thumb:

Keep 7 daily (fast recovery from accidental deletion)
Keep 4 weekly (last month's snapshots)
Keep 1 monthly (archive for compliance)

Retention example:
Daily snapshots:
├─ Today (Day 0)
├─ Day -1
├─ Day -2
├─ Day -3
├─ Day -4
├─ Day -5
└─ Day -6 (auto-delete Day -7)

Weekly snapshots (keep Sundays):
├─ Last Sunday (this week)
├─ Sunday -1 week
├─ Sunday -2 weeks
├─ Sunday -3 weeks
└─ Sunday -4 weeks (auto-delete older)

Monthly snapshots (keep 1st of month):
├─ June 1
├─ May 1
├─ April 1
├─ March 1
├─ February 1
└─ January 1 (auto-delete older)

Total backups kept: ~15 snapshots
Total storage: ~700 GB

Automation: Snapshot Scheduling

Use Cloud Scheduler + Cloud Functions:

python

# Cloud Function triggered daily at 2 AM
import google.cloud.compute_v1 as compute

def create_snapshot(request):
    compute_client = compute.DisksClient()
    
    # List all prod disks
    disks = compute_client.list(
        project=PROJECT_ID,
        zone=ZONE,
        filter='labels.backup=true'  # only disks with backup label
    )
    
    # Create snapshot for each
    for disk in disks:
        snapshot = compute.Snapshot()
        snapshot.name = f"{disk.name}-snap-{date}"
        snapshot.source_disk = disk.self_link
        
        # Schedule as incremental
        operation = compute_client.create_snapshot(
            project=PROJECT_ID,
            resource=snapshot
        )
    
    return {'status': 'snapshots scheduled'}

# Cleanup old snapshots (keep 7 days)
def cleanup_old_snapshots(request):
    compute_client = compute.SnapshotsClient()
    snapshots = compute_client.list(project=PROJECT_ID)
    
    cutoff_date = datetime.now() - timedelta(days=7)
    
    for snap in snapshots:
        snap_date = datetime.fromisoformat(snap.creation_timestamp)
        if snap_date < cutoff_date and 'weekly' not in snap.name:
            compute_client.delete(project=PROJECT_ID, resource=snap.name)
    
    return {'deleted': count}

Deployed as:

Cloud Scheduler (Daily 2 AM UTC)
├─ Trigger Cloud Function: create-snapshot
└─ Trigger Cloud Function: cleanup-old-snapshots

Restore from Snapshot: Step-by-Step

Scenario: PD Corrupted, Need to Restore from Yesterday's Snapshot

Prerequisites:

Know which snapshot to restore (yesterday's incremental or last full)
Know which zone to restore to (can be different zone/region)
Know disk size requirement (must match or be larger)

Steps:

Find snapshot:

bash

gcloud compute snapshots list \
  --filter='name:prod-disk' \
  --sort-by=~creation_timestamp | head -10
# Output:
# prod-disk-snap-20250601-incr  2025-06-01T02:00:00Z
# prod-disk-snap-20250531-full  2025-05-31T23:00:00Z

Create disk from snapshot:

bash

gcloud compute disks create prod-disk-restored \
  --source-snapshot=prod-disk-snap-20250601-incr \
  --zone=us-central1-a \
  --type=pd-ssd

Duration: 5 min (disk created, ready to use).

But data isn't fully populated yet (lazy restore).

Attach to instance for data verification:

bash

gcloud compute instances attach-disk corrupted-instance \
  --disk=prod-disk-restored \
  --zone=us-central1-a

# SSH to instance
gcloud compute ssh corrupted-instance --zone=us-central1-a

# Mount the disk
sudo lsblk  # find disk device (e.g., /dev/sdb)
sudo mkdir -p /mnt/restored
sudo mount /dev/sdb1 /mnt/restored

# Verify data
ls -la /mnt/restored/

If data OK, swap original:

bash

# Detach corrupted disk
gcloud compute instances detach-disk corrupted-instance \
  --disk=prod-disk

# Detach restored disk from verification instance
gcloud compute instances detach-disk verification-instance \
  --disk=prod-disk-restored

# Rename restored disk to original name
gcloud compute disks delete prod-disk  # or keep as backup
gcloud compute disks rename prod-disk-restored \
  --new-name=prod-disk

# Reattach to production instance
gcloud compute instances attach-disk production-instance \
  --disk=prod-disk

Verify application:

Check application logs, metrics:
├─ Application startup time
├─ Database integrity checks (FSCK, consistency checks)
├─ Business logic verification

Total restoration time: 15–45 min (depending on disk size, verification effort).

Cross-Region Failover: Using Replicated Snapshots

Scenario: Entire Region Down

Situation: us-central1 region down, need to restore in us-east1.

Prerequisites:

Snapshots replicated to us-east1 (daily copy)
Target zone has capacity (quota for new disks)
Application can restart in new region (DNS configured)

Recovery steps:

List available snapshots in us-east1:

bash

# Snapshots should be already copied there
gcloud compute snapshots list --filter='name:prod-disk'

Create disk in us-east1 from replicated snapshot:

bash

gcloud compute disks create prod-disk-us-east \
  --source-snapshot=prod-disk-snap-20250601-incr \
  --zone=us-east1-b \
  --type=pd-ssd

Mount to warm-up cache: Some organizations pre-attach disk to warm-up node (populate local cache):

bash

gcloud compute instances attach-disk warm-up-instance \
  --disk=prod-disk-us-east \
  --zone=us-east1-b

# On instance: read entire disk to populate cache
dd if=/dev/sdb of=/dev/null bs=1M

# Then detach
gcloud compute instances detach-disk warm-up-instance \
  --disk=prod-disk-us-east

Application startup: Restart application with disk attached.

Total RTO: Detection (5 min) + snapshot copy (if not pre-replicated, 30 min) + disk creation (5 min) + application startup (10 min) = 50–55 min.

If snapshots pre-replicated: RTO = 20–30 min.

Performance Considerations

Snapshot Metadata Operations

List/describe snapshots is slow if many snapshots:

bash

# This is slow if you have 10K+ snapshots
gcloud compute snapshots list

# Better: filter by resource
gcloud compute snapshots list --filter='name:prod-disk'

Recommendation: Use naming convention (e.g., prod-disk-snap-<date>-<type>) for easy filtering.

Restore Performance

Initial restore phase:

5 min: disk created, ready to attach
Lazy restore starts when first blocks accessed
First app startup will trigger I/O (slow)

Warm-up phase:

Keep disks lightly used for 30–60 min after restore
Allows GCP backend to populate local SSD caches
After warm-up, performance returns to normal

Anti-pattern: Restore disk, immediately push prod traffic.

First hour will see high latency
Better: warm-up first, then failover

Anti-Patterns

Anti-pattern 1: "Only Snapshot When Disaster Happens"

Symptom: Team takes snapshot manually only when they need to recover.

Problem:

Delay in recovery (snapshot takes 30–60 min)
Application down longer than necessary
Under stress, mistakes happen (wrong snapshot, wrong zone)

Right approach: Automate daily/weekly snapshots. Test restores quarterly.

Anti-pattern 2: "Incremental Snapshot Chains Get Too Long"

Symptom: Taking incremental snapshots daily for 6 months, chain becomes 180 snapshots.

Problem:

Restoration becomes slow (must process entire chain)
Cannot delete individual snapshots (chain breaks)
Management complex

Right approach: Re-baseline every 4 weeks (take full snapshot), restart incremental chain.

Anti-pattern 3: "Forgot to Copy Snapshots to Secondary Region"

Symptom: Snapshots exist in primary region but not copied to secondary.

Problem:

Region disaster → snapshots inaccessible
No cross-region recovery possible
Discover problem under disaster pressure

Right approach: Automate cross-region snapshot copy (daily). Verify copy exists via automated test.

Summary

Persistent Disk snapshots use copy-on-write for efficient backup. Incremental snapshots reduce cost but create chains (must delete in order). Cross-region replication achieves DR via manual copy or Async Replication (Premium disks only).

Key points:

CoW mechanism = instant snapshot creation, pay only for changed blocks
Incremental snapshots = weekly full + daily incremental reduces storage cost
Cross-region replication = manual or Async Replication for DR
Restore = 15–45 min typical (disk creation + warm-up)
Automate = Cloud Scheduler + Cloud Functions for daily snapshots + cleanup

Persistent Disk Snapshots & Cross-Region Replication ​

Tại Sao Điều Này Quan Trọng ​

Internal Model: How Snapshots Work ​

Copy-on-Write Mechanism ​

Snapshot Chains and Incremental Snapshots ​

Snapshot Restoration ​

Cross-Region Snapshot Replication ​

Async Replication: One-Way Copy ​

GCS-Based Backup (Most Common) ​

Scheduling Snapshots for DR ​

Decision: Full vs Incremental ​

Snapshot Retention Policy ​

Automation: Snapshot Scheduling ​

Restore from Snapshot: Step-by-Step ​

Scenario: PD Corrupted, Need to Restore from Yesterday's Snapshot ​

Cross-Region Failover: Using Replicated Snapshots ​

Scenario: Entire Region Down ​

Performance Considerations ​

Snapshot Metadata Operations ​

Restore Performance ​

Anti-Patterns ​

Anti-pattern 1: "Only Snapshot When Disaster Happens" ​

Anti-pattern 2: "Incremental Snapshot Chains Get Too Long" ​

Anti-pattern 3: "Forgot to Copy Snapshots to Secondary Region" ​

Summary ​

References ​

Persistent Disk Snapshots & Cross-Region Replication

Tại Sao Điều Này Quan Trọng

Internal Model: How Snapshots Work

Copy-on-Write Mechanism

Snapshot Chains and Incremental Snapshots

Snapshot Restoration

Cross-Region Snapshot Replication

Async Replication: One-Way Copy

GCS-Based Backup (Most Common)

Scheduling Snapshots for DR

Decision: Full vs Incremental

Snapshot Retention Policy

Automation: Snapshot Scheduling

Restore from Snapshot: Step-by-Step

Scenario: PD Corrupted, Need to Restore from Yesterday's Snapshot

Cross-Region Failover: Using Replicated Snapshots

Scenario: Entire Region Down

Performance Considerations

Snapshot Metadata Operations

Restore Performance

Anti-Patterns

Anti-pattern 1: "Only Snapshot When Disaster Happens"

Anti-pattern 2: "Incremental Snapshot Chains Get Too Long"

Anti-pattern 3: "Forgot to Copy Snapshots to Secondary Region"

Summary

References