My Awesome Project

Appearance

Cloud Storage Geo-Redundancy: Multi-Region, Dual-Region, Turbo Replication ​

Tại Sao Điều Này Quan Trọng ​

Cloud Storage là primary data store cho mọi pipeline: backups, ML training data, analytics, logs. Nếu region down, GCS bucket vẫn tồn tại nhưng không accessible (regional bucket) hoặc dữ liệu chưa được replicate (replication lag). Geo-redundancy là duy nhất cách để đảm bảo data accessible từ bất cứ region nào, bất kể disaster.

GCS offers ba loại geographic distribution, mỗi cái với trade-off khác nhau về cost, latency, consistency.

Internal Model: GCS Replication Mechanisms ​

Regional Bucket: Zero Cross-Region Redundancy ​

Geography:

us-central1 (single region)
├─ Zone A: data replicas
├─ Zone B: data replicas
└─ Zone C: data replicas (asynchronous)

Replication:

  • Data replicates across zones within same region (synchronous)
  • Zone failure = data still accessible from other zones
  • Region failure = data inaccessible (no cross-region backup)

Guarantees:

  • Availability: 99.95% SLA (region-wide outage excluded)
  • Durability: 99.999999999% (11 nines)

Consistency:

  • Strong consistency (reads return latest write)
  • Metadata changes visible immediately

Cost: Baseline (cheapest).

Use case: Non-critical data, dev/test environments.

Dual-Region Bucket: Asynchronous Cross-Region Replication ​

Geography:

Regions: us-central1 (primary) + us-east1 (secondary)

us-central1:
├─ Zone A: data replicas (write lands here first)
├─ Zone B: data replicas
└─ Zone C: data replicas

us-east1:
├─ Zone D: data replicas (asynchronous, ~1 hour RPO)
├─ Zone E: data replicas
└─ Zone F: data replicas

Replication flow:

T=0:00  Write received, stored in us-central1 (zones A, B, C)
        Write confirmed to client immediately
T=0:05  Replication to us-east1 begins
T=0:30  Partial replication (99% of data copied)
T=1:00  Full replication complete, data consistent
        Client can read from us-east1 now

Guarantees:

  • RPO: target of 1 hour (100% of objects within 12 hours)
  • Availability: 99.95% SLA (survives single region failure)
  • Durability: 99.999999999% (same as regional)

Consistency:

  • Strong consistency within region (us-central1)
  • Eventual consistency across regions (1 hour window)

Example:

T0: Write to dual-region bucket (us-central1 + us-east1)
├─ Write lands in us-central1 (0ms)
├─ Read from us-central1 (0ms) → sees new data
├─ Read from us-east1 (2ms) → might not see new data yet
└─ (replication ongoing, user might see stale data in us-east1)

T1: After 1 hour
├─ Write replicated to us-east1
└─ Both regions now consistent

Cost: 1.02× regional cost (small replication overhead).

Limitation: Predefined region pairs (e.g., us-central1 + us-east1 only, cannot choose arbitrary regions).

Multi-Region Bucket: Wide Distribution, Lower Cost ​

Geography:

Continent: North America (GCP chooses exact regions)

Region 1 (us-central or us-east, varies):
├─ Zone A: data replicas
└─ Zone B: data replicas

Region 2 (us-west or other, varies):
├─ Zone C: data replicas
└─ Zone D: data replicas

Region 3 (optional, >100 miles apart):
├─ Zone E: data replicas
└─ Zone F: data replicas

Replication flow:

T=0:00  Write lands in first region
        Write confirmed immediately
T=0:10  Replication to other regions starts
T=1:00  Target: 99.9% of objects replicated
T=12:00 Guarantee: 100% replicated

Guarantees:

  • RPO: target of 1 hour, guaranteed 12 hours
  • Availability: 99.95% SLA
  • Durability: 99.999999999%

Consistency:

  • Metadata synchronous across regions (fast)
  • Data asynchronous (eventual consistency, 1–12 hour window)

Cost: Cheaper than dual-region (0.99× regional cost or less, because Google distributes load).

Trade-off: You don't choose regions (GCP manages distribution). Less predictable for low-latency read from specific region.

Turbo Replication: Premium Fast Replication ​

Available for: Dual-region buckets only.

Feature: Guarantees faster replication:

  • RPO: 15 minutes (100% of new objects within 15 min)
  • vs regular dual-region (1 hour target)

Mechanism:

  • Uses premium replication infrastructure
  • Guarantees backed by SLA

Cost: Additional charge (~$0.04/GB ingested for first copy) + higher replication cost.

Use case: Disaster recovery where 1-hour RPO too risky (15-min RPO acceptable).

Example cost:

  • Regular dual-region: 100 GB/month = $2 storage
  • Turbo replication: 100 GB/month = $2 storage + $4 turbo cost = $6 total

Cross-Bucket Replication: Manual Control via Transfer Service ​

For fine-grained replication control, use Storage Transfer Service:

Setup:

Source bucket (us-central1)
         ↓
    Transfer Service
         ↓
Destination bucket (us-east1)

Configuration:

TransferJob:
├─ Source: gs://prod-data-us-central1
├─ Destination: gs://prod-data-us-east1
├─ Schedule: daily 2 AM UTC
├─ Sync: delete files from destination if deleted from source
├─ Filters: only .data files, skip logs/
└─ Metadata: copy permissions, timestamps

Advantages:

  • Manual control over replication schedule
  • Can replicate specific objects (not entire bucket)
  • Can sync or one-way copy
  • Can replicate from Cloud Storage to GCS bucket in different project

Duration: Depends on data size:

  • 100 GB: 30–60 min
  • 1 TB: 2–4 hours
  • 10 TB: 12–24 hours

RPO: Based on schedule (daily = max 24-hour RPO).

Cost: Storage + transfer cost (per GB transferred).

Disaster Scenarios: Recovery from Region Failure ​

Scenario 1: Regional Bucket, Region Down ​

Situation: us-central1 goes down (network failed, infrastructure failed).

Data status:

  • Data in us-central1 inaccessible
  • No other region has the data

Recovery options:

  1. Wait for region to come back online (GCP handles infrastructure recovery)
  2. If permanent, data is lost (no backup elsewhere)

Prevention:

  • Don't use regional bucket for critical data
  • Use dual-region or multi-region instead

Time to recovery: Depends on GCP infrastructure team (typically 2–8 hours).

Scenario 2: Dual-Region Bucket, Primary Region Down ​

Situation: us-central1 down. Bucket is dual-region (us-central1 + us-east1).

Data status:

  • Write landed in us-central1 at T=0:00
  • Replication to us-east1 started at T=0:05
  • At T=0:15, primary region fails
  • Replication 50% complete (50 GB replicated, 50 GB not)

Recovery:

  1. Data in us-east1 (50 GB) is accessible
  2. Data replicated is complete (consistency in us-east1)
  3. Data not replicated (50 GB) is lost (RPO = 15 min)

Mitigation:

  • Replication started asynchronously, wasn't fast enough
  • To prevent: use Turbo Replication (15-min RPO guaranteed)

Scenario 3: Multi-Region Bucket with Cross-Bucket Backup ​

Situation: Multi-region bucket exists. Critical data also backed up to secondary bucket via Transfer Service.

Architecture:

Primary: multi-region (us)
├─ gs://prod-data-multiregion
└─ Data accessible from any region

Backup: regional (us-east1)
├─ gs://prod-data-backup-us-east1
└─ Updated daily (24-hour RPO)

During primary region failure:

  1. Primary bucket still accessible (multi-region, replicated to other regions)
  2. Backup bucket accessible from us-east1
  3. Use primary if replication complete, use backup if not

RTO: Near-zero (immediate failover to other region in multi-region bucket).

RPO: 1 hour (multi-region RPO target).

Consistency Edge Cases ​

Edge Case 1: Read After Write in Dual-Region ​

Application writes object: gs://bucket/file.txt (1 MB)

T0:00  Write confirmed to client
       Data in us-central1 (primary)
       Replication to us-east1 scheduled

Read from us-east1 at T0:05 (5 min after write):
├─ Read hits us-east1
├─ File exists? Maybe, maybe not (50% chance, depends on replication speed)
├─ If exists: size might be 0 (partial replication)
└─ If not: 404 Not Found

Client application expects: read returns either new data or old data, not partial data
Reality: GCS might return partial data (consistency level = eventual)

Mitigation:
1. Always read from primary region (us-central1) if you need strong consistency
2. Accept eventual consistency (retry read until successful)
3. Use Turbo Replication (guarantees 15-min consistency)

Edge Case 2: Delete After Write in Dual-Region ​

T0:00  Write object X
       Data in us-central1

T0:10  Delete object X
       Delete confirmed to client
       Object deleted from us-central1

T0:15  Replication catches up
       But object was deleted before replication
       Result: object not replicated to us-east1

T0:20  Read from us-east1
       Object doesn't exist in us-east1
       (was never replicated, then deleted)
       Read returns 404 (consistent with primary)

This is correct behavior (delete is authoritative).

Edge Case 3: Concurrent Writes in Multi-Region ​

Two clients write to same object simultaneously:

Client A (us-central1): PUT file.txt (content: "version1")
Client B (us-east1):    PUT file.txt (content: "version2")

Both writes reach different regions:
├─ us-central1 sees version1 first, stores
├─ us-east1 sees version2 first, stores

During replication:
├─ version1 reaches us-east1 (but us-east1 already has version2)
├─ version2 reaches us-central1 (but us-central1 already has version1)

Question: which version is final?
Answer: GCS doesn't guarantee (depends on write order, timing)
        Eventually, one will win (last-write-wins based on timestamp)
        But during window, different regions might have different versions

Mitigation: Multi-region bucket not suitable for concurrent writes to same object. Use single-region bucket + application-level coordination.

Production DR Patterns ​

Pattern 1: Regular Multi-Region + Scheduled Backup ​

Production: multi-region (us) bucket
├─ Serves application reads/writes
├─ Auto-replicated across regions
├─ RPO = 1 hour (eventual consistency)

Backup: cross-project, regional (europe) bucket
├─ Daily copy via Transfer Service
├─ Isolated from production (separate project)
├─ RPO = 24 hours
└─ Preserved for compliance, disaster recovery

Failover sequence:
1. Region failure → GCS still accessible (multi-region)
2. Application re-routes to nearest accessible region
3. Data loss: max 1 hour (since last replication)
4. If primary project attacked/deleted:
   └─ Restore from backup bucket in separate project
   └─ Restore time: 2–4 hours (depends on size)

Pattern 2: Dual-Region + Turbo Replication for Critical Data ​

Critical data: dual-region (us) + Turbo Replication
├─ Geo-redundant storage
├─ Guaranteed 15-min RPO (Turbo backed by SLA)
├─ Cost: ~2× regional storage

Non-critical data: multi-region (us)
├─ Geo-redundant storage
├─ RPO = 1 hour acceptable
├─ Cost: ~0.99× regional storage

Application logic:
├─ Critical paths: read/write to dual-region bucket
├─ Non-critical: read/write to multi-region bucket

Cost Analysis ​

Storage cost per GB/month (us pricing, 2025):

  • Regional: $0.020
  • Dual-region: $0.020 (same)
  • Multi-region: $0.019 (cheaper!)

Transfer cost per GB (replication):

  • Intra-region: $0
  • Inter-region (within continent): $0.01
  • Inter-continent: $0.02–0.04

Turbo Replication: $0.04 additional per GB replicated.

Example: 1 TB/day new data, dual-region with Turbo:

  • Storage: 1 TB × $0.020 = $20
  • Turbo: 1 TB × $0.04 = $40
  • Monthly: (1 TB × 30 days) storage + turbo = $600 + $1200 = $1800

Anti-Patterns ​

Anti-pattern 1: "GCS Is Geo-Redundant by Default" ​

Symptom: Team assumes GCS bucket is automatically protected against region failure.

Problem:

  • Regional bucket has zero cross-region protection
  • Dual-region bucket eventually consistent (1-hour RPO)
  • Not all applications can tolerate 1-hour data loss

Right approach: Explicitly choose dual-region or multi-region. Understand RPO implications.

Anti-pattern 2: "Cross-Bucket Replication is Automatic" ​

Symptom: Team sets up source bucket + destination bucket, expects replication to happen.

Problem:

  • GCS doesn't auto-replicate between buckets
  • Must explicitly use Transfer Service or custom script
  • If not set up, destination bucket empty

Right approach: Configure Transfer Service with schedule. Verify replication is working (monitor bucket sizes).

Anti-pattern 3: "Eventual Consistency is Fine for Everything" ​

Symptom: Team uses multi-region bucket for transactional data that requires strong consistency.

Problem:

  • Concurrent writes lead to split-brain (different regions see different data)
  • Application logic might corrupt data (race conditions)

Right approach: Transactional data → single-region bucket with application-level replication (e.g., database). Multi-region bucket suitable for immutable, append-only data.

Summary ​

GCS offers three geographic options:

  • Regional: No cross-region protection
  • Dual-region: 1-hour RPO default, 15-min with Turbo
  • Multi-region: Distributed, cheaper, slower replication

Choose based on RPO requirement and cost budget. Cross-bucket replication via Transfer Service for fine-grained control.

References ​