Multi-Region Architecture: Active-Active vs Active-Passive
Tại Sao Điều Này Quan Trọng
Active-active vs active-passive không phải là "preference" hay "best practice" — chúng là fundamentally different system models với những tính chất, độ phức tạp, và failure modes hoàn toàn khác nhau. Chọn sai = split-brain disasters, data corruption, hoặc chi phí vô lý cao.
Internal Model: Hai Paradigm Cơ Bản
Active-Passive Architecture
Tại bất cứ thời điểm nào, chỉ có duy nhất một region (primary) được accept writes. Region kia (secondary/standby) là read-only replicas.
Primary Region Secondary Region
┌────────────────┐ ┌────────────────┐
│ GKE Cluster │────→│ GKE Cluster │
│ (accepts │ │ (read-only │
│ writes) │ │ replicas) │
│ │ │ │
│ Cloud SQL: │────→│ Cloud SQL: │
│ primary DB │ │ read replica │
│ │ │ │
│ PubSub Topic │────→│ (no writes) │
│ │ │ │
└────────────────┘ └────────────────┘Replication flow:
- Write đến primary region
- Primary region ghi vào database, topic, storage
- Secondary region replicate từ primary (async hoặc semi-sync)
- If primary down → manual failover (promote secondary to primary)
Consistency model: Strong consistency (within region), eventual consistency across regions.
CAP trade-off: Partition tolerance, Availability (during normal operations), sacrifice Consistency (across regions) or RTO (downtime during failover).
Active-Active Architecture
Cả hai regions chấp nhận writes đồng thời. System phải reconcile conflicts khi cả regions ghi vào same resource.
Region A Region B
┌────────────────┐ ┌────────────────┐
│ GKE Cluster │←───→│ GKE Cluster │
│ (accepts │ │ (accepts │
│ writes) │ │ writes) │
│ │ │ │
│ Cloud SQL: │←───→│ Cloud SQL: │
│ primary │ │ primary │
│ │ │ │
│ Distributed │ │ Distributed │
│ Consensus │ │ Consensus │
│ Coordinator │ │ Coordinator │
└────────────────┘ └────────────────┘Replication flow:
- Write đến region A hoặc region B (dùng distributed consensus coordinator)
- Coordinator quyết định version/ordering (avoid conflicts)
- Both regions replicate write to each other (real-time)
- If one region down → system continue operate (zero RTO)
Consistency model: Depends on implementation:
- Strong consistency (with higher latency) = synchronous replication
- Eventual consistency (with lower latency) = async replication + conflict resolution
CAP trade-off: Partition tolerance, Consistency, but sacrifice Availability when network partition occurs (need to choose between partition continuity or split-brain).
Core Difference: Recovery Time vs Consistency Cost
Active-Passive: Simple But Requires Manual Intervention
During normal operations:
- Primary region handles all writes
- Secondary region is passive (consuming storage/compute for no production traffic)
- Replication lag = RPO target
During primary failure:
- Detect primary failure (5–30 min)
- Promote secondary to primary (manual decision)
- Switch DNS to secondary (5–30 min)
- Client connections reroute (TTL-dependent, 1–5 min)
- Service restarts on secondary with recovered state
RTO = detection + promotion + DNS switch + client reroute = 15 min — 1 hour
Why promote is manual? Because you need human to confirm:
- Primary is actually dead (not just network partition)
- Secondary has caught up with replication
- No split-brain scenario (old primary still serving)
If promote automatic without these checks → cascading failures.
Advantage:
- Simple architecture
- Easy to reason about consistency (primary always authority)
- No distributed consensus complexity
- Disaster recovery procedure clear
Disadvantage:
- RTO measured in minutes/hours (downtime unavoidable)
- Manual failover depends on engineer availability
- During failover, system is blind (no serving region)
Active-Active: Zero Downtime But Requires Distributed Consensus
During normal operations:
- Both regions handle reads + writes
- Real-time replication between regions (bidirectional)
- Distributed consensus coordinator ensures consistency
During region failure:
- Health check detects failure (10–15 seconds)
- Consensus coordinator excludes failed region (immediate)
- Remaining region continues serving (zero RTO)
- Failed region rejoins automatically when recovered
RTO = 10–30 seconds (near-zero)
Why automatic works? Because system is designed for partition tolerance:
- Consensus protocol handles region failure automatically
- No manual decision needed (protocol decides)
- Remaining region has quorum (can continue)
Advantage:
- Zero downtime, automatic failover
- No manual intervention needed
- RTO = seconds
Disadvantage:
- Complex distributed consensus implementation
- Split-brain possible (if poorly designed)
- Higher latency (writes must replicate before confirming)
- More infrastructure cost (both regions always active)
Distributed Consensus: The Core Problem
Active-active requires distributed consensus — agreement between multiple regions on "what is current state" without a central authority.
Why is this hard?
Write arrives: "account balance += $100"
Region A receives at T=0, applies, balance = $1100
Region B receives at T=5ms, applies, balance = $1100
Consensus: "everyone agrees, quorum reached, commit"
But what if network partition?
Region A: sees write, applies, balance = $1100
Region B: doesn't see write (network down)
Region B: balance = $1000
Network heals after 1 minute:
Region A: balance = $1100 (correct)
Region B: balance = $1000 (stale)
Who is right? Both regions are alive.
→ Split-brain: two conflicting versions
Consensus protocol must decide:
- Discard Region B's writes? (RPO > 0)
- Which version is correct? (merge strategy?)
- Can clients trust the version they read?Solutions to consensus problem:
Raft/Paxos consensus: Leader election, quorum writes
- Ensures only one region can be "leader" at a time
- Prevents split-brain
- Cost: leader region must achieve quorum before write confirmed
- Higher latency
Vector clocks + conflict-free replicated data types (CRDT):
- Each region assigns timestamp to writes
- Track causality (which writes depend on which)
- On conflict, apply deterministic merge rule
- Cost: application must handle merge logic
- Risk: merge rule may not preserve business logic
Last-write-wins (LWW):
- Simplest: highest timestamp wins on conflict
- Fastest: no consensus protocol
- Risk: can lose data (overwrite without merging)
- Example: two regions update same field simultaneously, LWW discards one region's write
Consistency Guarantees: What You Actually Get
Active-Passive
Strong consistency within region, eventual consistency across regions.
Time ─────────────────────────────→
Primary writes: A, B, C
│
│ (replicate to secondary)
│
├─→ Secondary sees A (2ms replication lag)
├─→ Secondary sees B (5ms lag)
└─→ Secondary sees C (8ms lag)If read from secondary before replication complete → might not see latest write. But once replicated, it's correct.
Strong consistency guarantee: If you read from primary, you always see latest write.
Eventual consistency guarantee: If primary down and you fail over to secondary, you see state at failover time (some in-flight writes lost).
Active-Active with Raft Consensus
Strong consistency across regions, but higher latency.
Write to Region A:
│
├─→ Region A receives (0ms)
├─→ Region A sends to Region B (2ms)
├─→ Region B acknowledges (4ms)
├─→ Region A confirms to client (after quorum reached)
│
Result: write confirmed only after both regions agreeCost: Every write waits for cross-region round-trip before confirming.
Advantage: If either region fails after write confirmed, other region has the data.
Active-Active with Eventually-Consistent Replication (e.g., Cloud Spanner)
Eventual consistency with multi-region failover.
Cloud Spanner uses synchronous replication across regions but allows regional replicas to serve reads independently:
Write to Region A:
│
├─→ Region A primary (0ms)
├─→ Regions B, C replicate async (10–100ms)
├─→ Write confirmed to client immediately
Read from Region B during replication lag:
├─→ Might not see latest write
└─→ Or sees it (depends on replication speed)Trade-off: Lower write latency (confirm before replication complete) but eventual consistency.
Split-Brain: The Active-Active Nightmare Scenario
Split-brain = two regions both think they're primary, both accept writes, data diverges.
Time ─────────────────────────────→
Normal state:
Region A (primary): balance = 1000
Region B (secondary): balance = 1000
T=0: Network partition (regions can't talk)
Region A thinks: "B is down, I'm primary"
Region B thinks: "A is down, I'm primary"
User A deposits $500 to Region A
User B deposits $300 to Region B
Region A: balance = 1500
Region B: balance = 1300
T=1m: Network heals
Which version is correct?
- Both? (total $1800 created from thin air)
- Region A's? (Region B's user loses deposit)
- Region B's? (Region A's user loses deposit)
This is split-brain. Data corruption.Why split-brain happens:
In active-passive, no problem (only primary can write, secondary is passive).
In active-active without consensus:
- Both regions have write ability
- Network partition separates them
- Both think other is down
- Both accept writes independently
- When network heals, data is inconsistent
How consensus protocols prevent split-brain:
Raft consensus requires quorum:
3 regions: A, B, C
Quorum = 2 regions must agree
Partition 1: A, B (quorum exists, can continue)
Partition 2: C (quorum doesn't exist, must stop accepting writes)
Only Partition 1 can write after partition.
C is read-only until network heals.
When heals: C gets state from A+B, no conflict.Cost of preventing split-brain:
Need odd number of regions (3, 5, 7) to achieve quorum. Even 2 regions cannot prevent split-brain without external coordinator.
Architecture Decision Matrix
| Attribute | Active-Passive | Active-Active |
|---|---|---|
| RTO | 15 min — 2 hours | < 1 minute |
| RPO | Replication lag (1–60 min) | < 1 minute |
| Write latency | Low (single region) | High (cross-region consensus) |
| Infrastructure cost | 1.2× (secondary idle) | 2.5–3× (all active) |
| Complexity | Low (simple replication) | Very High (consensus protocol) |
| Split-brain risk | Low (only primary writes) | High (need consensus) |
| Failure scenario | Manual decision needed | Automatic, immediate |
| Data consistency | Strong (per region) | Strong (if Raft) or Eventually-consistent (if async) |
| Best for | RTO = 30 min+, complex state | RTO = < 5 min, can afford complexity |
GCP Managed Services: Built-in Support
Cloud SQL: Active-Passive by Default
Cloud SQL offers:
- High Availability (HA): Primary + sync replica in same region (different zones)
- Read replicas: Async replicas in different regions (active-passive pattern)
Primary (us-central1)
│
├─→ HA replica (us-central1, different zone, sync)
│
└─→ Read replicas (us-east1, asia-southeast1, async)Failover to HA replica is automatic (same region, fast). Failover to read replica is manual (cross-region).
Cloud Spanner: Active-Active by Default
Spanner is designed for active-active:
Multi-region Spanner instance
│
├─ Region us-central1 (leader or replica)
├─ Region us-east1 (leader or replica)
└─ Region asia-southeast1 (leader or replica)
All regions can serve reads.
Writes must go through leader (but leader can be any region, load-balanced).
Consensus: Paxos protocol (built-in).
Replication: Synchronous across regions (RPO = near-zero).
Write latency: 10–100ms extra per write (waiting for replication).Spanner trades write latency for strong consistency across regions.
Cloud Storage: Replication, Not Active-Active
Cloud Storage doesn't do active-active writes. It does:
- Regional bucket: Replicate across zones (same region)
- Dual-region bucket: Replicate across two regions (async, 1-hour target RPO)
- Multi-region bucket: Distribute across continent (async, 12-hour RPO)
Replication is one-way or policy-based (not consensus), so no split-brain. But also no active-active writes.
Real-World Architecture Patterns
Pattern 1: Active-Passive + Cold Standby (E-Commerce Catalog)
Service: Product catalog (millions of products, read-heavy).
Primary: us-central1
├─ GKE cluster (serves reads + writes)
├─ Cloud SQL primary (5TB)
└─ Cloud Storage regional
Secondary: us-east1
├─ Backup cluster (provisioned but mostly idle)
├─ Cloud SQL read replica (from primary, async)
└─ Cloud Storage regional (copied via Transfer Service)
Failover procedure:
1. Detect primary down (5 min)
2. Promote read replica to primary (manual, 5 min)
3. Switch DNS (1–5 min)
4. Scale up secondary cluster (2–3 min)
Total RTO: 15–20 min
RPO: replication lag + in-flight writes = 30–60 secondsWhy this architecture:
- Read-heavy (replication lag acceptable, only affects writes)
- RTO = 30 min acceptable (not critical e-commerce like checkout)
- Cost: 1.5× infrastructure (HA replica + read replica + backup cluster)
Pattern 2: Active-Active + Distributed Consensus (Financial Trading Platform)
Service: Real-time trade execution, must never go down.
Region A: us-central1
├─ GKE cluster
├─ Spanner multi-region instance (leader)
└─ PubSub topics
Region B: us-east1
├─ GKE cluster
├─ Spanner multi-region instance (replica/leader)
└─ PubSub topics
Region C: europe-west1 (external consensus)
├─ Spanner multi-region instance (replica)
└─ Detects split-brain via Paxos quorum
Failover:
1. Region A fails at T=0
2. Health check detects at T=15s
3. Remaining regions reach quorum, lock A out
4. B/C continue serving, Spanner keeps consensus
5. Clients auto-reroute (LB failover)
Total RTO: < 1 minute (automatic)
RPO: < 1 minute (Spanner sync replication across regions)Why this architecture:
- High-frequency trading (every second down = millions lost)
- RTO = < 1 min non-negotiable
- Strong consistency non-negotiable (can't lose trades)
- Cost: 3× infrastructure + Spanner multi-region licensing
Pattern 3: Active-Passive + Periodic Snapshots (Internal Analytics)
Service: Daily batch analytics job, read-only dashboard.
Primary: us-central1
├─ GKE cluster (job scheduler)
├─ BigQuery dataset (results)
└─ Snapshots every 24 hours
Secondary: us-east1
├─ Backup BigQuery snapshots copied daily
└─ (no active cluster)
Failover (if primary region completely unavailable):
1. Wait for daily snapshot (up to 24 hours)
2. Copy snapshot to secondary region
3. Create new BigQuery table in secondary
4. Update dashboard to point to secondary
5. Restart dashboard in secondary (or use cross-region access)
Total RTO: 4–6 hours (manual, during business hours)
RPO: up to 24 hours (last snapshot)Why this architecture:
- Analytics (not critical real-time)
- RTO = 8 hours acceptable (disaster recovery, business hours)
- RPO = 24 hours acceptable (re-run job from previous day)
- Cost: ~10% of production (only snapshots)
Anti-Patterns
Anti-pattern 1: "Active-Active Everywhere"
Symptom: Team tries to make every service active-active because "it sounds better."
Problem:
- Not every service needs zero RTO
- Active-active adds write latency (replication round-trip)
- Distributed consensus complexity introduces bugs
- Cost 2–3× infrastructure for services that could tolerate 30-min downtime
Right approach: Choose active-passive for services with RTO ≥ 30 min. Use active-active only when RTO < 5 min is true business requirement.
Anti-pattern 2: "We Have Replication, So Split-Brain Can't Happen"
Symptom: Team relies on async replication for active-active, assumes data safety.
Problem:
- Async replication doesn't prevent split-brain
- Network partition + async replication = writes from both regions, conflicts on merge
- If consensus protocol missing = split-brain is guaranteed
Right approach: Active-active requires consensus protocol (Raft, Paxos). Async replication alone is not enough.
Anti-pattern 3: "Failover Is Automatic, So We Don't Need Runbook"
Symptom: Team configures automatic failover, assumes disaster recovery is done.
Problem:
- Automatic failover might trigger on false positive (false alarm)
- Cascading failures if secondary also unhealthy
- Post-failover recovery procedure manual (rebuilding primary, replication sync, DNS rollback)
- Team doesn't practice failover, so surprised when it actually happens
Right approach: Even with automatic failover, write detailed runbook. Practice quarterly. Document what happens after automatic failover completes.
Summary
Active-passive is simpler but slower. Active-active is faster but requires distributed consensus + higher cost + more complexity.
Choose based on RTO requirement:
- RTO ≥ 30 min: Active-passive (simple, cost-effective)
- RTO < 5 min: Active-active + consensus (complex, necessary)
- 5–30 min: Evaluate on per-service basis (probably active-passive with semi-auto failover)