Kỹ thuật lưu lượng & Cân bằng tải đa đường

Vì sao quan trọng trong sản xuất

Cuối cùng, chúng ta kết hợp tất cả — cách GCP tối ưu hóa đường gói tin qua Jupiter fabric + Andromeda + backbone:

• Định tuyến ECMP: Phân phối tải qua đa đường giá bằng
• Lập kế hoạch dung lượng: Đảm bảo khoảng trống hỏng + đỉnh
• Kịch bản hỏng: Mạng phục hồi hỏng như thế nào
• Dẻo dai đa đường: Dự phòng tự động không can thiệp thủ công

Hiểu cấp độ này = bạn có thể kiến trúc mạng dẻo dai quy mô lớn.

Mô hình nội bộ: Kiến trúc định tuyến đa đường

Định tuyến ECMP (Bằng chi phí đa đường)

Định tuyến unicast chuẩn:
┌──────────────┐
│ VM nguồn     │
│ 203.0.113.1  │
└──────┬───────┘
       │
    ┌──▼──────────────────────────────────────────┐
    │ Tra cứu bảng định tuyến:                   │
    │ Đích: 35.201.123.45                        │
    │ Kết quả: 1 đường tốt nhất cổng            │
    └──────────┬─────────────────────────────────┘
               │
    ┌──────────▼──────────────────────────────┐
    │ Cổng đơn                                │
    │ Gói: →[Router]→[Switch]→[Sợi]→...      │
    └──────────┬───────────────────────────────┘
               │ (Nút thắt nếu router hỏng!)
    ┌──────────▼──────────────┐
    │ Đích                    │
    │ 35.201.123.45 (duy nhất)│
    └───────────────────────┘

Vấn đề:
├─ Đường đơn = điểm lỗi duy nhất
└─ Router hỏng: Tất cả lưu lượng bị chặn

Định tuyến đa đường ECMP:
┌──────────────┐
│ VM nguồn     │
│ 203.0.113.1  │
└──────┬───────┘
       │
    ┌──▼──────────────────────────────────────────────┐
    │ Tra cứu bảng định tuyến:                       │
    │ Đích: 35.201.123.45                            │
    │ Kết quả: 4 đường chi phí bằng (chỉ số giống)   │
    │ ├─ Đường A qua Cổng1 (chi phí 100)             │
    │ ├─ Đường B qua Cổng2 (chi phí 100)             │
    │ ├─ Đường C qua Cổng3 (chi phí 100)             │
    │ └─ Đường D qua Cổng4 (chi phí 100)             │
    └──────────┬─────────────────────────────────────┘
               │
    ┌──────────┴────────────┬────────────┬─────────────┐
    │                       │            │             │
    ▼                       ▼            ▼             ▼
┌─────────┐            ┌─────────┐ ┌─────────┐ ┌─────────┐
│Cổng1    │            │Cổng2    │ │Cổng3    │ │Cổng4    │
│(25%)    │            │(25%)    │ │(25%)    │ │(25%)    │
└────┬────┘            └────┬────┘ └────┬────┘ └────┬────┘
     │                      │           │           │
     │ (ECMP: Hash flow)    │           │           │
     │ Luồng1→Đường A       │           │           │
     │ Luồng2→Đường B       │           │           │
     │ Luồng3→Đường C       │           │           │
     │ Luồng4→Đường D       │           │           │
     │ Luồng5→Đường A (dùng) │          │           │
     │                      ▼           ▼           ▼
     └──────────────────────┴───────────┴───────────→ Đích

Lợi ích:
├─ Cân bằng tải: 25% lưu lượng mỗi đường
├─ Dẻo dai: Đường A down, Luồng1 định tuyến lại Đường B
├─ Lưu lượng: Dung lượng 4x so với đường đơn
└─ Tự động: Không can thiệp thủ công

Hàm Hash ECMP

Router thấy đa đường chi phí bằng:
├─ Thay vì dùng đường "đầu tiên"
├─ Tính: Hash(IP_nguồn, IP_đích, cổng_nguồn, cổng_đích)
├─ Kết quả: Giá trị hash 32-bit
└─ Dùng: hash % số_đường = đường chọn

Ví dụ 4 đường:
├─ Luồng (203.0.113.1:8000 → 35.201.123.45:443)
│  └─ Hash: 0x4a5f9c2e % 4 = 2 → Đường C
│
├─ Luồng (203.0.113.2:8001 → 35.201.123.45:443)
│  └─ Hash: 0x7b3e1d5a % 4 = 0 → Đường A
│
└─ Luồng giống luôn lấy đường giống (ổn định phiên TCP)

Lợi ích:
└─ Cân bằng tải mỗi luồng (không mỗi gói, sẽ sắp xếp lại)

Production Architecture Patterns

Pattern 1: Data Center Network with ECMP

Deployment: 200 VMs processing 50Gbps traffic

Network topology (Jupiter fabric):
├─ Leaf switches: 4 (one per rack)
├─ Spine switches: 8
│
└─ Paths per VM:
   ├─ To spine: 2 parallel links (LAG to 2 spines)
   ├─ From spine: 8 possible paths (to any spine → any leaf)
   └─ Total paths to any destination: 8 (ECMP)

Traffic distribution:
├─ 50Gbps = 50,000 flows (100Kbps per flow average)
├─ ECMP hash: Distribute flows across 8 paths
├─ Per path: 6.25Gbps (50Gbps ÷ 8)
├─ Per link: Can handle 100Gbps
└─ Utilization: 6% (plenty of headroom)

Failure scenario (1 spine down):
├─ Paths available: 7 (instead of 8)
├─ Per-path load: 7.1Gbps
├─ Still acceptable: 7% utilization
├─ Automatic: No config changes needed, ECMP recalculates

Failure scenario (1 leaf down):
├─ 50 VMs affected (on that leaf)
├─ Traffic rerouted: Via remaining leaves
├─ Temporary: Congestion spike (~15% util for 30 seconds)
├─ Recovery: Workloads reschedule to other leaves (~5 min)

Pattern 2: Global Multi-Region with ECMP

Global application: 4 regions (us-central1, eu-west1, asia-southeast1, us-east1)
Peak traffic: 100Gbps to any region

Per-region ECMP setup:
├─ Each region: 8 spines (ECMP paths to other regions)
├─ Regional egress: 4 primary paths to backbone
├─ Plus: 2 backup paths (in case of congestion/failure)
│
└─ Per-region capacity:
   ├─ 100Gbps to each other region (multiples)
   └─ Total: 300Gbps (to all 3 other regions)

Traffic engineering during peak:
├─ Normal: Load distributed via ECMP
├─ Congestion detected: Traffic shifted to secondary paths
├─ Result: Automatic re-balancing (milliseconds)

Failure scenario (transatlantic cable cut):
├─ Primary path (via Cable A): Down
├─ Backup path (via Cable B): Activated
├─ ECMP: Recalculates, includes backup path
├─ Result: Latency increases (longer route) but connectivity maintained
├─ Recovery: <100ms failover (ECMP convergence time)

Pattern 3: Capacity Planning for Failure Scenarios

Production SLA: 99.99% (52.6 min downtime/year)
Normal traffic: 10Gbps average
Peak traffic: 20Gbps (2x normal)
Target resilience: Survive any single failure

Capacity formula:
├─ Normal capacity needed: 10Gbps (baseline)
├─ Peak capacity needed: 20Gbps (expected peak)
├─ Failure resilience: Must absorb any 1 failure
├─ Required total: 20 * 1.5 = 30Gbps (1.5x for n+1)

Allocation strategy:
├─ Region 1: 10Gbps capacity (full traffic)
├─ Region 2: 10Gbps capacity (full traffic)
├─ Region 3: 10Gbps capacity (full traffic backup)
├─ Total provisioned: 30Gbps
│
└─ Failure scenarios:
   ├─ Region 1 fails: Region 2+3 can handle 20Gbps ✓
   ├─ Region 2 fails: Region 1+3 can handle 20Gbps ✓
   ├─ Region 3 fails: Region 1+2 can handle 20Gbps ✓
   └─ Any single failure: SLA maintained ✓

Real-world Failure Scenarios

Scenario 1: Asymmetric Path Failure (ECMP Challenge)

Setup: 4 ECMP paths from source to destination

Symptom:
├─ Throughput: 50Gbps → 25Gbps (50% drop)
├─ No packets lost
├─ No errors in logs

Root cause:
├─ Path A (ingress): Working normally
├─ Path A (egress): Congested (return path bottleneck)
├─ Path B,C,D: Not affected
│
└─ Impact:
   ├─ Inbound via Paths A,B,C,D: 12.5Gbps each
   ├─ Outbound via Path A only: 50Gbps (limited!)
   └─ Result: Bottleneck at egress

Investigation:
├─ Measure: One-way latency A→dest vs dest→A
├─ Result: Return path 10x slower
├─ Cause: Asymmetric congestion

Resolution:
├─ Shift egress traffic to use other paths
├─ Require: Application-level routing control
├─ Or: Manual traffic engineering (adjust BGP weights)

Scenario 2: ECMP Hash Collision (Unbalanced Load)

Symptom:
├─ VM1: 100% CPU (100Gbps traffic)
├─ VM2: 10% CPU (10Gbps traffic)
├─ VM3: 10% CPU
├─ Total traffic: 120Gbps

Expected:
└─ 40Gbps per VM (equal distribution)

Root cause:
├─ ECMP hash function: Hash(flow_tuple) % num_paths
├─ Large flows: All happen to hash to same path
├─ Example:
│  ├─ Flow: (203.0.113.1:50000 → 35.201.123.45:443) Hash=0 → Path 0 (VM1)
│  ├─ Flow: (203.0.113.1:50001 → 35.201.123.45:443) Hash=0 → Path 0 (VM1)
│  ├─ Flow: (203.0.113.1:50002 → 35.201.123.45:443) Hash=0 → Path 0 (VM1)
│  └─ Result: VM1 gets multiple large flows, others get small flows

Detection:
├─ Monitor per-flow distribution
├─ Alert: If any single path >70% of average
└─ Requires: Network telemetry (NetFlow/sFlow)

Resolution:
├─ Short-term: Redistribute flows manually (drain VM1)
├─ Long-term: Improve hash function (use more tuple elements)
├─ Or: Use application-level load balancing instead of ECMP

Scenario 3: Cascade Failures (Multiple Failures)

Failure sequence:
├─ T+0:00 Path A fails (high utilization)
├─ T+0:05 Remaining paths: 7 (instead of 8)
├─ T+0:10 Congestion: Each path now 14.3% loaded
│
├─ T+0:15 Path B fails (triggered by congestion on Path A's backup)
├─ T+0:20 Remaining paths: 6
├─ T+0:25 Congestion: Each path 16.7% loaded
│
├─ T+0:30 Path C fails (cascade continues)
│  └─ Pattern: Each failure triggers congestion, which triggers next failure
│
└─ Result: Cascading failure (failure chain reaction)

Why cascades happen:
├─ Congestion threshold: Path degrades after ~70% utilization
├─ Multiple failures: Remaining paths saturated
├─ Loss feedback: Lost packets trigger retransmits, more congestion
└─ Doom loop: Congestion → more loss → more congestion

Prevention:
├─ Design: n+2 redundancy (can lose 2 paths)
├─ Monitor: Alert when single path fails
├─ Failover: Drain load from remaining paths to prevent cascade
├─ Burst allowance: Built-in safeguards in GCP

Common Mistakes & Anti-Patterns

Mistake 1: Designing for n Redundancy (Instead of n+1)

❌ Wrong thinking:

"Have multiple paths, any one can fail and we're still ok"

✅ Correct understanding:

• n redundancy: Lose 1 path, continue working
• n+1 redundancy: Lose 1 path, no performance degradation
• Need: n+1 to maintain SLA during single failure

Prevention: Design formula: total_capacity = (peak_traffic * 1.5)

Mistake 2: Not Monitoring Per-Path Utilization

❌ Wrong thinking:

"Average utilization 40%, safe even if path fails"

✅ Correct understanding:

• Average: Meaningless if one path is 90% and others 10%
• ECMP hash collision: Can cause unbalanced load
• Must monitor: Per-path utilization

Prevention: Alert on any path >70% of average utilization.

Mistake 3: Assuming Automatic Failover "Always Works"

❌ Wrong thinking:

"ECMP automatic, failover handled by network"

✅ Correct understanding:

• ECMP failover: Automatic but takes 10-50ms
• Long connections: Might get disrupted
• TCP: Expects packet ordering, ECMP changes paths mid-connection
• Result: Possible packet loss during failover

Prevention: Design applications to handle connection resets. Use connection pooling.

GCP-native Implementation Guidance

Monitoring ECMP Load Distribution

# Enable VPC Flow Logs to observe path distribution
gcloud compute networks subnets update my-subnet \
  --enable-flow-logs \
  --region=us-central1

# Query flows with path information
gcloud logging read \
  'resource.type="gce_instance" AND jsonPayload.traffic_type="TCP"' \
  --format='table(jsonPayload.src_ip, jsonPayload.dst_ip, jsonPayload.bytes_sent)' \
  --limit=100 | sort -k3 -nr

# Analyze ECMP distribution (should be roughly equal)
gcloud logging read \
  'resource.type="gce_instance"' \
  --format='table(jsonPayload.end_time, jsonPayload.bytes_sent)' | \
  awk '{sum[$1]+=$2} END {for (t in sum) print t, sum[t]/1000000000 " GB"}' | sort

Capacity Planning Calculation

#!/bin/bash
# Calculate required capacity for failure resilience

PEAK_TRAFFIC_GBPS=50
RESILIENCE_FACTOR=1.5  # n+1 redundancy

REQUIRED_CAPACITY=$(echo "$PEAK_TRAFFIC_GBPS * $RESILIENCE_FACTOR" | bc)

echo "Peak traffic: ${PEAK_TRAFFIC_GBPS}Gbps"
echo "Resilience factor: $RESILIENCE_FACTOR (n+1)"
echo "Required capacity: ${REQUIRED_CAPACITY}Gbps"
echo ""
echo "Deployment strategy:"
echo "├─ Region 1: ${PEAK_TRAFFIC_GBPS}Gbps"
echo "├─ Region 2: ${PEAK_TRAFFIC_GBPS}Gbps"
echo "└─ Total provisioned: ${REQUIRED_CAPACITY}Gbps"
echo ""
echo "Failure scenario:"
echo "├─ 1 region fails: Remaining regions absorb ${PEAK_TRAFFIC_GBPS}Gbps"
echo "├─ Capacity used: ${PEAK_TRAFFIC_GBPS}Gbps"
echo "├─ Utilization: $(echo "scale=1; $PEAK_TRAFFIC_GBPS / $REQUIRED_CAPACITY * 100" | bc)%"
echo "└─ Result: SLA maintained ✓"

Testing Failover Scenarios

# Simulate failure by blocking a specific egress path
# 1. Identify path (via traceroute or NetFlow)
GATEWAY="192.168.1.1"

# 2. Block path using iptables (on test VM only!)
sudo iptables -A FORWARD -d $GATEWAY -j DROP

# 3. Observe application behavior
# - Does latency increase?
# - Do connections drop?
# - How long to recover?

# 4. Monitor logs
gcloud logging read "resource.type=gce_instance AND severity=WARNING" \
  --format='table(severity, timestamp, textPayload)'

# 5. Restore path
sudo iptables -D FORWARD -d $GATEWAY -j DROP

# 6. Verify recovery
# Analysis: How long until normal operation?

References

• ECMP Load Balancing in GCP — GCP implementation
• Network Resilience Best Practices — Architecture patterns
• Traffic Engineering with BGP — Advanced routing
• Capacity Planning Guide — Formal methodology

---

Conclusion: Putting It All Together

You've now covered:

1. Andromeda (SDN layer) — How VPCs implemented
2. Jupiter (Physical fabric) — Hardware topology
3. PoP (Edge) — How traffic enters GCP
4. Global backbone — Premium vs Standard routing
5. Latency SLA — Fiber path engineering
6. Anycast — Global load balancing
7. Traffic routing — Cold potato vs hot potato
8. Network tiers — SLA implementation
9. Bandwidth allocation — Capacity & costs
10. Regional services — Data sovereignty
11. Traffic engineering — Multi-path resilience

Together: A complete understanding of GCP's physical network architecture for production systems.

Next Steps

• Implement: Choose right tier/region for your workload
• Monitor: Set up per-path monitoring
• Design: Plan for n+1 resilience
• Audit: Verify regional constraints
• Optimize: Balance latency, cost, and compliance