Network Isolation & VPC Peering

Tại sao VPC Peering Là Nền Tảng Của Multi-Cluster Networking

Trước khi MCS, MCI, service mesh, có một câu hỏi cơ bản:

Làm sao pod ở cluster A reach pod ở cluster B?

Cluster A Pod (10.0.1.20) →
  Packet destination: 10.1.1.20 (cluster B pod)
  Routing table: "How do I reach 10.1.0.0/24?"
  → Need route export từ cluster B VPC
  → Need VPC connectivity giữa cluster A VPC ↔ cluster B VPC

VPC Peering = Kết nối hai VPC networks để pods có thể communicate:

Cluster A VPC: 10.0.0.0/16
Cluster B VPC: 10.1.0.0/16

Peering: cluster-a-vpc ↔ cluster-b-vpc
Result:
  - cluster A pod route: 10.1.0.0/16 → cluster-b-vpc (peer)
  - cluster B pod route: 10.0.0.0/16 → cluster-a-vpc (peer)
  - Packets flow directly (not through cloud router)

VPC Peering Model

Architecture

┌─────────────────────┐    ┌──────────────────────┐
│  Cluster A VPC      │    │  Cluster B VPC       │
│  10.0.0.0/16        │    │  10.1.0.0/16         │
│                     │    │                      │
│  ┌───────────────┐  │    │  ┌───────────────┐  │
│  │ Subnet        │  │    │  │ Subnet        │  │
│  │ 10.0.1.0/24   │  │    │  │ 10.1.1.0/24   │  │
│  │               │  │    │  │               │  │
│  │ Pod 10.0.1.20 │  │    │  │ Pod 10.1.1.20 │  │
│  └───────────────┘  │    │  └───────────────┘  │
│                     │    │                      │
└──────────┬──────────┘    └──────────┬───────────┘
           │                          │
           └──────────────────────────┘
                  VPC Peering
                  (layer 3 connection)

Key properties:

Non-transitive: A↔B, B↔C ≠ A↔C
Bidirectional route exchange: Routes auto-shared both ways
No hop charges: Traffic không tính bandwidth charges (unlike Cloud VPN)
Network isolation: Firewall rules không cross peering boundary

Route Exchange Mechanism

Automatic Route Propagation

Setup: cluster-a-vpc ↔ cluster-b-vpc peering

Cluster A:
  - Primary CIDR: 10.0.0.0/16
  - Pod CIDR (secondary): 10.0.128.0/18
  ↓ Routes exported via peering:
  - 10.0.0.0/16 (subnet route)
  - 10.0.128.0/18 (Pod CIDR)

Cluster B Routing Table:
  Destination: 10.0.0.0/16
  Next Hop: cluster-a-vpc (peer)
  
  Destination: 10.0.128.0/18
  Next Hop: cluster-a-vpc (peer)

Automatic routes: Only subnet routes (primary CIDR) auto-export. Secondary ranges (Pod CIDR) thường cần manual configuration.

Custom Route Exchange

bash

# Custom route ở cluster A (không auto-export)
gcloud compute routes create cluster-a-custom \
  --destination-range=192.168.0.0/16 \
  --next-hop-address=10.0.0.1 \
  --network=cluster-a-vpc

# Konfigurasi peering untuk export custom route
gcloud compute networks peerings update cluster-a-to-b \
  --export-custom-routes \
  --import-custom-routes \
  --network=cluster-a-vpc

# Cluster B routing table: now sees 192.168.0.0/16

Firewall Rules & Peering Boundary

Key Insight: Firewall Rules Don't Cross Peering

Cluster A Firewall Rule:
  Direction: Ingress
  Source IP: 10.0.0.0/16
  Allow: TCP port 8080

Cluster B Pod (10.1.1.20) trying to reach Cluster A Pod (10.0.1.20:8080):

1. Route check: 10.0.0.0/16 → cluster-a-vpc ✓
2. Peering check: connection allowed ✓
3. Firewall check: Source 10.1.1.20 in 10.0.0.0/16? ❌ NO
   → Firewall rule doesn't match
   → Connection DENIED

Result: Traffic blocked by Cluster A firewall (source IP outside rule range)

Solution: Cluster A firewall rule phải explicitly allow Cluster B CIDR:

bash

# Cluster A: Allow traffic from Cluster B
gcloud compute firewall-rules create allow-cluster-b \
  --network=cluster-a-vpc \
  --allow=tcp:8080 \
  --source-ranges=10.1.0.0/16,10.1.128.0/18  # cluster-b VPC + pod CIDR

Topology Options

Option 1: Mesh Topology (All-to-All Peering)

    A ──────── B
    │ ╲      ╱ │
    │  ╲    ╱  │
    │   ╲  ╱   │
    │    ╲╱    │
    │    ╱╲    │
    │   ╱  ╲   │
    │  ╱    ╲  │
    C ──────── D

Peering relationships: A↔B, A↔C, A↔D, B↔C, B↔D, C↔D
Total = 6 peerings (n×(n-1)/2 for 4 clusters)

Ưu điểm:

Direct connectivity: A pod → D pod direct path
Resilient: no single point of failure
Best latency

Nhược điểm:

Toil: complex firewall rules, many peerings
Scaling: n=10 clusters = 45 peerings

Use case: Small clusters (2-5), high performance requirement.

Option 2: Hub-and-Spoke Topology

        ┌─────────┐
        │   Hub   │
        │  (GKE)  │
        └────┬────┘
             │
    ┌────────┼────────┐
    │        │        │
┌───▼──┐ ┌──▼───┐ ┌──▼───┐
│Spoke1│ │Spoke2│ │Spoke3│
└──────┘ └──────┘ └──────┘

Peering: Hub↔Spoke1, Hub↔Spoke2, Hub↔Spoke3 (3 peerings for 3 spokes)

Limitation: Spoke1 ↔ Spoke2 traffic routes through Hub (extra hop, potential bottleneck).

Option 3: Hierarchical Topology (Region-based)

Region US:              Region EU:
  ┌─────┐                ┌─────┐
  │ US1 │                │ EU1 │
  ├─────┤                ├─────┤
  │ US2 │                │ EU2 │
  └─────┘                └─────┘
     ▲                        ▲
     │ Regional Peering       │
     │                        │
     └────────┬───────────────┘
          Cross-Region Peering

Structure:

US1 ↔ US2 (same region, low latency)
EU1 ↔ EU2 (same region, low latency)
US1 ↔ EU1 (cross-region, higher latency)

Benefit: Optimize local connectivity, limit cross-region traffic.

Non-Transitivity & Its Implications

VPC Peering is non-transitive:

A ↔ B (peered)
B ↔ C (peered)

Does A ↔ C work? NO!

A Pod: 192.168.1.1
C Pod: 192.168.3.1

A tries to reach C:
  Routing table A: no route to 192.168.3.0/24
  → Packet dropped

To connect A ↔ C: need explicit peering A ↔ C

Why design like this?

Network isolation boundary:
  A ↔ B: owner A explicitly allows B
  B ↔ C: owner B explicitly allows C
  
But: Owner A might not want C in their network!
  If transitive: A ↔ B ↔ C = C indirectly in A
  Risk: A assumes B is "friendly", but B might peer with "untrusted" C
  
Non-transitivity: A explicitly controls who talks to A

Production Patterns

✅ Pattern: Hub-Spoke dengan Firewall Segmentation

bash

# Hub VPC
gcloud compute networks create hub-network \
  --subnet-mode=custom \
  --bgp-routing-mode=regional

gcloud compute networks subnets create hub-subnet \
  --network=hub-network \
  --range=10.0.0.0/16

# Spoke VPCs
for i in 1 2 3; do
  gcloud compute networks create spoke-$i-network \
    --subnet-mode=custom
  
  gcloud compute networks subnets create spoke-$i-subnet \
    --network=spoke-$i-network \
    --range=10.$i.0.0/16
  
  # Peering
  gcloud compute networks peerings create hub-to-spoke-$i \
    --network=hub-network \
    --peer-network=spoke-$i-network \
    --auto-create-routes
done

# Firewall: Hub allows traffic từ tất cả spokes
gcloud compute firewall-rules create hub-allow-spokes \
  --network=hub-network \
  --allow=tcp,udp \
  --source-ranges=10.1.0.0/16,10.2.0.0/16,10.3.0.0/16

❌ Anti-Pattern: Relying on Non-Transitivity for Security

bash

# WRONG: Assuming A ↔ B + B ↔ C = C isolated from A
Cluster A ↔ Hub
Hub ↔ Cluster C (untrusted)

# If hub firewall rule allows any packet from B to hub:
# Then C can potentially reach A indirectly

Fix: Implement strict firewall rules, don't rely on routing topology

✅ Pattern: Network Policies + VPC Peering

yaml

# Cluster A Network Policy (Kubernetes-level isolation)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-network-policy
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: frontend  # Only frontend namespace
      ports:
        - protocol: TCP
          port: 8080

Layers:

VPC Peering layer: Route enables connectivity (Cluster A ↔ B)
GCP Firewall layer: Source IP ranges allowed
Kubernetes Network Policy layer: Pod-level ingress/egress rules
Service Mesh layer: SPIFFE-based authorization

Debugging Network Connectivity

Check Peering Status

bash

gcloud compute networks peerings list \
  --network=cluster-a-vpc

NAME              NETWORK               PEER_NETWORK   STATE
cluster-a-to-b    cluster-a-vpc          cluster-b-vpc  ACTIVE

Verify Route Exchange

bash

# Cluster A routing table
gcloud compute routes list \
  --filter="network:cluster-a-vpc"

NAME                              DEST_RANGE       NEXT_HOP        PRIORITY
cluster-a-default                 10.0.0.0/16      cluster-a-vpc   1000
peering-route-to-cluster-b        10.1.0.0/16      cluster-b-vpc   1000  # Via peering
peering-route-pod-cidr-cluster-b  10.1.128.0/18    cluster-b-vpc   1000

Test Connectivity

bash

# From Cluster A Pod
kubectl exec -it <pod-a> -- ping 10.1.1.20  # Cluster B pod

# If ping fails:
# 1. Check routing: kubectl exec <pod> -- ip route
# 2. Check firewall: gcloud compute firewall-rules describe allow-cluster-b
# 3. Check peering: gcloud compute networks peerings describe cluster-a-to-b --network=cluster-a-vpc

# If network reachable but service doesn't respond:
# → Check Network Policy in Cluster B
# → Check pod listening port: kubectl exec <pod-b> -- netstat -tlnp

Summary

Aspect	Implementation
Basic connectivity	VPC Peering (automatic route exchange)
Route control	Auto subnet routes, manual custom routes
Firewall	Must explicitly allow cross-VPC source IPs
Topology	Mesh (complex, all-to-all) vs Hub-Spoke (simple)
Non-transitivity	Design feature for isolation, not a bug
Defense layers	Peering → Firewall → Network Policy → Service Mesh

VPC Peering = foundation layer. Nó enables pod-to-pod connectivity. Layer trên (firewall, network policy, service mesh) provide additional security.

Network Isolation & VPC Peering ​

Tại sao VPC Peering Là Nền Tảng Của Multi-Cluster Networking ​

VPC Peering Model ​

Architecture ​

Route Exchange Mechanism ​

Automatic Route Propagation ​

Custom Route Exchange ​

Firewall Rules & Peering Boundary ​

Key Insight: Firewall Rules Don't Cross Peering ​

Topology Options ​

Option 1: Mesh Topology (All-to-All Peering) ​

Option 2: Hub-and-Spoke Topology ​

Option 3: Hierarchical Topology (Region-based) ​

Non-Transitivity & Its Implications ​

Production Patterns ​

✅ Pattern: Hub-Spoke dengan Firewall Segmentation ​

❌ Anti-Pattern: Relying on Non-Transitivity for Security ​

✅ Pattern: Network Policies + VPC Peering ​

Debugging Network Connectivity ​

Check Peering Status ​

Verify Route Exchange ​

Test Connectivity ​

Summary ​

References ​

Network Isolation & VPC Peering

Tại sao VPC Peering Là Nền Tảng Của Multi-Cluster Networking

VPC Peering Model

Architecture

Route Exchange Mechanism

Automatic Route Propagation

Custom Route Exchange

Firewall Rules & Peering Boundary

Key Insight: Firewall Rules Don't Cross Peering

Topology Options

Option 1: Mesh Topology (All-to-All Peering)

Option 2: Hub-and-Spoke Topology

Option 3: Hierarchical Topology (Region-based)

Non-Transitivity & Its Implications

Production Patterns

✅ Pattern: Hub-Spoke dengan Firewall Segmentation

❌ Anti-Pattern: Relying on Non-Transitivity for Security

✅ Pattern: Network Policies + VPC Peering

Debugging Network Connectivity

Check Peering Status

Verify Route Exchange

Test Connectivity

Summary

References