DNS Peering Across Clusters

Tại sao DNS Peering Là Cần Thiết

Vấn đề: Mỗi cluster có DNS zone riêng.

Cluster A DNS:
  api.backend.svc.cluster.local → 172.30.1.0 (ServiceImport VIP)

Cluster B DNS:
  api.backend.svc.cluster.local → 10.1.1.20 (local endpoint)

Pod ở Cluster A: curl api.backend.svc.cluster.local
  → query Cluster A DNS
  → get 172.30.1.0 (cross-cluster VIP) ✓
  
Pod ở Cluster B: curl api.backend.svc.cluster.local
  → query Cluster B DNS
  → get 10.1.1.20 (local endpoint) ✓

Nhưng nếu pod B muốn call api từ cluster A?

Pod B (Cluster B): curl api.backend.svc.cluster.local
  → query Cluster B's DNS
  → Cluster B doesn't know about Cluster A's ServiceImport
  → returns local endpoint 10.1.1.20
  
Problem: Pod B trying to reach Cluster A service, but gets local Cluster B endpoint
Result: Connection timeout or 404 (wrong service)

Solution: DNS Peering

Cluster A DNS Zone ↔ peer → Cluster B DNS Zone

Now:
Pod B query: api.backend.svc.cluster.local
  → Cluster B's resolver
  → delegates to Cluster A's zone (peering)
  → gets 172.30.1.0 (Cluster A endpoint)
  → Pod B can reach Cluster A service

Cloud DNS Peering Model

Per-Cluster Private Zones

┌────────────────────────────────┐
│    GCP Cloud DNS               │
├────────────────────────────────┤
│                                │
│  Cluster A Zone:               │
│  Name: backend.svc.cluster.    │
│  Type: Private (VPC attached)  │
│  Records:                      │
│    api → 172.30.1.0            │
│    web → 172.30.2.0            │
│  VPC bound: cluster-a-vpc      │
│                                │
│  Cluster B Zone:               │
│  Name: backend.svc.cluster.    │
│  Type: Private (VPC attached)  │
│  Records:                      │
│    api → 10.1.1.20             │
│    web → 10.1.2.20             │
│  VPC bound: cluster-b-vpc      │
│                                │
│  Peering:                       │
│  Cluster A zone ↔ Cluster B zone│
│  (enables cross-zone queries)  │
│                                │
└────────────────────────────────┘

Key: Mỗi cluster có zone riêng, nhưng peering allows resolution across zones.

Creating Peering

bash

# Cluster A: Create private DNS zone (auto-created by GKE)
gcloud dns managed-zones create cluster-a-zone \
  --dns-name=backend.svc.cluster.local. \
  --networks=cluster-a-vpc \
  --private-zone

# Cluster B: Create private DNS zone
gcloud dns managed-zones create cluster-b-zone \
  --dns-name=backend.svc.cluster.local. \
  --networks=cluster-b-vpc \
  --private-zone

# Setup peering: Cluster A zone ↔ Cluster B zone
gcloud dns managed-zones describe cluster-a-zone

# Get Cluster A's nameservers (private zone)
nameServers:
  - ns-cloud-a1.googledomains.com.
  - ns-cloud-a2.googledomains.com.

# Create peering policy on Cluster B zone
gcloud dns policies create cluster-b-peering-policy \
  --networks=cluster-b-vpc

gcloud dns policies rules create \
  --policy=cluster-b-peering-policy \
  --dns-name=backend.svc.cluster.local. \
  --alternative-name-servers=ns-cloud-a1.googledomains.com.,ns-cloud-a2.googledomains.com.
  # Now Cluster B zone delegates backend.svc.cluster.local to Cluster A nameservers

Record Types & Resolution Path

A Records (IPv4)

Cluster A DNS Zone:
  api.backend.svc.cluster.local.  A  172.30.1.0

Resolution path (from Cluster B Pod):
  1. Pod: getaddrinfo("api.backend.svc.cluster.local")
  2. Kubelet DNS (kube-dns/CoreDNS): 
     cache check: miss
  3. Cloud DNS (Cluster B zone):
     lookup "api.backend.svc.cluster.local"?
     local zone: not found
     → check peering policy
     → delegate to Cluster A nameservers
  4. Cluster A Cloud DNS:
     lookup "api.backend.svc.cluster.local"
     → return 172.30.1.0
  5. Kubelet DNS cache: 172.30.1.0
  6. Pod: receives 172.30.1.0 (Cluster A ServiceImport VIP)

Latency: ~100-200ms (including peering lookup).

SRV Records (Service Discovery)

Cluster A DNS Zone:
  _http._tcp.api.backend.svc.cluster.local.  SRV  0 0 8080 api.backend.svc.cluster.local.

Use case: Service mesh (Envoy) queries SRV records to discover endpoints + ports

Endpoint Updates & Propagation

Scenario: Pod ở Cluster A crashes.

t=0:   Pod crash
       Kubelet readiness probe fail

t=2s:  Kubernetes Endpoints updated
       Endpoints: remove crashed pod IP

t=5s:  MCS controller (Cluster A)
       watch Endpoints change
       update Cloud DNS (remove endpoint from SRV record)

t=10s: Cluster B Cloud DNS
       pull zone update (via peering)

t=15s: Cluster B pod query:
       get updated endpoints (crashed pod excluded)

Total propagation: ~15 seconds

TTL & Caching Implications

Default TTL: 30 seconds

Cluster A DNS: api.backend.svc.cluster.local. 30 IN A 172.30.1.0

Cluster B Pod query at t=0s:
  → receive 172.30.1.0 with TTL 30s
  → kubelet DNS caches for 30s

Cluster A pod crashes at t=15s:
  → Cluster A DNS remove endpoint
  → but Cluster B DNS cache still valid (TTL 30-15=15s remaining)
  → Cluster B pod still send traffic to dead endpoint

t=30s:  DNS cache expire, new query
        → get updated endpoint (or error if no endpoints)

Result: 15-30 second "black hole" period

Mitigation: Lower TTL

bash

gcloud dns record-sets update \
  api.backend.svc.cluster.local. \
  --zone=cluster-a-zone \
  --ttl=10  # Lower from 30 to 10 seconds

Trade-off:
  Benefit: Faster failover (10s vs 30s)
  Cost: More DNS queries, slightly higher latency

Headless Services & Endpoints

Use case: Pod directly resolve to endpoint IPs (tidak through VIP).

yaml

# Cluster A: Headless service
apiVersion: v1
kind: Service
metadata:
  name: api-headless
  namespace: backend
spec:
  clusterIP: None  # Headless!
  selector:
    app: api
  ports:
    - port: 8080
      name: http

DNS response:

Cluster A DNS:
  api-headless.backend.svc.cluster.local.  A  10.0.1.20
  api-headless.backend.svc.cluster.local.  A  10.0.1.21
  api-headless.backend.svc.cluster.local.  A  10.0.1.22

Pod query → get list semua pod IPs (tidak VIP)

Benefit: Client-side load balancing (pod choose target, not LB).

Peering: Cluster B can query & get Cluster A pod endpoints directly.

Failover Patterns

Pattern 1: DNS Round-Robin (Simple)

Cluster A: api.backend.svc.cluster.local → 172.30.1.0
Cluster B: api.backend.svc.cluster.local → 10.1.1.20

If Cluster A down:
  MCS controller remove ServiceImport endpoint
  Cluster A DNS: no response (NXDOMAIN)
  Cluster B pod: fallback to Cluster B endpoint (10.1.1.20)
  
Trade-off:
  Simple: no complex failover logic
  Slow: 10-30s detection + TTL wait

Pattern 2: Weighted DNS (Load Balancing)

Cluster A: api.backend.svc.cluster.local → 172.30.1.0 (weight 70)
Cluster B: api.backend.svc.cluster.local → 10.1.1.20 (weight 30)

Implementation: Cloud DNS Policy with weighted records
(Not standard DNS; custom orchestration needed)

Pattern 3: Geo-Failover

Cluster A (us-central): api → 172.30.1.0
Cluster B (us-west):    api → 10.1.1.20

Pod in us-central: prefer 172.30.1.0 (local)
Pod in us-west:    prefer 10.1.1.20 (local)

If Cluster A down: us-west queries → fallback to us-west endpoint

Production Patterns

✅ Pattern: DNS Peering + Health Checks

bash

# Cluster A zone with health-aware records
# (requires external health check endpoint)

gcloud dns record-sets transaction start --zone=cluster-a-zone

gcloud dns record-sets transaction add 172.30.1.0 \
  --name=api.backend.svc.cluster.local. \
  --ttl=10 \
  --type=A \
  --zone=cluster-a-zone

# Add health routing: if endpoint unhealthy, don't return
gcloud dns record-sets transaction add 172.30.2.0 \
  --name=api.backend.svc.cluster.local. \
  --ttl=10 \
  --type=A \
  --routing-policy-type=geolocation \
  --zone=cluster-a-zone

gcloud dns record-sets transaction execute --zone=cluster-a-zone

❌ Anti-Pattern: Hardcoding Cluster B's DNS Zone

yaml

# WRONG: Application hardcodes cluster B zone
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  api-endpoint: api.cluster-b.svc.cluster.local  # ❌ Hardcoded!

Problem:

If Cluster B down, no failover to Cluster A
Tight coupling to cluster topology
Hard to test

Better:

yaml

# ✓ Use DNS peering, let resolution find endpoint
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  api-endpoint: api.backend.svc.cluster.local  # Peered zone

Debugging DNS Resolution

Check Peering Status

bash

# Cluster B: verify peering to Cluster A zone
gcloud dns policies describe cluster-b-peering-policy

# Should show:
rules:
  - dns-name: backend.svc.cluster.local.
    alternative-name-servers:
      - ns-cloud-a1.googledomains.com.
      - ns-cloud-a2.googledomains.com.

Trace Resolution Path

bash

# From Cluster B Pod
kubectl exec -it <pod-b> -- nslookup -d api.backend.svc.cluster.local

# Expected output shows:
# 1. Query sent to local nameserver (kube-dns:53)
# 2. Kube-dns delegates to Cloud DNS (169.254.169.254)
# 3. Cloud DNS peering: delegates to Cluster A zone
# 4. Response: 172.30.1.0 (Cluster A endpoint)

Check DNS Records

bash

# Cluster A: verify records exist
gcloud dns record-sets list --zone=cluster-a-zone \
  --filter="name:api.backend.svc.cluster.local."

NAME                               TYPE TTL DATA
api.backend.svc.cluster.local.     A    10  172.30.1.0

# Cluster B: query peered zone
gcloud dns query api.backend.svc.cluster.local. \
  --zone=cluster-b-zone \
  --recursive

# Should return 172.30.1.0 (from Cluster A zone via peering)

Summary

Aspect	Implementation
Per-cluster zones	Private Cloud DNS zones bound to VPCs
Peering mechanism	Nameserver delegation via DNS policies
Resolution path	Local → Cloud DNS → peered zone → response
Record types	A (endpoints), SRV (service discovery)
TTL	Default 30s, can tune down for faster failover
Failover speed	10-30s (health check + TTL wait)
Headless services	Direct endpoint resolution cross-cluster

DNS Peering = critical for service discovery. Nó enables pods across clusters to find services via standard DNS queries, without needing application-level service registries.

Trade-off: Eventual consistency (TTL delays) vs. simplicity (standard DNS, no special tooling).

DNS Peering Across Clusters ​

Tại sao DNS Peering Là Cần Thiết ​

Cloud DNS Peering Model ​

Per-Cluster Private Zones ​

Creating Peering ​

Record Types & Resolution Path ​

A Records (IPv4) ​

SRV Records (Service Discovery) ​

Endpoint Updates & Propagation ​

TTL & Caching Implications ​

Mitigation: Lower TTL ​

Headless Services & Endpoints ​

Failover Patterns ​

Pattern 1: DNS Round-Robin (Simple) ​

Pattern 2: Weighted DNS (Load Balancing) ​

Pattern 3: Geo-Failover ​

Production Patterns ​

✅ Pattern: DNS Peering + Health Checks ​

❌ Anti-Pattern: Hardcoding Cluster B's DNS Zone ​

Debugging DNS Resolution ​

Check Peering Status ​

Trace Resolution Path ​

Check DNS Records ​

Summary ​

References ​

DNS Peering Across Clusters

Tại sao DNS Peering Là Cần Thiết

Cloud DNS Peering Model

Per-Cluster Private Zones

Creating Peering

Record Types & Resolution Path

A Records (IPv4)

SRV Records (Service Discovery)

Endpoint Updates & Propagation

TTL & Caching Implications

Mitigation: Lower TTL

Headless Services & Endpoints

Failover Patterns

Pattern 1: DNS Round-Robin (Simple)

Pattern 2: Weighted DNS (Load Balancing)

Pattern 3: Geo-Failover

Production Patterns

✅ Pattern: DNS Peering + Health Checks

❌ Anti-Pattern: Hardcoding Cluster B's DNS Zone

Debugging DNS Resolution

Check Peering Status

Trace Resolution Path

Check DNS Records

Summary

References