Multi-Cluster Services (MCS) Deep Dive

Tại sao MCS Là Nền Tảng Của Multi-Cluster Networking

Khi bạn có hai clusters chạy application, cách duy nhất để Pod trong cluster A gửi request tới service trong cluster B là gì?

Naïve approach: DNS discovery + IP routing

Pod A: DNS lookup svc-b.ns-b.svc.cluster.local
       → IP 10.1.0.5 (cluster B pod direct IP)
       → gửi packet
       → routing table: need route tới 10.1.0.0/16
       → phải setup VPC peering + route export (manual)

MCS approach: Virtual Service + Automatic Discovery

Pod A: DNS lookup svc-b.ns-b.svc.cluster.local
       → IP 172.30.1.0 (ServiceImport VIP)
       → iptables/eBPF DNAT VIP → endpoint từ cluster B
       → routing tự động via peering

MCS là abstraction layer che giấu complexity của cross-cluster service discovery. Nó cung cấp automatic endpoint propagation, DNS integration, và locality-aware routing — tất cả những thứ cần để services trong cluster A biết cách tìm và gọi services trong cluster B.

Internal Model: Cơ Chế Hoạt Động

ServiceExport: "Export tôi muốn chia sẻ service này"

yaml

# Trong cluster B: định nghĩa service muốn expose cross-cluster
apiVersion: net.gke.io/v1
kind: ServiceExport
metadata:
  namespace: backend
  name: api-service

Cơ chế:

Cluster B's MCS controller nhìn thấy ServiceExport resource
Controller kiểm tra: "Có ServiceExport không?"
Nếu có: controller tạo service-export-derived-endpoints trong Cloud DNS
Controller export endpoint DNS records sang các clusters khác trong Fleet

Ràng buộc:

Service phải tồn tại trong cùng namespace: svc-b.ns-b.svc.cluster.local
Service phải là ClusterIP hoặc Headless (LoadBalancer không support)
Endpoints phải healthy (pods chạy)

Không tạo gì mới: ServiceExport chỉ là "flag" để nói "expose cái này". Nó không tạo virtual IP hay bất cứ thứ gì khác.

ServiceImport: "Nhập service từ cluster khác"

yaml

# Cluster A automatically cơ cấu resource này
apiVersion: net.gke.io/v1
kind: ServiceImport
metadata:
  namespace: backend
  name: api-service
spec:
  type: ClusterSetIP
status:
  clusterSetIPs:
    - 172.30.1.100  # VIP, giống như ClusterIP nhưng cross-cluster

Cơ chế tự động:

Cluster B export service → MCS controller broadcast "I have api-service"
Cluster A's MCS controller nhận thông báo
Cluster A tự động tạo ServiceImport resource trong cùng namespace
MCS controller allocate virtual IP (172.30.1.100)
Tạo iptables/eBPF rules để DNAT traffic từ VIP → endpoints từ cluster B
Register DNS name với Cloud DNS (svc-b.ns-b.svc.cluster.local → 172.30.1.100)

Namespace sameness: ServiceImport trong cluster A phải trong namespace tương ứng. Nếu service ở backend/api-service trong cluster B, thì ServiceImport cũng phải ở backend/api-service.

Endpoint Propagation: "Các Pod nào sẽ handle request?"

yaml

# Cluster B: Pod endpoints
apiVersion: v1
kind: Endpoints
metadata:
  namespace: backend
  name: api-service
subsets:
  - addresses:
      - ip: 10.1.5.20
        targetRef:
          kind: Pod
          name: api-pod-1
          namespace: backend
      - ip: 10.1.5.21
        targetRef:
          kind: Pod
          name: api-pod-2
          namespace: backend
    ports:
      - port: 8080
        name: http

Propagation flow:

Cluster B: Pod 10.1.5.20, 10.1.5.21 (endpoints)
    ↓ MCS controller watches Endpoints
Cluster B: Export via Cloud DNS
    ↓ Cloud DNS peering
Cluster A: MCS controller learns endpoint IPs
    ↓ Create iptables/eBPF DNAT rules
Cluster A: Pod 10.0.x.y → VIP 172.30.1.100 → DNAT to 10.1.5.20 or 10.1.5.21

Key insight: VIP (172.30.1.100) là local cluster address. Nó không phải là "real" IP — chỉ trigger cho iptables DNAT tới real endpoints ở cluster B subnet.

DNS Integration: "Pod tìm service thế nào?"

Without MCS:

Pod A: curl api-service.backend.svc.cluster.local
       → DNS: "tidak biết, không tìm thấy"
       → error: service not found

With MCS:

Pod A: curl api-service.backend.svc.cluster.local
       ↓ Kubelet DNS (Cloud DNS)
       → Query: api-service.backend.svc.cluster.local
       → Cloud DNS (được peer từ cluster B)
       → Response: 172.30.1.100 (ServiceImport VIP)
       → Pod route traffic tới VIP
       → Cluster A iptables/eBPF DNAT: VIP → 10.1.5.20 (cluster B endpoint)

DNS propagation model:

Cluster B Cloud DNS:
  api-service.backend.svc.cluster.local → 172.30.1.100 (VIP)
  
Cluster A Cloud DNS (peered):
  api-service.backend.svc.cluster.local → 172.30.1.100 (VIP)
  
When Cluster B endpoint changes:
  → Cloud DNS updated (10s)
  → Cluster A pods see change (next DNS query, cached after TTL)

TTL implications: Default TTL là 30 giây. Nếu pod ở cluster A cache DNS entry, nó sẽ timeout & refresh setiap 30 giây. Pod baru yang scale up không segera biết tentang endpoint change.

Locality-Aware Load Balancing

Vấn đề: Nếu có 2 clusters, cluster A có 10 pods, cluster B có 2 pods, traffic bisa jadi imbalanced:

Cluster A: 10 pods
Cluster B: 2 pods

Without locality awareness:
  iptables round-robin: Pod-A1, Pod-A2, ..., Pod-A10, Pod-B1, Pod-B2
  → Pod-B1, Pod-B2 overwhelmed (5x traffic)
  
With locality awareness:
  Prefer Pod-A1-10 (local cluster)
  → Cross-cluster Pod-B1, Pod-B2 hanya jika lokal pods down

Implementation:

MCS supports locality-aware load balancing melalui topology.kubernetes.io/zone label:

yaml

# Cluster A pod
apiVersion: v1
kind: Pod
metadata:
  labels:
    topology.kubernetes.io/zone: us-central1-a

Preference order:

Same node (best latency, < 1ms)
Same zone (inter-zone latency, ~10ms)
Same region (inter-region latency, ~50-100ms)
Different region (cross-region, >100ms)

Konfigurasi di ServiceImport:

yaml

apiVersion: net.gke.io/v1
kind: ServiceImport
metadata:
  namespace: backend
  name: api-service
spec:
  type: ClusterSetIP
  sessionAffinity: ClientIP  # Optional: sticky sessions
  sessionAffinityConfig:
    clientIPConfig:
      timeoutSeconds: 10800

Endpoint Health Checking

Vấn đề: Một pod ở cluster B crash hoặc become unhealthy. Cluster A pods vẫn cố send traffic tới nó.

Solution: MCS controller ngamatkan endpoint health:

Cluster B: Kubelet probe Pod (readiness/liveness)
           → Pod unhealthy
           → kubelet remove từ Endpoints
           
Cluster B: MCS controller watch Endpoints change
           → endpoint removed
           → broadcast change via Cloud DNS
           
Cluster A: MCS controller notified
           → remove từ load balancing pool
           → traffic hanya ke healthy pods

Latency: ~10 giây (health check interval) + DNS propagation.

Eventual Consistency Model

MCS tidak memberikan strong consistency. Ada delay dalam endpoint propagation:

t=0s:   Pod-B1 crash
        
t=5s:   Cloud DNS updated
        
t=10s:  Cluster A's kube-dns updated
        
t=15s:  Existing connections drain, new connection route ke healthy pods

t=30s:  DNS cache expire, pods biết endpoint change

Implikasi:

Requests dapat fail selama 10-30s setelah pod crash
Tidak cocok untuk synchronous request yang tidak bisa retry
Cocok untuk async, eventual-consistency-friendly workloads

Constraints & Limitations

Network Requirements

VPC Peering: Clusters harus bisa reach each other via VPC peering
DNS Peering: Cloud DNS zones harus di-peer cross-cluster
Firewall rules: Ingress rules harus allow port 8080 (atau service port) dari peered VPCs

Service Requirements

Service type: ClusterIP atau Headless (LoadBalancer tidak support)
Port names: Port must be named: http, tcp, grpc (unnamed ports tidak propagate)
Namespace sameness: If service di cluster B di backend/api, ServiceImport harus di backend/api (same ns)
Cluster label requirement: Pods harus punya label gke.io/cluster

Operational Constraints

No cross-namespace import: ServiceImport tidak bisa di namespace berbeda
No dynamic service discovery: Service list tidak publish ke client (hanya individual services)
No shared state caching: Jika Pod di cluster A cache Endpoints, cache tidak invalidate otomatis

Scale Limits

Per-cluster: ~5000 services, ~100k endpoints
Cross-cluster: Performance degrades dengan lebih banyak clusters (~20+ clusters)

Common Failure Modes & Debugging

1. ServiceImport tidak muncul di Cluster A

Penyebab:

Cluster B: ServiceExport terbuat, but endpoint EMPTY
           (Pods belum schedule / readiness probe gagal)
           
→ MCS controller "sees" ServiceExport tapi no endpoints
→ tidak broadcast ke cluster A
→ Cluster A: ServiceImport tidak muncul

Debug:

bash

# Cluster B
kubectl get svc -n backend api-service
# View ENDPOINTS column: jika <none>, pods unhealthy

kubectl get endpoints -n backend api-service
# View EndpointSlices: jika kosong, pods belum ready

# Cluster A
kubectl get serviceimport -n backend
# Jika tidak ada api-service, check:
#   - Cluster B memiliki pods?
#   - Pods readiness probe OK?
#   - VPC peering aktif?

2. DNS Resolution Gagal

Penyebab:

Cluster A Pod: curl api-service.backend.svc.cluster.local
       → DNS query ke Cloud DNS
       → Cloud DNS tidak tahu (peering not configured)
       → error: NXDOMAIN

Debug:

bash

# Cluster A Pod
kubectl exec -it <pod> -- nslookup api-service.backend.svc.cluster.local

# Expected: 172.30.1.100 (ServiceImport VIP)
# Actual: NXDOMAIN

# Fix: Check DNS peering
gcloud dns managed-zones describe cluster-a-zone
# View "nameServerSet" dan "dnsSecurity"

# Check peering
gcloud dns managed-zones list-dns-security-policies cluster-a-zone

3. Traffic Reroutes Lambat Setelah Failover

Penyebab:

Pod di cluster B crash
→ Endpoints updated (5s)
→ Cloud DNS updated (5s)
→ Cluster A DNS cache TTL expiry (30s)
→ Pod baru query, get answer (0.5s)
→ Total: ~40 seconds before reroute!

Existing connection: tetap terbuka ke dead pod (timeout after app-level timeout)

Mitigasi:

yaml

# Lower DNS TTL (CloudDNS, default 30s)
# Not directly configurable via API, hardcoded

# Better: App-level resilience
# - Connection timeout: 10s (detect dead pod faster)
# - Retry logic: exponential backoff
# - Circuit breaker: fail fast after 3 failures

# Example: Client library
client = requests.Session()
retry_strategy = Retry(
    total=3,
    backoff_factor=1,
    status_forcelist=[500, 502, 503, 504]
)
adapter = HTTPAdapter(max_retries=retry_strategy)
client.mount("http://", adapter)

Production Patterns

✅ Pattern: MCS + Topology Spread

yaml

# Cluster A + Cluster B
apiVersion: apps/v1
kind: Deployment
metadata:
  name: consumer
spec:
  replicas: 10
  selector:
    matchLabels:
      app: consumer
  template:
    metadata:
      labels:
        app: consumer
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: consumer
      containers:
        - name: app
          image: consumer:v1
          env:
            - name: BACKEND_SERVICE
              value: "api-service.backend.svc.cluster.local"
            # DNS auto-discovery: no hardcoded cluster labels

✅ Pattern: Readiness Probe untuk Quick Failover

yaml

# Backend Pod (Cluster B)
apiVersion: v1
kind: Pod
spec:
  containers:
    - name: api
      readinessProbe:
        httpGet:
          path: /healthz
          port: 8080
        initialDelaySeconds: 5
        periodSeconds: 5  # Check every 5s
        failureThreshold: 2
        timeoutSeconds: 2

Lý do: Quick readiness detection → fast endpoint removal → fast failover.

❌ Anti-Pattern: LoadBalancer Service Type

yaml

# WRONG: MCS doesn't support LoadBalancer
apiVersion: v1
kind: Service
metadata:
  name: api-service
spec:
  type: LoadBalancer  # ❌ Won't export across clusters
  ports:
    - port: 80
      targetPort: 8080

Use ClusterIP instead:

yaml

apiVersion: v1
kind: Service
metadata:
  name: api-service
spec:
  type: ClusterIP  # ✓ Correct
  ports:
    - port: 8080
      targetPort: 8080
      name: http  # ✓ Named port (required)

Summary

Aspect	Implementation
Service Export	Manual ServiceExport resource (flag)
Service Discovery	Automatic ServiceImport + DNS
Endpoint Propagation	MCS controller watches, updates via Cloud DNS
Load Balancing	iptables/eBPF DNAT to multi-cluster endpoints
Consistency	Eventual consistency, 10-30s propagation lag
Health Checking	Readiness probes → Endpoints → DNS update
Locality	Prefer same zone, fallback cross-region
Troubleshooting	Check Endpoints, DNS query, VPC peering, Cloud DNS

Next: MCS cung cấp service-to-service discovery. Untuk ingress traffic (clients ngoài), gunakan Multi-Cluster Ingress (MCI) — yang kita pelajari di chapter berikutnya.

Multi-Cluster Services (MCS) Deep Dive ​

Tại sao MCS Là Nền Tảng Của Multi-Cluster Networking ​

Internal Model: Cơ Chế Hoạt Động ​

ServiceExport: "Export tôi muốn chia sẻ service này" ​

ServiceImport: "Nhập service từ cluster khác" ​

Endpoint Propagation: "Các Pod nào sẽ handle request?" ​

DNS Integration: "Pod tìm service thế nào?" ​

Locality-Aware Load Balancing ​

Endpoint Health Checking ​

Eventual Consistency Model ​

Constraints & Limitations ​

Network Requirements ​

Service Requirements ​

Operational Constraints ​

Scale Limits ​

Common Failure Modes & Debugging ​

1. ServiceImport tidak muncul di Cluster A ​

2. DNS Resolution Gagal ​

3. Traffic Reroutes Lambat Setelah Failover ​

Production Patterns ​

✅ Pattern: MCS + Topology Spread ​

✅ Pattern: Readiness Probe untuk Quick Failover ​

❌ Anti-Pattern: LoadBalancer Service Type ​

Summary ​

References ​

Multi-Cluster Services (MCS) Deep Dive

Tại sao MCS Là Nền Tảng Của Multi-Cluster Networking

Internal Model: Cơ Chế Hoạt Động

ServiceExport: "Export tôi muốn chia sẻ service này"

ServiceImport: "Nhập service từ cluster khác"

Endpoint Propagation: "Các Pod nào sẽ handle request?"

DNS Integration: "Pod tìm service thế nào?"

Locality-Aware Load Balancing

Endpoint Health Checking

Eventual Consistency Model

Constraints & Limitations

Network Requirements

Service Requirements

Operational Constraints

Scale Limits

Common Failure Modes & Debugging

1. ServiceImport tidak muncul di Cluster A

2. DNS Resolution Gagal

3. Traffic Reroutes Lambat Setelah Failover

Production Patterns

✅ Pattern: MCS + Topology Spread

✅ Pattern: Readiness Probe untuk Quick Failover

❌ Anti-Pattern: LoadBalancer Service Type

Summary

References