My Awesome Project

Appearance

Multi-Cluster Ingress (MCI) — Global Load Balancing ​

Tại sao MCI Là Cần Thiết Cho Ingress ​

MCS giải quyết service-to-service discovery (east-west traffic giữa pods). Nhưng bạn cần gì cho ingress traffic từ clients bên ngoài cluster (north-south)?

┌─────────────────────────────────┐
│         Internet Clients         │
├─────────────────────────────────┤
│                                 │
│  Client: GET http://app.example.com
│  
│  Which cluster should handle request?
│  How does GCP know target clusters?
│  How does it maintain health across clusters?
│  
└─────────────────────────────────┘

Single cluster solution: Standard Ingress + LoadBalancer service (regional LB)

Client → Regional LB → endpoints tất cả ở 1 cluster
Problem: không hay nếu cluster down

Multi-cluster solution: Multi-Cluster Ingress (MCI)

Client → Global HTTP(S) LB → anycast IP → nearest POP → 
         NEG-A (cluster-a endpoints) + NEG-B (cluster-b endpoints) →
         health-aware routing → target cluster → pod

MCI kết nối global load balancer infrastructure của GCP với multiple clusters, cho phép single virtual IP để handle traffic từ tất cả clusters, với automatic failover & locality-aware routing.

Internal Model: Cơ Chế Hoạt Động ​

Config Cluster Pattern: Star Topology ​

Multi-Cluster Ingress menggunakan config cluster model:

┌──────────────────────────────────────┐
│      Config Cluster (Cluster C)      │
│  (Defines MCI resources)             │
│                                      │
│  MultiClusterIngress resource        │
│  MultiClusterService resource        │
│                                      │
│  (Không chạy workloads)              │
└──────┬───────────────────────────────┘
       │ controls
    ┌──┴──┐
    │     │
┌───▼──┐ ┌─▼────┐
│  A   │ │  B   │
│ (members) │
└───────┘ └──────┘

Cơ chế:

  1. Config Cluster: Chứa MultiClusterIngress + MultiClusterService định nghĩa
  2. Member Clusters: Chạy Pod, expose thông qua MultiClusterService
  3. Control plane: GKE managed (tách rời các clusters), phát ngoài global LB

Lý do config cluster: Simplify management — một nơi để define global routing rules, thay vì replicate Ingress ở tất cả clusters.

Network Endpoint Groups (NEGs): Tracking Pod Endpoints ​

Vấn đề: Global LB cần biết danh sách endpoints (pod IPs) ở tất cả clusters. Nhưng LB không biết gì về Kubernetes.

Giải pháp: Network Endpoint Groups

yaml
# GCP Compute resource (created by MCI controller)
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeNetworkEndpointGroup
metadata:
  name: cluster-a-neg
  namespace: gcp
spec:
  networkEndpointType: GCE_VM_IP_PORT  # Pod IP + port
  networkRef:
    name: cluster-a-vpc
  zone: us-central1-a

Tracking mechanism:

Cluster A: ServiceExport api-service
           ↓ MCS controller
           Endpoints: 10.0.1.20:8080, 10.0.1.21:8080
           ↓
GCP Cloud: Create/update NEG
           cluster-a-neg: [10.0.1.20:8080, 10.0.1.21:8080]

Key insight: NEG là "list of endpoints" yang GCP LB bisa maintain. Không phải Kubernetes resource — pure GCP resource.

Health Checking Cross-Cluster ​

┌──────────────────────────────────┐
│       Global LB Health Check      │
├──────────────────────────────────┤
│                                  │
│  Every 5-10 seconds:             │
│  ┌──────────────────────────────┐
│  │ Cluster A endpoints:         │
│  │ GET /healthz:8080 → 10.0.1.20
│  │ GET /healthz:8080 → 10.0.1.21
│  └──────────────────────────────┘
│  
│  ┌──────────────────────────────┐
│  │ Cluster B endpoints:         │
│  │ GET /healthz:8080 → 10.1.1.20
│  │ GET /healthz:8080 → 10.1.1.21
│  └──────────────────────────────┘
│                                  │
│  Status: all healthy             │
│  → balance traffic 50/50         │
│                                  │
│  Status: cluster-b unhealthy     │
│  → route 100% to cluster-a       │
│                                  │
└──────────────────────────────────┘

Latency: ~10-15 giây từ pod down hingga LB detect + reroute.

Traffic Routing Path ​

Client (10.50.1.100):
  GET http://app.example.com

1. DNS: app.example.com → <MCI-anycast-IP> (e.g., 1.2.3.4)
   
2. BGP anycast: Client's ISP routes packet tới nearest Google PoP
   (user-centric routing → 100+ PoPs globally)
   
3. Global LB: Packet arrives at PoP
   → lookup backend service
   → check NEG-A, NEG-B health
   → select backend group (preferably same-region)
   
4. Cross-region routing (if needed):
   → packet tunnels within GCP backbone
   → arrives at cluster VPC
   
5. Cluster ingestion:
   → Ingress controller (Nginx, ALB, etc.)
   → application pod

Example:

┌──────────────────────────┐ ┌──────────────────────────┐
│   US Client (10.0.0.0)   │ │  EU Client (20.0.0.0)    │
└────────────┬─────────────┘ └──────────────┬────────────┘
             │                              │
             │ nearest PoP                  │ nearest PoP
             ▼                              ▼
        ┌─────────────┐              ┌─────────────┐
        │ US-Central  │              │ EU-West     │
        │ PoP         │              │ PoP         │
        └────┬────────┘              └────┬────────┘
             │                            │
             ▼                            ▼
        ┌─────────────┐            ┌─────────────┐
        │ Cluster-US  │            │ Cluster-EU  │
        │ NEG: healthy│            │ NEG: healthy│
        │             │            │             │
        │ Pod-1       │            │ Pod-3       │
        │ Pod-2       │            │ Pod-4       │
        └─────────────┘            └─────────────┘

Result: US client hits Cluster-US (low latency, ~10ms)
        EU client hits Cluster-EU (low latency, ~8ms)

MultiClusterIngress & MultiClusterService Resources ​

Defining Global Ingress ​

yaml
# Config Cluster: Define global routing
apiVersion: cloud.google.com/v1
kind: MultiClusterIngress
metadata:
  name: global-app
  namespace: default
spec:
  template:
    spec:
      backend:
        serviceName: api-service
        servicePort: 8080

What it does:

  1. Create global HTTP(S) load balancer
  2. Allocate global anycast IP
  3. Connect to NEGs (dari ServiceExport)
  4. Setup health checks

What it doesn't do:

  • Tidak define routing rules (use HTTPRoute dalam Gateway API untuk itu)
  • Không handle TLS termination (konfigurasi terpisah via Certificate resource)

Defining Multi-Cluster Service Targets ​

yaml
# Config Cluster: Which clusters should receive traffic?
apiVersion: net.gke.io/v1
kind: MultiClusterService
metadata:
  name: api-service
  namespace: default
spec:
  template:
    spec:
      selector:
        app: api
  ports:
    - port: 8080
      targetPort: 8080
      name: http

Mechanism:

MultiClusterService (dalam config cluster)
  ↓ Controller watches untuk ServiceExport di member clusters
  ↓ Finds Endpoints dari cluster A, B, C
  ↓ Creates NEG untuk tiap cluster
  ↓ Links NEG ke MultiClusterIngress

Traffic Splitting & Canary Deployments ​

MCI dengan NEG memungkinkan traffic splitting per-cluster:

yaml
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeBackendService
metadata:
  name: mci-backend
spec:
  backends:
    - group: projects/PROJECT/zones/us-central1-a/networkEndpointGroups/cluster-a-neg
      balancingMode: RATE
      maxRatePerEndpoint: 1000
      
    - group: projects/PROJECT/zones/us-west1-b/networkEndpointGroups/cluster-b-neg
      balancingMode: RATE
      maxRatePerEndpoint: 100  # 10x less traffic to cluster-b (canary)

Canary workflow:

1. Deploy v1.1 to cluster-b (10% capacity)
2. MCI routes 10% traffic (maxRate: 100 vs 1000)
3. Monitor error rate: if OK, scale up; if FAIL, drain v1.1
4. Gradually: 100→200→500→1000 maxRate
5. Promote: v1.1 becomes stable

Constraints & Limitations ​

Architecture Constraints ​

  • Config cluster must be member: Config cluster juga harus jadi member (tidak bisa pure control plane)
  • Same project requirement: Semua clusters harus ở same project (MCI managed resource)
  • VPC Peering required: Member clusters harus routable via VPC peering (NEG health check perlu reach pods)

Service Requirements ​

  • Headless service bukan target: MCI hanya bisa target ClusterIP (stateful)
  • Health check endpoint required: Service harus expose /healthz atau endpoint yang bisa di-health-check
  • Consistent port names: Port harus named dan consistent across clusters

Performance & Scale ​

  • NEG capacity: ~65k endpoints per backend service
  • Cluster limit: ~200+ member clusters per MCI (tested)
  • Health check latency: ~10s detection + propagation

Production Patterns ​

✅ Pattern: Global LB + Regional Failover ​

yaml
# Config cluster
apiVersion: cloud.google.com/v1
kind: MultiClusterIngress
metadata:
  name: app-ingress
spec:
  template:
    spec:
      backend:
        serviceName: app
        servicePort: 8080
---
apiVersion: net.gke.io/v1
kind: MultiClusterService
metadata:
  name: app
spec:
  template:
    spec:
      selector:
        app: api
  ports:
    - port: 8080
      name: http
---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeHealthCheck
metadata:
  name: app-health
spec:
  httpHealthCheck:
    port: 8080
    requestPath: /healthz
    checkIntervalSec: 10
    timeoutSec: 5
    healthyThreshold: 2
    unhealthyThreshold: 2

Failover behavior:

All clusters healthy: traffic 50/50
Cluster A down:      traffic 100% → Cluster B (automatic)
Cluster B down:      traffic 100% → Cluster A (automatic)
Both down:           Global LB returns 503 error

❌ Anti-Pattern: Single Cluster + MCI ​

yaml
# WRONG: Why use MCI if only one cluster?
apiVersion: cloud.google.com/v1
kind: MultiClusterIngress
spec:
  template:
    spec:
      backend:
        serviceName: app
        servicePort: 8080
# Only cluster-a in fleet

Better: Use standard Ingress + LoadBalancer service:

yaml
# RIGHT: Standard Ingress for single cluster
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
spec:
  ingressClassName: gce  # Google-managed LB
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app
                port:
                  number: 8080

Reason: MCI adds complexity (config cluster, NEGs, fleet). Untuk single cluster, standard Ingress lebih simple.

Common Issues & Debugging ​

1. MCI IP Not Assigned ​

Symptom:

bash
kubectl get mci -n default
NAME         VIP   STATUS
app-ingress  <none> PENDING

Causes:

  • Config cluster belum di-register sebagai member
  • Quota exhausted (global static IPs)
  • Network policy blocks LB

Debug:

bash
# Check config cluster status
gcloud container fleet memberships list
# Ensure config cluster di list

# Check quota
gcloud compute project-info describe --project=PROJECT \
  --format='value(quotas[name="GLOBAL_STATIC_ADDRESSES"].limit)'

# Check LB resource
gcloud compute forwarding-rules list | grep mci

2. NEG Health Check Failing ​

Symptom:

Cluster A NEG: UNHEALTHY (all endpoints)
Cluster B NEG: HEALTHY
→ Traffic 100% to cluster B (imbalanced)

Causes:

  • Pod /healthz endpoint down
  • Firewall rule blocks health check
  • VPC peering not configured (health check can't reach pod IP)

Debug:

bash
# Cluster A: simulate health check
kubectl exec -it <pod> -- curl localhost:8080/healthz

# If fails: fix pod health check endpoint

# Check firewall: GCP health checks from 35.191.0.0/16 and 130.211.0.0/22
gcloud compute firewall-rules list --filter="sourceRanges:35.191.0.0/16"

# If missing, create:
gcloud compute firewall-rules create allow-health-check \
  --direction=INGRESS \
  --priority=1000 \
  --sourceRanges=35.191.0.0/16,130.211.0.0/22 \
  --allow=tcp:8080

3. Asymmetric Routing: A→B fast, B→A slow ​

Symptom:

Client A → Cluster B: 50ms
Client B → Cluster A: 200ms

Asymmetry indicates: VPC peering one-directional or cross-region routing suboptimal

Debug:

bash
# From Cluster A pod, check latency to Cluster B pod
kubectl exec -it <pod-a> -- ping 10.1.1.20  # Cluster B pod IP
# Should be <50ms

# From Cluster B pod, check latency to Cluster A pod
kubectl exec -it <pod-b> -- ping 10.0.1.20  # Cluster A pod IP
# If >100ms, routing suboptimal

# Check VPC peering bidirectional
gcloud compute networks peerings list --network=cluster-a-vpc
gcloud compute networks peerings list --network=cluster-b-vpc
# Should see peering both directions with ACTIVE status

Summary ​

AspectImplementation
Ingress TypeGlobal HTTP(S) load balancer (GCP managed)
Backend DiscoveryNEG tracking pod endpoints per cluster
RoutingHealth-aware, locality-aware (nearest cluster first)
FailoverAutomatic, <15s detection + reroute
Traffic SplittingPer-NEG maxRate for canary deployments
ComplexityHigh (config cluster, NEG management, health checks)

Next: MCI & MCS handle north-south & east-west networking. Untuk advanced routing rules (host/path-based routing, request rewriting), gunakan Gateway API — topic selanjutnya.

References ​