Multi-Cluster Ingress & Multi-Cluster Gateway — Global Load Balancing

Tại Sao Quan Trọng Trong Production

MCS giải quyết east-west traffic (service-to-service xuyên clusters). North-south traffic (internet → backend services) cần different approach. Multi-Cluster Ingress (MCI) là legacy solution, Multi-Cluster Gateway (MCG) là modern approach — cả hai dùng GCP's Global External Application Load Balancer để distribute internet traffic xuyên multiple clusters.

Multi-Cluster Ingress — Legacy Pattern

Architecture

MCI dùng config cluster pattern: một cluster được designate là config cluster — trung tâm điều hành global load balancer config. MCI resources (MultiClusterIngress, MultiClusterService) chỉ có effect khi deployed trên config cluster; deployment lên member clusters sẽ bị ignore.

fleet-topology:
├── config-cluster (us-central1)
│   ├── MultiClusterIngress backend-ingress
│   ├── MultiClusterService backend-svc
│   └── All MCI traffic control configuration
├── member-cluster-1 (us-east1)
│   └── Pods serving requests (khỏi cần MCI/MCS resources)
└── member-cluster-2 (eu-west1)
    └── Pods serving requests

MultiClusterService (MCS CRD)

Khác với ServiceImport từ Multi-Cluster Services, MCI dùng riêng MultiClusterService CRD:

apiVersion: compute.gke.io/v1
kind: MultiClusterService
metadata:
  name: backend-service
  namespace: default
spec:
  selector:
    app: backend
  template:
    spec:
      ports:
      - name: http
        port: 80
        targetPort: 8080
  clusters:
  - name: member-cluster-1
  - name: member-cluster-2
  # Omit 'clusters' để match tất cả member clusters

MultiClusterService selector match pods xuyên fleet, tương tự như Service nhưng scope across clusters.

MultiClusterIngress (MCI CRD)

apiVersion: compute.gke.io/v1
kind: MultiClusterIngress
metadata:
  name: backend-ingress
  namespace: default
  annotations:
    ingress.gke.io/pre-shared-cert: my-ssl-cert  # Global SSL cert
spec:
  template:
    spec:
      ingressClassName: gce
      rules:
      - host: api.example.com
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: backend-service
                port:
                  number: 80
      - host: admin.example.com
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: admin-service
                port:
                  number: 8080

MCI controller tạo Google Cloud Load Balancer resource:

• Global static IP
• URL maps for host/path routing
• Backend services referencing NEGs từ member clusters
• Health checks

NEG Creation Across Clusters

Khi MCI được applied:

1. Config cluster MCI controller identify member clusters từ MultiClusterService.spec.clusters
2. Deploy NEG controller agent vào member clusters (nếu chưa có)
3. NEG agent trong member clusters tạo Network Endpoint Groups (NEGs) từ pods matching selector
4. MCI controller configure GCLB backend services để point tới những NEGs

Pod (member-cluster-1, us-east1)
  ↓
NEG Agent watches pod creation
  ↓
Create NEG endpoint entry: {pod-ip}:{port}
  ↓
MCI Controller aggregate endpoints từ tất cả NEGs
  ↓
GCLB backend service reference NEGs
  ↓
Traffic from internet → GCLB → nearest cluster → pod

Traffic steering:

• GCLB dùng locality information (pod's region/zone) để prefer closest endpoints
• Health checks từ GCLB lên NEG endpoints — nếu pod unresponsive, remove từ GCLB backend

Namespace Sameness Requirement

MultiClusterService phải tồn tại ở cùng namespace trên tất cả member clusters:

• Config cluster: default/backend-service (MCS resource chỉ exist ở config cluster)
• Member clusters: phải có Pods với label app: backend ở namespace default

Namespace mismatch = pods không được included ở MCS endpoint list.

Multi-Cluster Gateway — Modern Approach

MCG là newer generation dùng Gateway API (Kubernetes-native, CNCF standard) thay vì proprietary MCI CRD. MCG nhằm thay thế MCI, mặc dù MCI vẫn work cho backward compatibility.

GatewayClass gke-l7-global-external-managed-mc

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: global-gateway
  namespace: default
spec:
  gatewayClassName: gke-l7-global-external-managed-mc
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - name: my-cert
        kind: Secret
    allowedRoutes:
      namespaces:
        from: All
  - name: http
    port: 80
    protocol: HTTP
---
apiVersion: v1
kind: Secret
metadata:
  name: my-cert
type: kubernetes.io/tls
data:
  tls.crt: ...
  tls.key: ...
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: default
spec:
  parentRefs:
  - name: global-gateway
  hostnames:
  - api.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: backend-service
      kind: Service
      port: 80
      weight: 100
  - matches:
    - path:
        type: PathPrefix
        value: /admin
    backendRefs:
    - name: admin-service
      kind: Service
      port: 8080
      weight: 100

HTTPRoute reference Service names — chỉ name, không cần MultiClusterService CRD. Gateway Controller tự động discover services xuyên clusters (dùng MCS ServiceImport internally).

Multi-Cluster Routing With HTTPRoute

MCG dùng HTTPRoute để define routing rules xuyên clusters:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: canary-route
spec:
  parentRefs:
  - name: global-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /new-feature
    backendRefs:
    - name: backend-service-v2
      namespace: default
      port: 80
      weight: 10  # 10% traffic
    - name: backend-service-v1
      namespace: default
      port: 80
      weight: 90  # 90% traffic

Weight-based routing distribute traffic theo percentage — useful cho canary deployments xuyên clusters.

Sự Khác Biệt MCI vs MCG

Aspect	MCI	MCG
API	Compute MCI CRD	Gateway API (standard Kubernetes)
Config Cluster	Bắt buộc	Không cần explicit designation
Service Reference	MultiClusterService	Service + MCS ServiceImport (transparent)
Routing Capabilities	Host/path based	Host/path, weighted, header match, regex
Traffic Splitting	Limited	Native weight-based traffic split
TLS Management	Annotation-based	Gateway Listener + Secret ref
Status Reporting	MCI status	HTTPRoute + Gateway status
Backward Compat	Deprecated	Recommended new default

Health Checks Xuyên Clusters

Cả MCI và MCG dùng Container-Native Load Balancing — health checks từ GCLB trực tiếp đến pod IPs, không qua kube-proxy.

GCLB Health Check (from Google edge)
  ↓
GET http://{pod-ip}:{port}/healthz
  ↓
Pod liveness probe cùng health check logic
  ↓
Return 200 OK → endpoint considered healthy
  ↓
Include trong load balancer backend

Health check configuration:

• Check interval: mặc định 10s
• Timeout: 5s
• Healthy threshold: 2 successful checks
• Unhealthy threshold: 3 failed checks

Khi pod marked unhealthy, GCLB remove từ backend — requests dán lại cluster khác.

Cross-cluster implications: Health check latency từ Google edge cloud tới pod có thể là 50–200ms tuỳ region. Long-running health check endpoint (ví dụ expensive database query) có thể timeout, mark pod unhealthy, cause flapping.

Config vs Member Clusters — Traffic Flow

Internet Client → Google Front End (GFE) @ edge
  ↓
Global Load Balancer (GCLB) chooses backend cluster
  (dựa vào proximity + health)
  ↓
GCLB forwards request → cluster's edge (Google Private Service Connection)
  ↓
Cluster's container-native LB choose pod từ NEG
  ↓
Pod process request

Traffic không flow qua config cluster. Config cluster chỉ role là "control plane configuration" — quyết định load balancer setup. Member clusters chính là nơi serve requests.

Nếu config cluster crash, GCLB vẫn route traffic bình thường — LB config đã được provisioned.

Constraints Và Operational Considerations

Namespace Sameness: Pods phục vụ load balancer phải được deployed trên exact same namespace xuyên clusters. Namespace mismatch = pods không được included.

HTTPS requires pre-allocated IP: Để dùng HTTPS, phải allocate static IP beforehand. gcloud compute addresses create global address trước khi apply Ingress/Gateway resource.

gRPC Requirements: MCG support gRPC nhưng cần:

• HTTP/2 between client và GCLB (always true)
• HTTP/2 between GCLB và backend (require traefik/envoy sidecar hoặc mTLS between LB and backend)
• mTLS disabled khi dùng MCI (potential security implication)

Rate Limiting: Cloud Armor integration (WAF) available nhưng rate limiting là per-GCLB instance, không distributed xuyên clusters.

Session Affinity: ClientIP affinity route lại client tới same backend pod — useful để preserve session state. Nhưng pod scale/restart break affinity.

When to Use MCI vs MCG vs LoadBalancer Service

Use MCG (Multi-Cluster Gateway):

• ✅ Internet-facing multi-cluster service
• ✅ Want standard Kubernetes Gateway API
• ✅ Need traffic splitting/weighted routing
• ✅ Want future-proof solution

Use MCI (Multi-Cluster Ingress):

• ✅ Existing setup already using MCI
• ✅ Backward compatibility requirement
• ⚠️ Avoid for new deployments — MCG is recommended

Use LoadBalancer Service type:

• ✅ Single-cluster exposure
• ✅ Non-HTTP protocols (TCP, UDP)
• ✅ Simple use case, don't need global routing

Use Service Mesh (Istio):

• ✅ Need advanced traffic management (circuit breaking, retries)
• ✅ Cross-namespace dependencies
• ✅ Detailed observability (per-service metrics)
• ✅ Already invested in mesh

References

• Multi-Cluster Ingress
• Multi-Cluster Gateway API
• Container-Native Load Balancing
• Gateway API