My Awesome Project

Appearance

Anthos Service Mesh Multi-Cluster — Mesh Xuyên Cluster ​

Tại Sao Quan Trọng Trong Production ​

Service mesh (Istio) ở single cluster handle: traffic management, mTLS, observability, retry logic, circuit breaking, v.v. Khi scale đến multi-cluster, service mesh cần extend ra ngoài ranh giới cluster:

  • Service A ở cluster 1 gọi Service B ở cluster 2: cần service discovery + mTLS
  • Observability: trace requests xuyên clusters
  • Traffic management: canary deployments xuyên clusters
  • Security: mutual TLS certificates xuyên clusters with different CAs

Anthos Service Mesh (ASM) là Google's managed Istio implementation cho GKE, với first-class support cho multi-cluster.

Anthos Service Mesh Components ​

Single Cluster ASM ​

Single-cluster ASM standard Istio components:

  • Istiod: Control plane, configures data plane proxies
  • Envoy sidecar proxies: Data plane, intercept pod traffic
  • Gateways: Ingress/egress traffic management

Multi-Cluster ASM — Trust Federation ​

Multi-cluster ASM thêm trust federation layer — allows different clusters have different root CAs nhưng vẫn trust lẫn nhau.

Cluster A:
├── Root CA-A (self-signed)
├── Istiod-A
└── Pods + Envoy sidecars (certs signed by CA-A)

Cluster B:
├── Root CA-B (self-signed)
├── Istiod-B
└── Pods + Envoy sidecars (certs signed by CA-B)

Trust Federation:
├── Istiod-A trust Root CA-B (via fleet configuration)
├── Istiod-B trust Root CA-A
└── Pods ở cluster A accept certificates signed by CA-B

Khác với single-cluster Istio dùng centralized CA, multi-cluster ASM cho phép operational independence: mỗi cluster manage root CA riêng, nhưng still trust hình peer-to-peer.

SPIFFE Identity ​

Multi-cluster ASM dùng SPIFFE (Secure Production Identity Framework for Everyone) làm identity model:

SPIFFE identity format: spiffe://{trust-domain}/{namespace}/{pod-sa-name}
  ↓
Cluster A: spiffe://cluster-a.my-company/payments/payment-processor
Cluster B: spiffe://cluster-b.my-company/payments/payment-processor

Trust domain là organizational identity — cùng trust domain xuyên clusters cho phép trust federation. SPIFFE certificate issued by cluster's CA nhưng trust domain consistent xuyên fleet.

Multi-Cluster Connectivity Architecture ​

Istiod Cross-Cluster Discovery ​

Một Istiod instance không biết về endpoints ở cluster khác — cần cross-cluster service discovery mechanism:

Istiod-A (cluster A):
├── Watch Kubernetes API server ở cluster A (local endpoints)
└── Watch Kubernetes API server ở cluster B (via cross-cluster reader setup)
    → Discover services + endpoints from cluster B
    → Push Envoy configs đến sidecars ở cluster A với cluster B endpoints

Setup:

  1. Create service account ở cluster B với permission read Services, Endpoints
  2. Install reader credentials (kubeconfig) đến cluster A's istiod
  3. Istiod ở cluster A use credentials để list resources từ cluster B

Reverse cũng true: Istiod-B discover endpoints từ cluster A.

Endpoint List Propagation ​

Khi pod ở cluster A create, local kube-proxy update Endpoints object. Istiod-A:

  1. Watch local Endpoints
  2. Update local Envoy configs
  3. Forward information (via cross-cluster reader) tới Istiod-B
  4. Istiod-B push updated configs tới sidecars ở cluster B

Delay: thường 5–15 giây từ pod create đến endpoint available di cluster khác (phụ thuộc watch latency).

mTLS Xuyên Cluster ​

Certificate Chain ​

Root CA-A                              Root CA-B
    ↓                                      ↓
Intermediate CA-A                   Intermediate CA-B
    ↓                                      ↓
Workload Cert (cluster A pod)     Workload Cert (cluster B pod)

mTLS connection:
Pod A → Present Workload Cert-A
         Envoy sidecar verify: signed by CA-A or trusted root
         (CA-B trust Root CA-A)
      ← Cluster B Pod present Workload Cert-B

Certificate rotation:

  • Istiod-A periodically rotate certificates cho pods ở cluster A
  • Istiod-B rotate certificates cho pods ở cluster B
  • Rotation không synchronize — pod ở cluster A cập nhật cert ở thời gian X, pod ở cluster B cập nhật cert ở thời gian Y

Khi pod cập nhật cert, existing connections từ other clusters có thể fail. Envoy sidecar implement connection pooling + retry để handle cert rotation gracefully.

Fleet-Based ASM Enrollment ​

Google simplified multi-cluster ASM setup bằng fleet. Thay vì manually configure cross-cluster readers, credentials, trust domains, dùng:

bash
# Enable ASM for fleet
gcloud container fleet mesh enable --project=FLEET_PROJECT

# Enroll cluster ở ASM (can be in-cluster or managed istiod)
gcloud container fleet mesh memberships update MEMBERSHIP \
    --management=automatic \
    --project=FLEET_PROJECT

Fleet ASM enrollment automatically:

  1. Deploy Istiod ở member clusters
  2. Configure cross-cluster discovery
  3. Setup trust federation
  4. Inject sidecars vào annotated pods

Sau enrollment, kubectl apply VirtualService/DestinationRule tới bất kỳ cluster — istiod instances coordinate traffic management.

Traffic Management Xuyên Cluster ​

VirtualService with Cross-Cluster Destination ​

yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payment-api
  namespace: payments
spec:
  hosts:
  - payment-api  # Kubernetes service name
  - payment-api.payments  # FQDN
  - payment-api.payments.svc.cluster.local
  http:
  - match:
    - uri:
        prefix: /api
    route:
    - destination:
        host: payment-api.payments.svc.cluster.local
        port:
          number: 8080
        subset: v1
      weight: 80
    - destination:
        host: payment-api.payments.svc.cluster.local
        port:
          number: 8080
        subset: v2
      weight: 20  # Canary: 20% traffic to v2
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payment-api
  namespace: payments
spec:
  host: payment-api.payments.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 100
        maxRequestsPerConnection: 2
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

Khi applied tới cluster A:

  • Envoy sidecars ở cluster A configure traffic splitting 80/20 tới v1/v2
  • v1 pods ở cluster A + cluster B serve requests
  • v2 pods (canary) ở cluster A + cluster B serve 20% requests
  • Circuit breaker: nếu pod dùng 5 consecutive 5xx errors, temporarily remove

Weight-based routing, circuit breakers, retries apply transparently xuyên clusters — tidak khác single-cluster behavior.

Locality-Aware Load Balancing ​

ASM support locality-aware LB xuyên clusters:

yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
spec:
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
        distribute:
        - from: "us-east-1/us-east-1a/*"
          to:
            "us-east-1/us-east-1a/*": 100
        - from: "us-east-1/us-east-1b/*"
          to:
            "us-east-1/us-east-1b/*": 100
        - from: "eu-west-1/*"
          to:
            "eu-west-1/*": 100
            "us-east-1/*": 0

Config prefer local endpoints — pods ở eu-west region route tới eu-west backends, fallback tới us-east chỉ khi local unavailable.

Observability Xuyên Cluster ​

Distributed Tracing ​

ASM integrate Istio tracing (dùng OpenTelemetry hoặc Jaeger) tới Cloud Trace:

Request từ cluster A pod
    ↓
Envoy sidecar generate trace ID
    ↓
Request flow: A → B pod (same cluster) → C pod (other cluster)
    ↓
Jaeger collector collect spans từ A, B, C sidecars
    ↓
Export tới Cloud Trace project (fleet host project)
    ↓
Trace view: single trace xuyên 2 clusters, tất cả request hops visible

Latency breakdown: per-hop latency visible trong trace.

Metrics Aggregation ​

ASM metrics (request rates, latencies, errors) xuyên clusters aggregated:

Prometheus scrape endpoints (sidecars, istiod) từ mọi clusters
    ↓
Aggregate tới fleet Prometheus (hoặc Cloud Monitoring)
    ↓
Dashboards: request latency distribution xuyên clusters
            error rates per service + destination cluster
            traffic volume by service pair + cluster

Constraints Và Operational Challenges ​

Eventual consistency in discovery: Endpoint propagation giữa clusters mất 5–15s. Nếu pod terminate, other clusters vẫn send traffic tới old endpoint cho tới khi discovery update. Retry logic + circuit breaker là critical.

Certificate rotation coordination: Certificates rotate independently per cluster. Cần graceful handling khi cert update — pods shouldn't drop connections instantly.

Shared control plane vs distributed: Fleet ASM dùng distributed istiod (one per cluster), không shared control plane. Nếu muốn centralized control, phải setup Istio tantra differently (advanced setup không recommend production).

Resource overhead: Envoy sidecars (dù lightweight) +Istiod (control plane) consume resources. Mỗi cluster hosting ASM cần sufficient compute.

Troubleshooting complexity: Multi-cluster ASM troubleshooting khó hơn single-cluster — need check xem endpoint propagation working, certificates valid, mTLS handshake success, trust domains match, v.v.

Anti-Pattern: Missing Circuit Breaker ​

Phổ biến mistake: enable ASM multi-cluster nhưng không setup circuit breaker/outlier detection. Khi pod bị overload ở cluster B:

  1. Cluster A keep sending traffic tới B pod (vì endpoint still in Envoy LB pool)
  2. Kết quả: cascade failure, amplified latency
  3. Without circuit breaker, failing pod drain connection pool, cause thundering herd

Always setup:

  • Outlier detection: detect failing endpoints, temporary remove
  • Connection pool limits: prevent overwhelming backend
  • Timeouts: prevent hanging connections

References ​