Cross-Cluster Service Mesh & Trust Federation

Tại sao Service Mesh Cần Cross-Cluster Support

Service mesh (Istio/ASM) cung cấp:

mTLS: Pod-to-pod encryption + mutual authentication
Observability: Automatic tracing, metrics, access logs
Traffic management: Circuit breaker, retry, timeout
Policy enforcement: Rate limiting, authorization policies

Nhưng trong multi-cluster:

Cluster A Pod A → (traffic)  → Cluster B Pod B
                 (question: bao giờ encrypted?)
                 (question: B trust A's certificate?)
                 (question: authorization policy A→B?)

Service mesh federation = extending mTLS & authorization across cluster boundaries.

mTLS Federation: Root CA & Certificate Distribution

The Trust Model

┌──────────────────────────────────────────────┐
│  Shared Root CA (Kubernetes cluster)          │
├──────────────────────────────────────────────┤
│                                              │
│  ┌────────────────────────────────────────┐ │
│  │ Cluster A: Intermediate CA             │ │
│  │ (signed by root)                       │ │
│  │                                        │ │
│  │ Pod A: certificate (signed by int-ca) │ │
│  └────────────────────────────────────────┘ │
│                                              │
│  ┌────────────────────────────────────────┐ │
│  │ Cluster B: Intermediate CA             │ │
│  │ (signed by root)                       │ │
│  │                                        │ │
│  │ Pod B: certificate (signed by int-ca) │ │
│  └────────────────────────────────────────┘ │
│                                              │
└──────────────────────────────────────────────┘

mTLS handshake (Pod A ↔ Pod B):
1. Pod A sends cert (signed by cluster-a int-ca, validated via root)
2. Pod B verifies: cert chain valid? (int-ca → root)
3. If valid: establish TLS connection
4. Same for Pod B → Pod A (mutual TLS)

Requirement: Cả hai clusters phải trust cùng root CA. Nếu root CA khác → TLS handshake fails.

Certificate Distribution Mechanism

Scenario: Cluster A Pod A muốn call Cluster B Pod B melalui mTLS.

1. Pod A:
   - Needs certificate
   - Needs root CA public key để verify Pod B cert
   
2. Certificate issuance:
   - Istiod (control plane) watches ServiceExport
   - When Pod B exported, Istiod learns about it
   - Istiod contacts root CA (Kubernetes cluster)
   - Root CA issues short-lived certificate để Pod B
   - Certificate pushed to Pod B via SDS (Secret Discovery Service)
   
3. Root CA certificate distribution:
   - Istiod periodically publishes root CA via special endpoint
   - Istiod lain (cluster A) pull certificate
   - Pod A sidecar (Envoy) receives root CA via SDS

Timing:

t=0:   Pod B created
t=5s:  Istiod cluster B: detect ServiceExport
t=10s: Root CA issue certificate
t=15s: Istiod cluster A: pull root CA + Pod B cert
t=20s: Pod A sidecar: receive certs via SDS
t=25s: Pod A→Pod B: mTLS handshake successful

SPIFFE Identity Across Clusters

SPIFFE (Secure Production Identity Framework for Everyone):

Pod identity = SPIFFE ID

Format: spiffe://cluster-id/ns/namespace/sa/service-account

Example:
  Cluster A Pod: spiffe://cluster-a/ns/backend/sa/api
  Cluster B Pod: spiffe://cluster-b/ns/backend/sa/api

Identity verification:

Pod A cert:
  Subject: spiffe://cluster-a/ns/backend/sa/api
  Issued by: Kubernetes cluster root CA
  
Pod B verifies:
  1. Certificate valid? (signature check via root CA)
  2. Identity (SPIFFE ID) matches expectations?
  3. If yes: trust, establish connection

Implication: Meskipun nama KSA sama ("api"), SPIFFE ID berbeda (cluster-a vs cluster-b). Ini allows distinguishing traffic per-cluster dalam authorization policies.

Authorization Policies Across Clusters

AuthorizationPolicy: Menyaring Traffic Per-Cluster

yaml

# Cluster B: Allow traffic only dari Cluster A api pods
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-cluster-a-api
  namespace: backend
spec:
  selector:
    matchLabels:
      app: api
  rules:
    - from:
        - source:
            principals:
              - "spiffe://cluster-a/ns/backend/sa/api"  # Only cluster-a
      to:
        - operation:
            methods: ["GET", "POST"]
            ports: ["8080"]

Traffic allowed:

Cluster A Pod (app=api) → Cluster B Pod (app=api): ✓ Allowed
Cluster B Pod (app=web) → Cluster B Pod (app=api): ❌ Denied
Cluster B Pod (app=api) → Cluster B Pod (app=api): ❌ Denied (from different principal)

Service Mesh Operator Setup (ASM/Cloud Service Mesh)

One Mesh (Unified Control Plane)

┌──────────────────────────────────────────┐
│  Cloud Service Mesh (managed Istiod)     │
├──────────────────────────────────────────┤
│  Single Istiod instance managing:        │
│  - Cluster A config (CRDs, services)     │
│  - Cluster B config (CRDs, services)     │
│  - Shared root CA (Kubernetes cluster)   │
│  - Certificate distribution              │
│  - Traffic management across clusters    │
│                                          │
└──────────────────────────────────────────┘
    ▲                    ▲
    │ CRD watch          │ workload cert
    │ config push        │ delivery (SDS)
    │                    │
┌────────────┐      ┌────────────┐
│ Cluster A  │      │ Cluster B  │
│ (members)  │      │ (members)  │
└────────────┘      └────────────┘

Advantage:

Unified control: single Istiod manages all clusters
Simplified operations: one place to define policies
Automatic distribution: no manual cert sync

Limitation:

Requires Fleet setup
Control plane becomes critical (outage = all clusters affected)

Multi-Mesh (Per-Cluster Control Plane)

Cluster A Istiod ────────┐
                        (Root CA sync)
Cluster B Istiod ────────┤
                        (periodic pull)
Cluster C Istiod ────────┘

Setup:
1. Each cluster runs own Istiod
2. Istiods manually configured to trust same Root CA
3. Periodic sync of root CA certificates
4. Complex but: each cluster independent

Advantage:

Cluster isolation: cluster A problem doesn't affect B
Flexible: each cluster can have different policies

Disadvantage:

Operational toil: sync root CA, manage certificate distribution
Eventual consistency: changes propagate slowly

Production Patterns

✅ Pattern: One Mesh dengan Fleet

bash

# Prerequisite: Create GKE Fleet
gcloud container fleet create \
  --project=PROJECT_ID

# Register clusters
gcloud container fleet memberships register cluster-a \
  --gke-cluster=us-central1/cluster-a

gcloud container fleet memberships register cluster-b \
  --gke-cluster=us-west1/cluster-b

# Enable Cloud Service Mesh (one mesh)
gcloud container fleet mesh enable

# Wait for Istiod to be deployed (automatic)
kubectl get pods -n istio-system -n cluster-a
# istiod-XXXX running

# Provision Kubernetes cluster for root CA (automatic with CSM)

✅ Pattern: AuthorizationPolicy for Multi-Cluster Service Access

yaml

# Cluster B: backend-api service
---
apiVersion: v1
kind: Service
metadata:
  name: backend-api
  namespace: backend
spec:
  selector:
    app: api
  ports:
    - port: 8080
      name: http

---
# Allow only Cluster A frontend pods
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: backend-api-authz
  namespace: backend
spec:
  selector:
    matchLabels:
      app: api
  rules:
    - from:
        - source:
            principals:
              - "spiffe://cluster-a/ns/frontend/sa/web"
            namespaces:
              - frontend
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/api/*"]
            ports: ["8080"]

❌ Anti-Pattern: Mismatch Root CA Across Clusters

yaml

# WRONG: Cluster A has root-ca-a, Cluster B has root-ca-b
Cluster A: Istiod signed certs with root-ca-a
Cluster B: Istiod signed certs with root-ca-b

Result:
  Pod A cert (signed by root-ca-a)
  Pod B rejects: "Root CA unknown"
  → TLS handshake fails
  → "connection refused" error

Fix: Ensure single shared root CA across all clusters in mesh.

Debugging Service Mesh Issues

Check Certificate Trust Chain

bash

# Cluster A Pod A: check its certificate
kubectl exec -it <pod-a> -c istio-proxy -- \
  cat /etc/istio/certs/cert-chain.pem | openssl x509 -text -noout

# Check issuer
# Issuer: Kubernetes cluster (the root CA)
# Subject CN: spiffe://cluster-a/ns/backend/sa/api

# Verify root CA
kubectl exec -it <pod-a> -c istio-proxy -- \
  cat /etc/istio/certs/root-cert.pem | openssl x509 -text -noout
# Issuer: Kubernetes cluster (root)

Verify mTLS Connection

bash

# Cluster A Pod A calling Cluster B Pod B
kubectl logs <pod-a> -c istio-proxy | grep -i "mTLS\|TLS"

# Expected output:
# "outbound listener TCP 8080 mTLS TLS configuration [...]"

Check AuthorizationPolicy Evaluation

bash

# Access logs show policy evaluation
kubectl logs -n backend <pod-b> -c istio-proxy | grep authorization

# Expected:
# "AuthorizationPolicy: allow from spiffe://cluster-a/ns/frontend/sa/web"
# "AuthorizationPolicy: deny from spiffe://cluster-b/ns/frontend/sa/web"

Summary

Aspect	Implementation
Trust model	Shared root CA across clusters
Certificate lifecycle	Istiod issues, distributes via SDS
Identity	SPIFFE per-pod, includes cluster ID
Authorization	PolicyList SPIFFE principals cross-cluster
Deployment	One Mesh (unified Istiod) vs Multi-Mesh (per-cluster)
Complexity	High (certificate management, policy coordination)

Service mesh federation transforms multi-cluster from network-level to application-level security. Nó provides mTLS + authorization natively, removing need for explicit VPN/firewall rules.

Trade-off: Simplifies security model (mTLS everywhere), nhưng adds operational complexity (certificate distribution, policy management).

Cross-Cluster Service Mesh & Trust Federation ​

Tại sao Service Mesh Cần Cross-Cluster Support ​

mTLS Federation: Root CA & Certificate Distribution ​

The Trust Model ​

Certificate Distribution Mechanism ​

SPIFFE Identity Across Clusters ​

Authorization Policies Across Clusters ​

AuthorizationPolicy: Menyaring Traffic Per-Cluster ​

Service Mesh Operator Setup (ASM/Cloud Service Mesh) ​

One Mesh (Unified Control Plane) ​

Multi-Mesh (Per-Cluster Control Plane) ​

Production Patterns ​

✅ Pattern: One Mesh dengan Fleet ​

✅ Pattern: AuthorizationPolicy for Multi-Cluster Service Access ​

❌ Anti-Pattern: Mismatch Root CA Across Clusters ​

Debugging Service Mesh Issues ​

Check Certificate Trust Chain ​

Verify mTLS Connection ​

Check AuthorizationPolicy Evaluation ​

Summary ​

References ​

Cross-Cluster Service Mesh & Trust Federation

Tại sao Service Mesh Cần Cross-Cluster Support

mTLS Federation: Root CA & Certificate Distribution

The Trust Model

Certificate Distribution Mechanism

SPIFFE Identity Across Clusters

Authorization Policies Across Clusters

AuthorizationPolicy: Menyaring Traffic Per-Cluster

Service Mesh Operator Setup (ASM/Cloud Service Mesh)

One Mesh (Unified Control Plane)

Multi-Mesh (Per-Cluster Control Plane)

Production Patterns

✅ Pattern: One Mesh dengan Fleet

✅ Pattern: AuthorizationPolicy for Multi-Cluster Service Access

❌ Anti-Pattern: Mismatch Root CA Across Clusters

Debugging Service Mesh Issues

Check Certificate Trust Chain

Verify mTLS Connection

Check AuthorizationPolicy Evaluation

Summary

References