Workload Identity Across Clusters

Tại sao Workload Identity Là Phức Tạp Trong Multi-Cluster

Trong single cluster, Workload Identity đơn giản:

Pod → Metadata Server (cùng cluster) → STS → IAM binding
↓
Pod nhận access token → call GCP APIs

Nhưng trong multi-cluster, vấn đề nảy sinh:

Cluster A Pod X: "Tôi là ksa-x trong namespace-a"
Cluster B Pod Y: "Tôi là ksa-x trong namespace-a"  (cùng tên!)

Câu hỏi:
  - Có phải chúng là "same identity"? (Yes, vì cùng project)
  - Chúng có cùng quyền? (Yes, nếu IAM binding same)
  - Điều này có an toàn? (NO if cluster B untrusted!)
  - Làm sao để restrict access per-cluster?

Identity Sameness: Same Project = Same Identity Pool

Key principle:

Workload Identity Pool per project: PROJECT_ID.svc.id.goog

Cluster A ──┐
Cluster B ──┼──→ Shared pool: my-project.svc.id.goog
Cluster C ──┘

Result:
  Pod với KSA "default/app" từ cluster A
  Pod với KSA "default/app" từ cluster B
  → IAM vệ nhân ở pool → ĐẠI DIỆN GIỐNG NHAU

Cơ chế:

yaml

# Cluster A Pod
apiVersion: v1
kind: Pod
metadata:
  name: app-pod
  namespace: default
spec:
  serviceAccountName: app
  containers:
    - name: app
      env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/run/secrets/workload-identity/identity.json

---

# Metadata server: Pod identity
{
  "iss": "https://iam.googleapis.com/projects/PROJECT/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/providers/CLUSTER_ID",
  "aud": "sts.googleapis.com",
  "iat": 1621234567,
  "exp": 1621238167,
  "sub": "projects/PROJECT/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/default/sa/app"
}

Principal format: projects/PROJECT/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/default/sa/app

Nếu cluster A Pod & cluster B Pod cùng namespace & KSA name → cùng principal string → IAM sees them as cùng identity.

Principal Definitions: Granularity Levels

Level 1: Per-ServiceAccount (Tight)

yaml

# IAM binding: chỉ app KSA có quyền
gcloud iam service-accounts add-iam-policy-binding \
  gcp-service-account@project.iam.gserviceaccount.com \
  --role=roles/iam.workloadIdentityUser \
  --member='principalSet://iam.googleapis.com/projects/PROJECT/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/attribute.namespace_name/default'

Effect:

Cluster A: Pod app (ns: default) → CAN access
Cluster B: Pod app (ns: default) → CAN access (problem!)
Cluster B: Pod web (ns: default) → CANNOT access

Problem: Cluster B untrusted pod "app" cùng quyền như Cluster A

Level 2: Per-Namespace (Medium)

yaml

# IAM binding: tất cả pods trong namespace
gcloud iam service-accounts add-iam-policy-binding \
  gcp-service-account@project.iam.gserviceaccount.com \
  --role=roles/iam.workloadIdentityUser \
  --member='principalSet://iam.googleapis.com/projects/PROJECT/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/attribute.namespace_name/backend'

Effect:

Cluster A: Namespace "backend" → all pods → CAN access
Cluster B: Namespace "backend" → all pods → CAN access (problem!)

Level 3: Per-Cluster (Safest)

yaml

# CANNOT do directly in IAM (no per-cluster principal)
# Solution: Use separate projects per cluster

Cluster A: project-a
Cluster B: project-b

Workload Identity Pool:
  - Cluster A: projects/project-a.svc.id.goog
  - Cluster B: projects/project-b.svc.id.goog

IAM binding: project-a.svc.id.goog → full access
            project-b.svc.id.goog → limited access (or none)

Or: Use conditional IAM policies

yaml

# IAM binding with condition (CEL)
gcloud iam service-accounts add-iam-policy-binding \
  gcp-service-account@project.iam.gserviceaccount.com \
  --role=roles/iam.workloadIdentityUser \
  --member='principalSet://iam.googleapis.com/projects/PROJECT/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goop/...' \
  --condition='resource.matchTag("cluster", "cluster-a")'

Security Implications: Untrusted Cluster Risk

Scenario: Cluster A (trusted), Cluster B (untrusted friend)

Cluster A: Pods run payment processing
           KSA "app" → can access Cloud SQL (payments database)

Cluster B: Attacker compromised cluster
           Attacker creates pod: KSA "app" (same name)
           Pod: "I'm default/app, give me Cloud SQL access"
           → Metadata server: YES, authenticated
           → Pod reads: secret payment records

Result: Data breach!

Root cause: Identity sameness + insufficient isolation.

Mitigation Strategies

1. Separate Projects (Strongest)

bash

# Cluster A: project-trusted
# Cluster B: project-untrusted

# Create service accounts per project
gcloud iam service-accounts create app \
  --project=project-trusted

gcloud iam service-accounts create app \
  --project=project-untrusted

# Bind different roles
gcloud projects add-iam-policy-binding project-trusted \
  --role=roles/cloudsql.client \
  --member='serviceAccount:app@project-trusted.iam.gserviceaccount.com'

# project-untrusted: no Cloud SQL access

2. Attribute-Based IAM Conditions (Weaker)

yaml

# Try to restrict per-cluster using attributes
# Problem: No built-in "cluster" attribute

# Workaround: Custom attribute via annotation
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app
  namespace: default
  annotations:
    iam.gke.io/cluster: cluster-a  # Custom attribute

---

# IAM binding: but GAP - no way to enforce annotation in IAM policy
# Conditions can't read pod annotations directly

Limitation: IAM conditions don't have access to pod-level metadata (only JWT claims). Since both clusters sign JWT with same issuer, can't distinguish.

3. Namespace Isolation + Pod Security Policies (Weaker)

yaml

# Restrict who can create pods in namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-deployer
  namespace: backend
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: deployer
subjects:
  - kind: User
    name: "cluster-a@example.com"  # Only Cluster A admins

---

# If Cluster B is untrusted, ensure they can't create pods in "backend" ns

Limitation: This is Kubernetes RBAC, not IAM. Doesn't prevent Cluster B admin from deploying pods if they want.

4. Service Account Impersonation (Recommended)

yaml

# Don't use direct IAM binding
# Instead: Use annotation to impersonate GCP service account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app
  namespace: backend
  annotations:
    iam.gke.io/gcp-service-account: app-sa@project.iam.gserviceaccount.com

---

# IAM binding: allow "app" KSA from cluster-a to impersonate
gcloud iam service-accounts add-iam-policy-binding \
  app-sa@project.iam.gserviceaccount.com \
  --role=roles/iam.workloadIdentityUser \
  --member='principalSet://iam.googleapis.com/projects/PROJECT/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goop/subject/ns/backend/sa/app'

# For cluster-b: SEPARATE binding or DENY
gcloud iam service-accounts remove-iam-policy-binding \
  app-sa@project.iam.gserviceaccount.com \
  --role=roles/iam.workloadIdentityUser \
  --member='principalSet://iam.googleapis.com/projects/OTHER_PROJECT/locations/global/workloadIdentityPools/OTHER_PROJECT_ID.svc.id.goop/subject/ns/backend/sa/app'

Cross-Cluster Token Exchange (Advanced)

Scenario: Cluster A Pod cần call API ở Cluster B.

Cluster A Pod:
  1. Get WIF token từ metadata server
  2. Call: sts.googleapis.com (token exchange)
  3. Return: short-lived credential

Cluster B Pod:
  1. Verify credential came from trusted Cluster A
  2. Process request

Token exchange endpoint:

bash

# Cluster A Pod
curl -H "Authorization: Bearer $TOKEN" \
  https://sts.googleapis.com/v1/token \
  -d "grant_type=urn:ietf:params:oauth:grant-type:token-exchange" \
  -d "subject_token=$WORKLOAD_IDENTITY_TOKEN"

Trust establishment:

Cluster A OIDC issuer: https://iam.googleapis.com/projects/PROJECT/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goop/providers/cluster-a
Cluster B validate: Is issuer trusted? (check JWKS endpoint)
                    Is audience correct?
                    Is token expired?

Production Patterns

✅ Pattern: Separate Projects for Trust Boundaries

bash

# Architecture:
# Project A: Trusted clusters (payments, auth)
# Project B: Untrusted clusters (user-generated workloads)

# Cluster setup:
gcloud container clusters create trusted-cluster \
  --project=project-a \
  --workload-pool=project-a.svc.id.goop

gcloud container clusters create untrusted-cluster \
  --project=project-b \
  --workload-pool=project-b.svc.id.goop

# Service accounts:
# project-a: app-sa (can read secrets, access databases)
# project-b: app-sa (limited: only read public data)

✅ Pattern: Service Account Impersonation for Cross-Cluster

yaml

# Cluster B Pod needs to call Cluster A service

# Cluster B KSA
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app
  namespace: default

---

# Cluster A: Grant impersonation rights to Cluster B KSA
gcloud iam service-accounts add-iam-policy-binding \
  app-sa@project-a.iam.gserviceaccount.com \
  --role=roles/iam.serviceAccountTokenCreator \
  --member='principalSet://iam.googleapis.com/projects/project-b/locations/global/workloadIdentityPools/project-b.svc.id.goop/subject/ns/default/sa/app'

---

# Cluster B Pod application code
from google.auth.transport.requests import Request
from google.auth import default as auth_default
from google.auth import impersonated_credentials

# Get Cluster B's identity
creds, _ = auth_default()

# Impersonate Cluster A service account
target_scopes = ['https://www.googleapis.com/auth/cloud-platform']
impersonated = impersonated_credentials.Credentials(
  source_credentials=creds,
  target_principal='app-sa@project-a.iam.gserviceaccount.com',
  target_scopes=target_scopes
)

# Now use impersonated creds to call Cluster A APIs

❌ Anti-Pattern: Trusting All Clusters Equally

yaml

# WRONG: All clusters shared project, no isolation
Cluster A: project-shared → app-sa (full database access)
Cluster B: project-shared → app-sa (full database access)  ❌ Untrusted!
Cluster C: project-shared → app-sa (full database access)  ❌ Untrusted!

Result: One compromised cluster = all data compromised

Better:

yaml

# Separate trust boundaries
Cluster A: project-trusted (data processing, database access)
Cluster B: project-user-workloads (limited, no database access)
Cluster C: project-development (sandbox, test only)

Workload pools:
  project-trusted.svc.id.goop ← trusted credentials
  project-user-workloads.svc.id.goop ← sandboxed credentials
  project-development.svc.id.goop ← development credentials

Debugging Cross-Cluster Identity Issues

Check Principal String

bash

# Cluster A Pod
kubectl exec -it <pod> -- cat /var/run/secrets/workload-identity/identity.json | jq .sub

# Expected output:
# "sub": "projects/PROJECT/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goop/subject/ns/backend/sa/app"

Verify IAM Binding

bash

# Check who has workloadIdentityUser role
gcloud iam service-accounts get-iam-policy \
  app-sa@project.iam.gserviceaccount.com

bindings:
  - members:
      - principalSet://iam.googleapis.com/projects/PROJECT/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goop/subject/ns/backend/sa/app
    role: roles/iam.workloadIdentityUser

Test Token Exchange

bash

# Cluster A Pod
kubectl exec -it <pod> -- curl -X GET \
  -H "Authorization: Bearer $(cat /var/run/secrets/workload-identity/identity.json | jq -r .)" \
  https://www.googleapis.com/oauth2/v4/tokeninfo

# Should return: "email": "app-sa@project.iam.gserviceaccount.com"

Summary

Aspect	Implementation
Identity pooling	Shared per-project pool across clusters
Principal format	ns/sa hierarchy (cannot filter per-cluster)
Security model	Trust boundary = project boundary
Recommendation	Separate projects for trust levels
Alternative	Impersonation + conditional IAM policies

Key takeaway: Multi-cluster Workload Identity simplifies authentication (no key management), nhưng requires careful project/namespace planning để avoid identity leakage giữa untrusted clusters.

Workload Identity Across Clusters ​

Tại sao Workload Identity Là Phức Tạp Trong Multi-Cluster ​

Identity Sameness: Same Project = Same Identity Pool ​

Principal Definitions: Granularity Levels ​

Level 1: Per-ServiceAccount (Tight) ​

Level 2: Per-Namespace (Medium) ​

Level 3: Per-Cluster (Safest) ​

Security Implications: Untrusted Cluster Risk ​

Mitigation Strategies ​

1. Separate Projects (Strongest) ​

2. Attribute-Based IAM Conditions (Weaker) ​

3. Namespace Isolation + Pod Security Policies (Weaker) ​

4. Service Account Impersonation (Recommended) ​

Cross-Cluster Token Exchange (Advanced) ​

Production Patterns ​

✅ Pattern: Separate Projects for Trust Boundaries ​

✅ Pattern: Service Account Impersonation for Cross-Cluster ​

❌ Anti-Pattern: Trusting All Clusters Equally ​

Debugging Cross-Cluster Identity Issues ​

Check Principal String ​

Verify IAM Binding ​

Test Token Exchange ​

Summary ​

References ​

Workload Identity Across Clusters

Tại sao Workload Identity Là Phức Tạp Trong Multi-Cluster

Identity Sameness: Same Project = Same Identity Pool

Principal Definitions: Granularity Levels

Level 1: Per-ServiceAccount (Tight)

Level 2: Per-Namespace (Medium)

Level 3: Per-Cluster (Safest)

Security Implications: Untrusted Cluster Risk

Mitigation Strategies

1. Separate Projects (Strongest)

2. Attribute-Based IAM Conditions (Weaker)

3. Namespace Isolation + Pod Security Policies (Weaker)

4. Service Account Impersonation (Recommended)

Cross-Cluster Token Exchange (Advanced)

Production Patterns

✅ Pattern: Separate Projects for Trust Boundaries

✅ Pattern: Service Account Impersonation for Cross-Cluster

❌ Anti-Pattern: Trusting All Clusters Equally

Debugging Cross-Cluster Identity Issues

Check Principal String

Verify IAM Binding

Test Token Exchange

Summary

References