Workload Identity Failures: Token Exchange Debugging

Symptoms Recognition

Workload Identity failures xuất hiện khi:

Pod cannot access Google Cloud APIs (permission denied)
Error: 403 Forbidden từ GCP services
Error: 400 Bad Request during token exchange
Metadata server không respond (timeout, Connection refused)
Error: "quota exceeded" (token exchange)
gke-metadata-server pod OOMKilled
Service account token validation fails

Why This Matters

Workload Identity là modern pattern để authenticate Pods tới GCP services tanpa hardcoding API keys. Saat WI fails, Pods không có thể:

Read từ Cloud Storage
Write đến Pub/Sub
Query BigQuery
Access Secret Manager
Anything mà cần GCP API auth

Understanding token exchange mechanics → capability debug complex auth issues.

Information Gathering — Quick Diagnostics

Step 1: Check Workload Identity Setup

bash

# Verify cluster memiliki Workload Identity enabled
gcloud container clusters describe <cluster> --zone=<zone> \
  | grep workloadIdentity

# Check node pool WI enabled
gcloud container node-pools describe <pool> \
  --cluster=<cluster> --zone=<zone> \
  | grep workloadIdentity

# Verify OIDC issuer URL configured
gcloud container clusters describe <cluster> --zone=<zone> \
  | grep issuerUrl

Expected output:

workloadIdentityConfig.workloadPool: <project-id>.svc.id.goog
Node pools should have WI enabled
issuerUrl should be https://oidc.eks.googleapis.com/ or similar

Step 2: Check Kubernetes ServiceAccount & IAM Binding

bash

# Verify KSA exists
kubectl get serviceaccounts -n <namespace>
kubectl describe serviceaccount <ksa-name> -n <namespace>

# Check WI annotation (point to GSA)
kubectl get serviceaccount <ksa-name> -n <namespace> -o yaml | \
  grep -i "iam.gke.io/gcp-service-account"

# Verify GSA exists trong GCP
gcloud iam service-accounts list | grep <gsa-name>

# Check IAM binding (KSA → GSA)
gcloud iam service-accounts get-iam-policy <gsa-email>
# Should show: roles/iam.workloadIdentityUser binding với KSA principal

Expected binding format:

members:
- serviceAccount:project-123456-compute@developer.gserviceaccount.com
  # OR
- principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/attribute.namespace/<namespace>

Step 3: Test Token Exchange from Pod

bash

# Exec into pod
kubectl exec -it <pod-name> -n <namespace> -- /bin/bash

# Inside pod:
# Step 1: Get Kubernetes token
KSA_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
echo $KSA_TOKEN | cut -d. -f2 | base64 -d | python3 -m json.tool

# Step 2: Check metadata server accessibility
curl -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity

# Step 3: Exchange token để GSA token
curl -X POST https://sts.googleapis.com/v1/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=urn:ietf:params:oauth:grant-type:token-exchange&audience=//iam.googleapis.com/projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/<POOL_ID>/providers/<PROVIDER_ID>&requested_token_use=access_token&subject_token=$KSA_TOKEN&subject_token_type=urn:ietf:params:oauth:token-type:jwt"

# Step 4: Use token để call GCP API
ACCESS_TOKEN=<token từ step 3>
curl -H "Authorization: Bearer $ACCESS_TOKEN" \
  https://www.googleapis.com/storage/v1/b

Step 4: Check Metadata Server Health

bash

# View gke-metadata-server pod
kubectl get pod -n gke-system -l component=gke-metadata-server
kubectl describe pod <metadata-server-pod> -n gke-system

# Check logs
kubectl logs <metadata-server-pod> -n gke-system

# Check memory/CPU usage
kubectl top pod <metadata-server-pod> -n gke-system

# Check if pod is in CrashLoop
kubectl get pod <metadata-server-pod> -n gke-system --watch

Step 5: Verify IAM Role Permissions

bash

# Check GSA membuat roles necessary
gcloud iam service-accounts describe <gsa-email>

# List IAM roles để GSA
gcloud projects get-iam-policy <project> \
  --flatten="bindings[].members" \
  --filter="bindings.members:<gsa-email>" \
  --format="table(bindings.role)"

# Expected roles (depends trên what API calls):
# roles/storage.objectViewer (for Cloud Storage read)
# roles/pubsub.editor (for Pub/Sub)
# roles/bigquery.user (for BigQuery)

Diagnostic Decision Tree

Workload Identity Failure
│
├─ WI enabled di cluster?
│  ├─ NO → enable: gcloud container clusters update --workload-pool=<project>.svc.id.goog
│  │
│  └─ YES → continue
│
├─ Node pool WI enabled?
│  ├─ NO → enable node pool
│  │
│  └─ YES → continue
│
├─ KSA exists và configured?
│  ├─ NO → create KSA, annotate với GSA
│  │
│  └─ YES → continue
│
├─ GSA exists?
│  ├─ NO → create GSA
│  │
│  └─ YES → continue
│
├─ IAM binding correct (KSA → GSA)?
│  ├─ NO → add binding: gcloud iam service-accounts add-iam-policy-binding
│  │
│  └─ YES → continue
│
├─ Metadata server accessible?
│  ├─ NO → check gke-metadata-server pod
│  │  └─ Check pod logs, restart if needed
│  │
│  └─ YES → continue
│
├─ Token exchange successful?
│  ├─ NO → check error message
│  │  ├─ 403 Forbidden → IAM permission issue
│  │  ├─ 429 Too Many Requests → quota limit
│  │  ├─ 400 Bad Request → token/audience format wrong
│  │
│  └─ YES → continue
│
├─ GSA memiliki permissions?
│  ├─ NO → grant roles đến GSA
│  │
│  └─ YES → success
│
└─ Token not recognized bởi API?
   └─ Check if token expired, cache stale, clock skew

Common Root Causes & Fixes

Root Cause 1: KSA to GSA Binding Missing / Incorrect

Dấu hiệu:

Error: "403 Forbidden" when calling GCP APIs
Pod logs: "The caller does not have permission"
IAM check shows no binding

Nguyên nhân: KSA không di-annotate với GSA email, hoặc IAM binding không exist.

Diagnostic:

bash

# Check KSA annotation
kubectl get serviceaccount <ksa-name> -n <namespace> -o yaml

# Check IAM binding
gcloud iam service-accounts get-iam-policy <gsa-email>

# Manual principal check
# Expected principal format:
# serviceAccount:projects/<project>/serviceAccounts/<ksa>@<project>.iam.gserviceaccount.com
# OR
# principalSet://iam.googleapis.com/projects/<project-id>/locations/global/workloadIdentityPools/<pool>/attribute.namespace/<ns>

Immediate Fix:

bash

# Option 1: Add WI annotation to KSA
kubectl annotate serviceaccount <ksa-name> \
  -n <namespace> \
  iam.gke.io/gcp-service-account=<gsa-email> \
  --overwrite

# Option 2: Add IAM binding (KSA direct)
gcloud iam service-accounts add-iam-policy-binding <gsa-email> \
  --role=roles/iam.workloadIdentityUser \
  --member=serviceAccount:<ksa>@<project>.iam.gserviceaccount.com

# Option 3: Add IAM binding (namespace-level)
gcloud iam service-accounts add-iam-policy-binding <gsa-email> \
  --role=roles/iam.workloadIdentityUser \
  --member=principalSet://iam.googleapis.com/projects/<project-id>/locations/global/workloadIdentityPools/<project-id>.svc.id.goog/attribute.namespace/<namespace>

Permanent Fix:

Enforce KSA annotation policy (admission webhook)
Document WI binding patterns
Automate KSA+GSA creation trong CI/CD

Root Cause 2: Metadata Server Pod Crash / OOM

Dấu hiệu:

gke-metadata-server pod CrashLoop
Pod OOMKilled (memory exceeded)
Error: "Connection refused" when Pod tries metadata server
Event: "gke-metadata-server: Memory limit exceeded"

Nguyên nhân: metadata server pod sở hữu memory limits terlalu nhỏ, hoặc memory leak. Xảy ra saat:

Cluster punya nhiều KSA (> 3000)
Token exchange rate tinggi
metadata-server memory leak (rare)

Diagnostic:

bash

# Check pod status
kubectl get pod -n gke-system -l component=gke-metadata-server
kubectl describe pod <pod-name> -n gke-system

# Check memory usage trend
kubectl top pod -n gke-system <pod-name>

# Count KSAs di cluster (major factor)
kubectl get serviceaccounts --all-namespaces | wc -l

# Check metadata-server logs
kubectl logs <pod-name> -n gke-system | tail -100
# Look để OOMKilled events

Immediate Fix:

bash

# Option 1: Increase metadata-server memory request
# Edit DaemonSet
kubectl edit daemonset -n gke-system gke-metadata-server

# Find container spec, increase memory:
# resources:
#   requests:
#     memory: "256Mi" → "512Mi" hoặc "1Gi"

# Option 2: Reduce KSA count
# Consolidate KSAs (use per-namespace KSA instead of per-app)

# Option 3: Restart metadata-server pods
kubectl rollout restart daemonset/gke-metadata-server -n gke-system

Permanent Fix:

Size metadata-server properly:
- Default: 256Mi memory
- Large clusters (> 1000 KSAs): 512Mi or 1Gi
- Monitor trends
Consolidate KSAs:
- Instead of creating KSA per app, use shared KSA per namespace
- Map multiple apps to same KSA (if permissions allow)

Monitor metadata-server:

bash

# Alert if pod memory > 80% limit
# Alert if pod OOMKilled

Root Cause 3: Token Exchange Rate Limit (Quota Exceeded)

Dấu hiệu:

Error: 429 Too Many Requests during token exchange
Event: "quota exceeded: 6000 requests/minute"
Spike trong pod startup hoặc update wave

Nguyên nhân: STS (Security Token Service) sở hữu rate limit: 6000 token exchange requests/minute per project. Xảy ra saat:

Large pod rolling update (many pods request tokens simultaneously)
High pod churn (frequent recreations)
Token caching not working properly

Diagnostic:

bash

# Estimate token exchange rate
# Assume: 1 token exchange per pod startup + 1 per hourly refresh
# If 1000 pods, each refresh hourly: ~17 requests/sec = 1000/min (OK)
# If 5000 pods with 10s startup time: 500/sec = 30000/min (EXCEEDS 6000/min!)

# Check token exchange logs (if available)
# Cloud Logging → Filter by token exchange errors

# Monitor requests per minute
# Cloud Monitoring → sts.googleapis.com metrics (if exposed)

Immediate Fix:

bash

# Option 1: Stagger pod updates (rolling update)
# Don't scale up all pods simultaneously:
kubectl set image deployment/<dep> <container>=<image> --record
# Use maxUnavailable/maxSurge để control rate

# Option 2: Implement token caching
# Most GCP client libraries cache tokens (1 hour default)
# Verify caching enabled trong app code

# Option 3: Request quota increase
# GCP Console → Quotas and System Limits
# Search: "Token Exchange Requests (regional) per minute per region"
# Request increase to 10000 hoặc higher

Permanent Fix:

Plan scaling carefully:
- Calculate max token exchange rate needed
- Account để pod startup wave, rolling updates
- Pre-request quota increase

Implement token caching:

Default client libraries cache 1 hour
Tokens valid 1 hour, so caching prevents redundant exchanges
Verify aplikasi menggunakan latest client library

Monitor token exchange rate:
- Alert if rate > 4000 req/min (80% of quota)

Root Cause 4: GSA Missing Required Roles/Permissions

Dấu hiệu:

Token exchange succeeds (403 errors gone)
But API calls still fail with "permission denied"
Error: "does not have storage.objects.get permission"

Nguyên nhân: GSA không sở hữu role để required GCP service.

Diagnostic:

bash

# List current roles để GSA
gcloud projects get-iam-policy <project> \
  --flatten="bindings[].members" \
  --filter="bindings.members:<gsa-email>" \
  --format="table(bindings.role)" | sort

# Needed roles depend upon what APIs app calls:
# Cloud Storage read → roles/storage.objectViewer
# Cloud Storage write → roles/storage.objectCreator
# Pub/Sub → roles/pubsub.editor
# BigQuery → roles/bigquery.dataEditor
# Secret Manager → roles/secretmanager.secretAccessor

Immediate Fix:

bash

# Grant required role
gcloud projects add-iam-policy-binding <project> \
  --member=serviceAccount:<gsa-email> \
  --role=roles/storage.objectViewer

# Or using custom role (fine-grained permissions)
# Recommended để production (least privilege)

Permanent Fix:

Document required roles để each app
Create custom roles để least-privilege access
Implement RBAC review process

Root Cause 5: Token Format / Audience Mismatch

Dấu hiệu:

Token exchange returns error: "400 Bad Request"
Message: "Invalid token", "Invalid audience"
JWT decode shows wrong claims

Nguyên nhân: Token format incorrect:

Wrong audience claim
Token from wrong cluster
Clock skew (system time wrong)

Diagnostic:

bash

# Get token từ pod
KSA_TOKEN=$(kubectl exec <pod> -n <namespace> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)

# Decode token
echo $KSA_TOKEN | cut -d. -f2 | base64 -d | python3 -m json.tool

# Check claims:
# "iss" = issuer (should be cluster's OIDC issuer)
# "aud" = audience (should include "sts.googleapis.com")
# "exp" = expiration (shouldn't be past)

# Verify cluster OIDC issuer
gcloud container clusters describe <cluster> --zone=<zone> | grep issuerUrl

# Check system time sync
date  # Should be within ±5 seconds of actual time

Immediate Fix:

bash

# Option 1: Fix system clock (if skewed)
# On GCP VMs, NTP usually working
# Force sync:
timedatectl set-ntp on
systemctl restart chronyd  # or ntpd

# Option 2: Recreate pod (get fresh token)
kubectl delete pod <pod-name> -n <namespace>

# Option 3: Check cluster OIDC issuer URL
gcloud container clusters describe <cluster> | grep issuerUrl
# Should be valid URL

Prevention & Monitoring

Comprehensive WI Monitoring

bash

# Cloud Monitoring dashboards

# Dashboard: Workload Identity Health
# Metrics:
#   - Token exchange success/failure rate
#   - Token exchange latency
#   - Metadata server pod health
#   - GSA API call errors

# Alert Rules:
#   - Token exchange error rate > 1% → Alert
#   - Metadata server pod CrashLoop → Page
#   - GSA permission denied errors → Alert
#   - Token exchange quota > 80% → Alert

Automation & Policy

yaml

# Recommended: Admission webhook để enforce WI setup
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: workload-identity-validator
webhooks:
- name: validate-workload-identity
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["serviceaccounts"]
  # Webhook ensure every KSA has WI annotation
---
# Best practice: Use namespace-level KSA
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: production
  annotations:
    iam.gke.io/gcp-service-account: app-gsa@project.iam.gserviceaccount.com

Escalation Criteria

Escalate if:

Metadata server pod persistent CrashLoop
Token exchange quota request denied (contact GCP support)
Suspected GSA impersonation breach
Token validation fails despite correct setup (crypto issue)

Pod Creation Failures — Nếu admission webhook WI enforcement block pod
Control Plane Issues — Nếu API server not responding to token exchange

Workload Identity Failures: Token Exchange Debugging ​

Symptoms Recognition ​

Why This Matters ​

Information Gathering — Quick Diagnostics ​

Step 1: Check Workload Identity Setup ​

Step 2: Check Kubernetes ServiceAccount & IAM Binding ​

Step 3: Test Token Exchange from Pod ​

Step 4: Check Metadata Server Health ​

Step 5: Verify IAM Role Permissions ​

Diagnostic Decision Tree ​

Common Root Causes & Fixes ​

Root Cause 1: KSA to GSA Binding Missing / Incorrect ​

Root Cause 2: Metadata Server Pod Crash / OOM ​

Root Cause 3: Token Exchange Rate Limit (Quota Exceeded) ​

Root Cause 4: GSA Missing Required Roles/Permissions ​

Root Cause 5: Token Format / Audience Mismatch ​

Prevention & Monitoring ​

Comprehensive WI Monitoring ​

Automation & Policy ​

Escalation Criteria ​

Related Sections ​

References ​

Workload Identity Failures: Token Exchange Debugging

Symptoms Recognition

Why This Matters

Information Gathering — Quick Diagnostics

Step 1: Check Workload Identity Setup

Step 2: Check Kubernetes ServiceAccount & IAM Binding

Step 3: Test Token Exchange from Pod

Step 4: Check Metadata Server Health

Step 5: Verify IAM Role Permissions

Diagnostic Decision Tree

Common Root Causes & Fixes

Root Cause 1: KSA to GSA Binding Missing / Incorrect

Root Cause 2: Metadata Server Pod Crash / OOM

Root Cause 3: Token Exchange Rate Limit (Quota Exceeded)

Root Cause 4: GSA Missing Required Roles/Permissions

Root Cause 5: Token Format / Audience Mismatch

Prevention & Monitoring

Comprehensive WI Monitoring

Automation & Policy

Escalation Criteria

Related Sections

References