Control Plane Issues: API Server & etcd Health

Symptoms Recognition

Control plane issues xuất hiện khi:

kubectl commands hang hoặc timeout
API server latency cao (> 1 second)
Watch connections timeout
etcd stuck trên control plane VMs
List operations (GET pods) lambat
New Pod deployments delayed
Admission webhooks timeout

Why This Matters

Control plane là single point of health cho cluster. Nếu API server overload hoặc etcd slow, cascading failures xảy ra:

Pod deployment delayed
Node heartbeats delayed
Service discovery failures
Monitoring data không dipush

Production incidents sering dimulai từ control plane issues. Understanding bottleneck di layers này là critical để platform engineers.

Information Gathering — Quick Diagnostics

Step 1: Check API Server Health & Latency

bash

# Ping API server
kubectl cluster-info

# Measure API latency
kubectl top nodes --use-protocol-buffers=true

# Get API server metrics
gcloud container clusters describe <cluster> \
  --zone=<zone> --format='value(monitoringConfig)'

# Check control plane dashboard (if metrics enabled)
gcloud container clusters describe <cluster> | grep monitoring

Interpretasi:

Latency consistently > 1s = API server overload
Latency spikes (100ms → 2s) = temporary overload hoặc GC pause
Latency uniform across operations = possibly etcd issue

Step 2: Collect Control Plane Metrics

bash

# Enable control plane metrics (if not already)
gcloud container clusters update <cluster> --zone=<zone> \
  --enable-cloud-logging --logging-service=logging.googleapis.com/kubernetes

# Query control plane metrics trong Cloud Monitoring
# Metric names:
# - kubernetes.io/apiserver/request_duration_seconds
# - kubernetes.io/apiserver/request_count
# - kubernetes.io/apiserver/inflight_requests
# - storage/etcd/db/total_size_in_bytes

# Via gcloud (nếu have permission):
gcloud monitoring timeseries list \
  --filter='metric.type=kubernetes.io/apiserver/request_duration_seconds' \
  --interval-start-time=$(date -u -d '10 minutes ago' +%Y-%m-%dT%H:%M:%S)Z \
  --interval-end-time=$(date -u +%Y-%m-%dT%H:%M:%S)Z

Step 3: Check etcd Status

bash

# GKE control plane runs etcd (managed, không accessible từ kubectl)
# But metrics available:
# Metric: storage/etcd/db/total_size_in_bytes
# Metric: storage/etcd/server/has_leader

# Check etcd size estimate
# Large etcd = slow list operations
kubectl get all --all-namespaces | wc -l  # Total objects
kubectl get endpoints --all-namespaces --no-headers | wc -l

# Estimate etcd size:
# Rough: 1 API object ≈ 1-5 KB trong etcd
# Total: num_objects * avg_size

Step 4: Analyze Request Types

bash

# Identify slow operations
# Cloud Monitoring → Kubernetes → API Server
# Filter by: operation (LIST, GET, PATCH, etc.)

# List operations are heaviest:
# - GET with labels selector
# - LIST pods (returns all pod objects)
# - WATCH operations

# Patch/Update relatively fast:
# - Single resource update

Step 5: Check Webhook Latency

bash

# Nếu admission webhooks, mereka có thể delay API responses
# Check webhook timeouts
kubectl get validatingwebhookconfigurations -o yaml | grep -i "timeout"

# Webhook latency usually separate từ API server latency
# If API latency high despite fast webhooks → etcd issue

Diagnostic Decision Tree

Control Plane Issue
│
├─ API server responding?
│  ├─ NO → API server down/unavailable
│  │  └─ GKE managed, check GCP status
│  │
│  └─ YES → continue
│
├─ API latency high? (> 1s)
│  ├─ LIST operations specifically slow?
│  │  ├─ YES → etcd performance issue
│  │  │  ├─ etcd size too large?
│  │  │  ├─ etcd slow disk?
│  │  │  └─ Watch cache exhaustion?
│  │  │
│  │  └─ ALL operations slow → API server overload
│  │
│  └─ NO (< 1s) → continue
│
├─ Inflight requests high?
│  ├─ YES → API server processing queue full
│  │  ├─ Scale down workloads sending requests
│  │  ├─ Implement rate limiting trên clients
│  │  └─ Watch for requests that don't complete
│  │
│  └─ NO → continue
│
├─ Watch connections failing?
│  ├─ YES → watch cache issue
│  │  └─ Restart API server replicas
│  │
│  └─ NO → continue
│
├─ etcd size growing?
│  ├─ YES → object count explosion
│  │  ├─ Check Events → etcd database
│  │  ├─ Implement event TTL
│  │  └─ Cleanup old resources
│  │
│  └─ NO → continue
│
└─ Webhook timeout?
   ├─ YES → slow admission webhook
   │  └─ Optimize webhook hoặc increase timeout
   │
   └─ NO → performance acceptable

Common Root Causes & Fixes

Root Cause 1: etcd Oversize / Compaction Not Running

Dấu hiệu:

etcd size growing over time
LIST operations increasingly slow
Metric: storage/etcd/db/total_size_in_bytes > 5GB
API latency slowly increasing (not sudden spike)

Nguyên nhân: etcd accumulates historical revisions (để watch caching). Nếu compaction không run, database grows indefinitely. Large etcd = slow LIST operations.

Diagnostic:

bash

# Estimate etcd size
# Cloud Monitoring → storage/etcd/db/total_size_in_bytes
# If > 3GB, probably too large

# Check Event count (major contributor)
kubectl get events --all-namespaces | wc -l
# If > 100K events, need cleanup

# Check pod count
kubectl get pods --all-namespaces --no-headers | wc -l

Immediate Fix:

bash

# Option 1: Cleanup old Events (major etcd bloat)
# Note: This deletes old events, cannot be undone
# Create a script to delete events older than N days:

kubectl get events --all-namespaces -o json | \
  jq '.items[] | select(.metadata.creationTimestamp < "'$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)'") | .metadata.name' | \
  while read event; do
    kubectl delete event $event -n $(kubectl get event $event -o jsonpath='{.metadata.namespace}')
  done

# Option 2: Nếu using local development cluster, restart:
# (GKE managed, không cần manual etcd restart)

# Option 3: Scale down event-heavy workloads
# Identify noisy workloads generating many events

Permanent Fix:

Reduce Event retention:

bash

# GKE mengatur event TTL otomatis (1 hour default)
# Verify di control plane configuration

Implement object cleanup:
- Cleanup old completed jobs, pods
- TTL controller để temporary resources

Monitor etcd size:

bash

# Alert if etcd > 5GB
# Proactive cleanup pipeline

Prevention:

Regular etcd size monitoring
Event retention policy enforcement
Completed Job cleanup automation

Root Cause 2: API Server CPU/Memory Overload

Dấu hiệu:

API request latency spikes frequently
Admission webhooks timeout
Watch operations fail intermittently
Control plane VM CPU > 80%
OOMKilled events vào API server pods

Nguyên nhân: API server xử lý terlalu nhiều requests:

Large cluster (> 5000 pods)
High event rate (many pod updates per second)
Heavy monitoring pulling metrics
Admission webhooks không efficient

Diagnostic:

bash

# Cloud Monitoring → Kubernetes → Control Plane
# Metric: api_server_request_duration_seconds (histogram)
#  → 99th percentile > 1s = overload

# Identify request types consuming CPU:
# Metric: api_server_request_count by operation
# If LIST operations very high → filtering/listing issues

# Check admission webhook duration:
# Custom metric: admission_webhook_request_duration_seconds

Immediate Fix:

bash

# Option 1: Identify heavy workloads
# Pod frequently updateing annotations, labels?
kubectl get pods --all-namespaces --sort-by=.metadata.generation

# Option 2: Reduce update frequency
# Batch updates, debounce label updates
# Example: Istio sidecar injection → multiple label updates
#          Consolidate đến single update

# Option 3: Rate limiting on clients
# Implement exponential backoff
# Cloud Client libraries already do this

# Option 4: Disable expensive webhooks (if non-critical)
kubectl delete validatingwebhookconfigurations <webhook-name>
# Or set failurePolicy: Ignore (kurang ideal)

Permanent Fix:

Implement request rate limits:

bash

# Cloud Monitoring → set up alert
# If requests/sec > threshold (e.g., 1000 req/s), alert

Optimize heavy operations:
- Kubernetes API priority & fairness (APF)
- Implement selective fields (fieldSelector) instead of full LIST
- Cache frequently accessed metadata
Use API Server metrics để capacity planning:
- If latency trending up → scale cluster before hitting limit

Root Cause 3: Watch Cache Exhaustion

Dấu hiệu:

Watch connections timeout (clients get disconnected)
Event: "watch cache size limit exceeded"
Kubelet unable to pull pod spec updates
Scheduler missing endpoint updates

Nguyên nhân: Watch cache (in-memory buffer di API server) đạt đến limit. Xảy ra saat:

Pod/endpoint update rate terlalu tinggi
Cluster > 5000 nodes
Dataplane changes mà frequent

Diagnostic:

bash

# Check watch cache metrics
# Cloud Monitoring → kubernetes.io/apiserver/watch_events_sizes

# Identify object types với high churn:
# Endpoints (frequently updated)
# Pod status (many updates)
# Events (high volume)

Immediate Fix:

bash

# Option 1: Increase watch cache size (if control plane has capacity)
# Note: GKE managed, không có thể adjust langsung via kubectl
# Contact Google Cloud Support để control plane tuning

# Option 2: Reduce watch activity
# Endpoints watch can be noisy:
# - Implement pod anti-affinity để reduce endpoint changes
# - Reduce pod update frequency

# Option 3: Use kube-proxy ipvs mode (faster endpoint updates)
gcloud container clusters update <cluster> \
  --enable-ip-alias --kube-proxy-mode=ipvs

Permanent Fix:

API Server resource planning:
- Large clusters need larger API server allocations
- GKE automatically scales, but has limits
- Monitor watch cache size trends
Workload optimization:
- Batch pod updates (reduce frequency)
- Avoid unnecessary status updates

Root Cause 4: Admission Webhook Timeout

Dấu hiệu:

Pod creation slower than expected
Event: "admission webhook timeout"
Webhook pod struggling (high CPU/memory)
Latency histogram show spike at operation start

Nguyên nhân: Custom admission webhooks slow, causing API server to wait. Webhook có thể:

Make external API calls (slow)
Have CPU-heavy logic
Not respecting timeout
Resource exhaustion (OOMKilled, CPU throttling)

Diagnostic:

bash

# List webhooks và check timeout
kubectl get validatingwebhookconfigurations -o yaml | grep -E "name:|timeoutSeconds"
kubectl get mutatingwebhookconfigurations -o yaml | grep -E "name:|timeoutSeconds"

# Check webhook pod status
kubectl get pods -n <webhook-namespace> -l app=<webhook-app>
kubectl logs -n <webhook-namespace> <webhook-pod> | tail -50

# Check webhook resource usage
kubectl top pod -n <webhook-namespace> <webhook-pod>

Common Issues & Fixes:

Issue	Fix
Webhook timeout set to 30s	Reduce đến 5s (faster failure)
Webhook making external API calls	Add retry/circuit breaker, implement caching
Webhook processing logic slow	Profile, optimize, reduce scope
Webhook pod OOMKilled	Increase memory requests

Immediate Fix:

bash

# Option 1: Reduce webhook timeout
kubectl patch validatingwebhookconfigurations <name> -p \
  '{"webhooks":[{"name":"<webhook>","timeoutSeconds":5}]}'

# Option 2: Disable webhook for namespace (temporary debug)
kubectl label namespace <namespace> skip-<webhook>=true

# Option 3: Implement webhook request caching
# Modify webhook logic để cache results

# Option 4: Increase webhook pod resources
kubectl patch deployment <webhook> -n <namespace> -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"requests":{"memory":"512Mi","cpu":"500m"}}}]}}}}'

Permanent Fix:

Webhook best practices:
- Timeout < 5s (fail fast)
- Stateless, no external dependencies
- Efficient CEL expressions (avoid regex)
Webhook testing trong CI/CD:
- Load test webhooks trước production deploy
- Measure latency distribution
Monitoring:
- Custom metric: webhook_request_duration_seconds
- Alert if > 1s regularly

Root Cause 5: Large Cluster (> 5000 Nodes) Scaling Limits

Dấu hiệu:

Cluster mendekati/at 5000 nodes
New nodes không join efficiently
kubelet registration timeout
Controller manager restart frequency increasing

Nguyên nhân: Control plane designed để 5000 nodes max. Melebihi đó → architectural issues:

etcd object count explosion
API server connection limit
Watch cache exhaustion

Diagnostic:

bash

# Count nodes
kubectl get nodes | wc -l

# If close to 5000, need architecture redesign
# Recommended: multi-cluster setup

# Check pod density
kubectl get pods --all-namespaces | wc -l
# Rough: pods/nodes ratio should be < 150 pods/node

Immediate Fix:

bash

# Option 1: Multi-cluster deployment
# Split workloads across multiple clusters
# Use Fleet for management

# Option 2: Large node machines (fewer nodes, more pods each)
# Reduces total node count
# Example: 5000 nodes × 1 pod = 1 node × 5000 pods (not ideal)
# Better: 1000 nodes × 50 pods/node (more balanced)

# Option 3: Workload consolidation
# Move less critical workloads to different cluster

Permanent Fix:

Multi-cluster architecture for production scale
Implement cluster federation (GKE Fleet)
Define cluster size limits trong policy

Prevention & Monitoring

Comprehensive Monitoring Setup

bash

# Create Cloud Monitoring dashboards

# Dashboard 1: Control Plane Health
# Metrics:
#   - api_server_request_duration_seconds (99th percentile)
#   - api_server_inflight_requests
#   - etcd_db_total_size_in_bytes
#   - etcd_server_has_leader

# Dashboard 2: Performance Trends
# Metrics:
#   - api_server_request_count (by operation type)
#   - api_server_watch_events
#   - kubelet_node_config_assignments

# Alert Rules:
#   - API latency > 1s for 5 min → Page on-call
#   - etcd size > 5GB → Notify ops
#   - Inflight requests > 1000 → Notify ops
#   - Watch cache churn > threshold → Alert

Preventive Configuration

yaml

# Recommend: ResourceQuota để limit API server load
apiVersion: v1
kind: ResourceQuota
metadata:
  name: api-load-limit
spec:
  hard:
    requests.cpu: "100"
    requests.memory: "200Gi"
  scopeSelector:
    matchExpressions:
    - operator: NotIn
      scopeName: PriorityClass
      values: ["system"]
---
# Cleanup controller để old Events
apiVersion: v1
kind: ConfigMap
metadata:
  name: event-cleanup
data:
  script: |
    #!/bin/bash
    # Delete events older than 7 days (cron job)
    kubectl get events --all-namespaces -o json | \
      jq '.items[] | select(.metadata.creationTimestamp < "'$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)'")' | \
      kubectl delete -f -

Escalation Criteria

Escalate to Google Cloud Support if:

API server frequently latency > 5s
etcd size > 10GB despite cleanup efforts
Control plane repeatedly restarting
Watch connections persistent timeouts
Node join failures at cluster scale limits

Pod Creation Failures — Nếu admission webhooks block
Scheduling Failures — Nếu scheduler not receiving updates
Networking Issues — Nếu apiserver latency affects DNS

Control Plane Issues: API Server & etcd Health ​

Symptoms Recognition ​

Why This Matters ​

Information Gathering — Quick Diagnostics ​

Step 1: Check API Server Health & Latency ​

Step 2: Collect Control Plane Metrics ​

Step 3: Check etcd Status ​

Step 4: Analyze Request Types ​

Step 5: Check Webhook Latency ​

Diagnostic Decision Tree ​

Common Root Causes & Fixes ​

Root Cause 1: etcd Oversize / Compaction Not Running ​

Root Cause 2: API Server CPU/Memory Overload ​

Root Cause 3: Watch Cache Exhaustion ​

Root Cause 4: Admission Webhook Timeout ​

Root Cause 5: Large Cluster (> 5000 Nodes) Scaling Limits ​

Prevention & Monitoring ​

Comprehensive Monitoring Setup ​

Preventive Configuration ​

Escalation Criteria ​

Related Sections ​

References ​

Control Plane Issues: API Server & etcd Health

Symptoms Recognition

Why This Matters

Information Gathering — Quick Diagnostics

Step 1: Check API Server Health & Latency

Step 2: Collect Control Plane Metrics

Step 3: Check etcd Status

Step 4: Analyze Request Types

Step 5: Check Webhook Latency

Diagnostic Decision Tree

Common Root Causes & Fixes

Root Cause 1: etcd Oversize / Compaction Not Running

Root Cause 2: API Server CPU/Memory Overload

Root Cause 3: Watch Cache Exhaustion

Root Cause 4: Admission Webhook Timeout

Root Cause 5: Large Cluster (> 5000 Nodes) Scaling Limits

Prevention & Monitoring

Comprehensive Monitoring Setup

Preventive Configuration

Escalation Criteria

Related Sections

References