My Awesome Project

Appearance

Control Plane Debugging — API, etcd, Scheduler, Webhooks ​

Tại sao quan trọng ở Production ​

Control plane adalah "brain" của cluster. Kalau control plane slow atau broken:

  • All Kubernetes API calls slow (kubectl commands hang)
  • Scheduler berhenti place pods baru (Pending pods)
  • New deployments rollout hang
  • ConfigMap/Secret updates tidak propagate
  • Webhooks timeout → admission reject valid requests

Operator sering tidak realize ini control plane problem karena mereka fokus pada pod/node. Tapi kubectl hang atau api-server not responding adalah giveaway.

Internal Model: Control Plane Architecture ​

Control Plane Components in GKE ​

GKE managed control plane (bukan self-managed like kubeadm):

GCP Managed Control Plane (Highly Available):
├── API Server (multiple replicas)
├── etcd (persistent data store)
├── Scheduler
├── Controller Manager
└── Cloud Auditing Layer

GCP manage HA, so usually not totally down. Tapi bisa slow.

API Server: Request Path ​

Ketika kubectl run atau application make API call:

1. kubectl call → API Server HTTP/2 endpoint
2. API Server receive request
3. Authentication (check certificate/token)
4. Authorization (check RBAC)
5. Admission Control (validating/mutating webhooks)
6. Request processing (create object, update state)
7. etcd write (persist object)
8. Webhooks fire (post-admission)
9. Return response

Jika slow anywhere dalam path → entire request slow.

etcd: Persistent State ​

etcd adalah key-value store. Setiap Kubernetes object (Pod, Service, etc) store di etcd:

/kubernetes.io/namespaces/default
/kubernetes.io/pods/default/my-pod
/kubernetes.io/services/default/my-service
...

etcd performance critical karena:

  • Every object write must persist (synchronous)
  • Slow etcd write → all object create/update slow
  • Scheduler read pod/node info dari etcd → slow etcd → slow scheduling

Scheduler: Pod Placement Decision ​

Scheduler watch unscheduled pods, try place on nodes:

1. Watch for Pending pods
2. For each pod:
   a. Filter nodes (node selector, affinity, resource fit)
   b. Score nodes (spreading, preferred affinity)
   c. Bind pod to best-scoring node
3. If bind fail → pod stay Pending

Slow scheduler bisa from:

  • High scheduler latency (take long to filter/score)
  • etcd slow (cannot read pod/node info fast)
  • Many pending pods (overload scheduler)

Admission Webhooks: Validation & Mutation ​

Validating Webhook:
  ├── Check request (e.g., image must from approved registry)
  └── If fail → reject request (HTTP 400/403)

Mutating Webhook:
  ├── Modify request (e.g., inject sidecar)
  └── Return modified request → continue processing

Webhooks call external service (you deploy it). Kalau webhook slow atau timeout:

  • Slow to create/update resources
  • Eventually timeout → request fail → resource not created

Debugging API Server Latency ​

Symptom: kubectl Slow ​

bash
# kubectl take >30s to return result
time kubectl get pods
# real 0m35.234s

# Compare with direct API call
time curl https://<api-server>/api/v1/pods
# Also slow? → API server problem
# Fast? → kubectl authentication overhead

Step 1: Check API Server Availability ​

bash
# Check if API server responding at all
kubectl version
# If timeout/error → API server problem

# Check component health
kubectl get componentstatuses
# Deprecated in 1.19+, but might still work
# Or check control plane dashboard in GCP console

Step 2: Identify Bottleneck ​

bash
# Enable API server request auditing (GKE can do this)
# Then check logs for request latency

# Or manually check:
# Option A: Check etcd performance
# Option B: Check scheduler latency
# Option C: Check admission webhook latency

Step 3: Check etcd Performance ​

etcd expensive operation: list all objects, watch objects.

bash
# Via Control Plane Logs (GCP Cloud Logging)
gcloud logging read \
  'resource.type="k8s_cluster" AND jsonPayload.component="etcd"' \
  --limit 50

# Look untuk slow operations:
# "operation-duration" > 1 second
# "apply-index-duration" > 1 second

# From API server logs, check etcd calls:
gcloud logging read \
  'resource.type="k8s_cluster" AND jsonPayload.component="kube-apiserver"' \
  --limit 50

# Search untuk "etcd" latency messages

If etcd slow:

  1. Check etcd member status:

    bash
    # Can't direct query etcd (GCP managed), but can check via API server logs
# API server health endpoint might report etcd latency

  2. Check database size:

    bash
    # Via GCP console → GKE cluster → storage quota
# etcd default storage: ~1GB per 100k objects
# If exceed quota → evict old data or increase quota

  3. Check for large objects:

    bash
    # Get all objects sizes
kubectl api-resources --no-headers=true | awk '{print $1}' | while read r; do
  echo "$r: $(kubectl get $r -A --no-headers 2>/dev/null | wc -l)"
done

# Find large CRDs
kubectl get crds --sort-by=.status.storedVersions

Step 4: Check Scheduler Latency ​

Scheduler slow = new pods pending longer.

bash
# Check scheduler logs
gcloud logging read \
  'resource.type="k8s_cluster" AND jsonPayload.component="scheduler"' \
  --limit 50

# Look untuk "operation-latency" or "schedule-attempt-duration"
# Normal: <100ms
# Slow: >1s

If scheduler slow:

  1. Check pending pods count:

    bash
    kubectl get pods --field-selector=status.phase=Pending --all-namespaces
# Many pending? → scheduler overload

  2. Check node count:

    bash
    kubectl get nodes
# Many nodes (>500)? → scheduler filter/score slower

  3. Check complex affinity rules:

    bash
    kubectl get pods -o yaml | grep -A10 "affinity:"
# Complex pod/node affinity → scheduler slower

Solution:

  • Scale scheduler replicas (not typical in GKE managed, but possible)
  • Simplify pod affinity rules
  • Use node labels for better filtering

Step 5: Check Admission Webhook Latency ​

Admission webhook timeout common cause of create/update hang.

bash
# Check ValidatingWebhookConfiguration
kubectl get validatingwebhookconfigurations
kubectl describe validatingwebhookconfigurations <name>

# Check MutatingWebhookConfiguration
kubectl get mutatingwebhookconfigurations

# Check timeout setting
kubectl get validatingwebhookconfigurations -o yaml | grep -i timeout
# Default: 10 seconds
# If webhook take >10s → timeout → request fail

Debug slow webhook:

  1. Check webhook logs:

    bash
    # Webhook is user-deployed pod
kubectl logs -f -l app=webhook

  2. Check webhook service latency:

    bash
    # From pod in cluster
kubectl exec <pod> -- time curl https://<webhook-service>/validate

  3. Check if webhook backing service healthy:

    bash
    kubectl get endpoints <webhook-service>
# Should have addresses (backends)

kubectl get pods -l app=webhook
# Should be running

If webhook too slow:

  1. Increase timeout (in webhookconfiguration):

    yaml
    timeoutSeconds: 30  # Increase from 10

  2. Optimize webhook code (less processing)

  3. Scale webhook replicas

  4. Use failure policy:

    yaml
    failurePolicy: Ignore  # Or Fail
# Ignore: if webhook fail, let request through
# Fail: if webhook fail, reject request

Debugging Scheduler Failures ​

Symptom: Pods Stuck Pending Despite Free Resources ​

bash
kubectl get pods
# STATUS: Pending, no "reason" visible

kubectl describe pod <name>
# Events section might show "Unschedulable" or "Failed to bind"

Root Causes ​

1. Scheduler not running
   - Check scheduler pod: kubectl get pods -n kube-system | grep scheduler

2. Scheduler cannot read pod/node info
   - Check scheduler logs for API errors

3. Scheduler cannot bind pod to node
   - Check kubelet on target node

4. Pod affinity not satisfiable
   - Pod require specific node label → label missing
   - Pod require co-location with other pod → pod not scheduled yet

Debugging Affinity Issues ​

bash
# Check pod affinity rules
kubectl get pod <name> -o yaml | grep -A20 "affinity:"

# Example: nodeAffinity
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: gpu
          operator: In
          values:
          - "true"

# Check if node exist with this label
kubectl get nodes --show-labels | grep gpu
# If no node with gpu=true → pod cannot schedule

# Fix: either
# 1. Add label to node: kubectl label nodes <node> gpu=true
# 2. Remove/relax affinity requirement

Debugging Pod Affinity (Inter-Pod) ​

yaml
affinity:
  podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - database
      topologyKey: kubernetes.io/hostname
# Meaning: schedule on same node as pod with app=database

Debug:

bash
# Check if matching pod exist
kubectl get pods -l app=database
# If no pods → new pod cannot schedule (chicken-egg problem)

# Solution: deploy database pod first, then depend pods

Debugging Webhook Failures ​

Symptom: create/update Resources Fail ​

bash
kubectl apply -f my-resource.yaml
# Error: admission webhook validation failed
# or webhook timeout

Check Webhook Configuration ​

bash
# List all webhooks
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

# Describe webhook
kubectl describe validatingwebhookconfigurations <name>
# Check:
# - clientConfig.service (webhook service address)
# - namespaceSelector (which namespaces apply)
# - objectSelector (which objects)
# - timeoutSeconds
# - failurePolicy

Debug Webhook Connectivity ​

bash
# Check webhook service exist
kubectl get service <webhook-service> -n <webhook-namespace>

# Check endpoints (backend pods)
kubectl get endpoints <webhook-service> -n <webhook-namespace>
# Should have IPs

# Test connect to webhook (from API server pod would be ideal, but hard to exec)
# Instead, check webhook pod logs
kubectl logs -l app=webhook -n <webhook-namespace>
# Look untuk request entries and response status

Webhook Timeout Issue ​

bash
# If webhook pod slow or crash:

# 1. Check pod status
kubectl get pods -l app=webhook
# NotReady or CrashLoopBackOff?

# 2. Check pod logs
kubectl logs <webhook-pod>

# 3. Check webhook latency
# Add timing in webhook code, or use APM tool

# 4. Solution: increase timeoutSeconds, or fix slow webhook

GKE-Specific Control Plane Debugging ​

GCP Cluster Control Plane Status ​

bash
# Check control plane health via gcloud
gcloud container clusters describe <cluster-name> \
  --format='value(status,statusMessage)'

# Check control plane add-ons
gcloud container clusters describe <cluster-name> \
  --format='value(addonsConfig)'

# Check master authorized networks (might block API access)
gcloud container clusters describe <cluster-name> \
  --format='value(masterAuthorizedNetworksConfig.cidrBlocks[*].cidrBlock)'

Cloud Monitoring for Control Plane ​

GCP provide metrics for managed control plane:

bash
# Check API server request latency
gcloud monitoring timeseries list \
  --filter='metric.type="kubernetes.io/apiserver/request_duration_seconds"' \
  --format='table(metric.labels, points[0].value.double_value)'

# Check etcd latency
gcloud monitoring timeseries list \
  --filter='metric.type="kubernetes.io/etcd/server/has/leader"' \
  --format='table(points[0].value.double_value)'

Cloud Audit Logs ​

GCP automatically audit Kubernetes API requests:

bash
# Check who created/modified resources
gcloud logging read \
  'resource.type="k8s_cluster" AND operation.response.@type="CreateStatus"' \
  --limit 10 \
  --format='table(timestamp, operation.first(protoPayload.request.name))'

# Check which resources modified at specific time
gcloud logging read \
  'timestamp>="2025-06-24T10:00:00Z" AND resource.type="k8s_cluster"' \
  --limit 50

Operational Practices ​

1. Monitor Control Plane Health ​

bash
# Prometheus/Cloud Monitoring alerts:
# - kube_apiserver_request_duration_seconds_bucket{le="1"} < 0.95  # 95% requests <1s
# - etcd_disk_backend_commit_duration_seconds_bucket{le="0.1"} > 0.9  # 90% commits <100ms
# - scheduler_schedule_attempts_total{result="unschedulable"} rate increase

2. Webhook Best Practices ​

yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: my-webhook
webhooks:
- name: validation.example.com
  clientConfig:
    service:
      name: webhook
      namespace: webhooks
      path: "/validate"
    caBundle: LS0tLS1...  # Base64 CA cert
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["apps"]
    apiVersions: ["v1"]
    resources: ["deployments"]
  timeoutSeconds: 5
  failurePolicy: Fail
  admissionReviewVersions: ["v1"]
  sideEffects: None
  namespaceSelector:
    matchLabels:
      webhook: enabled  # Only apply to labeled namespaces

Best practices:

  • Set appropriate timeout (not too long, not too short)
  • Use namespace selector (don't apply globally)
  • Keep webhook simple (fast to execute)
  • Scale webhook replicas
  • Use structured logging untuk debug

3. Manage etcd Size ​

bash
# Prevent unbounded growth:

# 1. Set object quotas per namespace
resourcequota:
  hard:
    pods: "100"
    configmaps: "50"

# 2. Periodic cleanup of old objects
# CronJob to delete old Jobs, Pods, Logs

# 3. Monitor etcd size
kubectl describe nodes -l cloud.google.com/gke-nodepool=...
# Under "Allocated resources"
# Watch storage trends

References ​