My Awesome Project

Appearance

Controller Manager Scalability: Work Queues, Workers, Reconciliation ​

Role of Controller Manager ​

Controller manager runs ~20 Kubernetes built-in controllers (Deployment, StatefulSet, Job, ReplicaSet, Service, Endpoint, Node, DaemonSet, PVC, etctera). Mỗi controller:

  1. Watches resource (Pod, Deployment) via informer
  2. Enqueues changes vào work queue
  3. Dequeues and reconciles (apply desired state)

At scale (1000 nodes, 100K Pods):

  • Deployment controller watches 100K Pods, maintains reconciliation queue
  • StatefulSet controller watches subset Pods
  • Job controller tracks job completion
  • PVC controller manages volumes

Nếu controller falls behind, resources go out-of-sync với spec (e.g., Deployment.replicas=100 but only 50 Pods running).

Internal Model: Reconciliation Loop ​

Simplified Controller Loop ​

go
for {
  // 1. Dequeue item from queue
  item := queue.Get()
  
  // 2. Reconcile: Fetch current object, apply desired state
  obj := apiserver.Get(item.key)  // e.g., fetch Deployment
  desired := obj.Spec
  actual := obj.Status
  
  // 3. Compute diff (what Pods to create/delete)
  diff := computeDiff(desired, actual)
  
  // 4. Apply changes
  for pod := range diff.toCreate {
    apiserver.Create(pod)
  }
  for pod := range diff.toDelete {
    apiserver.Delete(pod)
  }
  
  // 5. Mark success and remove from queue
  queue.Done(item)
  
  // If failed, re-queue with backoff
  if err != nil {
    queue.AddRateLimited(item)
  }
}

Concurrency: Controller typically runs 10-16 workers (concurrent reconciliations). More workers = higher throughput, but higher CPU/memory overhead.

Work Queue Mechanics ​

GKE uses rate-limited work queues (token bucket algorithm):

queue.AddRateLimited(item)

Rate limiting: prevent thundering herd (e.g., 1000 Pods become unschedulable simultaneously → shouldn't retry 1000 times immediately).

Queue behavior:

  • Max retries per item: ~15 (exponential backoff)
  • Backoff: 1ms → 2ms → 4ms ... → 40s
  • After max retries, item dropped

Size: Queue unbounded by default (memory-limited). If controller can't keep up:

  • Queue depth grows
  • Memory usage increases
  • Eventually: OOMKill

Bottleneck: Reconciliation Latency ​

Scenario: Deployment Rollout ​

User creates Deployment:

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 100
  selector:
    matchLabels:
      app: app
  template:
    spec:
      containers:
      - name: app
        image: app:v1

Deployment controller:

  1. Watches Deployment create event
  2. Enqueues "app" to work queue
  3. Dequeues "app" (1-100ms latency)
  4. Fetches current Deployment from API server
  5. Computes desired Pods: 100 replicas
  6. Creates 100 Pods sequentially (or batched)
  7. Updates Deployment status.replicas

Latency: 100 Pod creations × 100ms per create (API server RPC) = 10 seconds until all Pods created.

At scale: If 10 Deployments rolling out simultaneously:

  • All 10 Deployments competing for API server
  • API server bandwidth limited
  • Deployment controller queue grows
  • Reconciliation latency increases

Optimization: Batch Operations ​

Problem: Creating 100 Pods sequentially = 100 API calls (expensive).

Solution: Batch creation

Some controllers (Deployment, StatefulSet in GKE) batch Pod creation:

POST /api/v1/namespaces/default/pods  // create multiple Pods

But Kubernetes API doesn't natively support batch Pod create. Workaround: client-side batching (create 10 Pods per RPC, reduce calls from 100 to 10).

Result: 10 API calls × 100ms = 1 second, much faster.

Controller Manager Scaling Limits ​

CPU and Memory ​

Default GKE controller manager:

  • 2 CPU cores
  • 2 GB memory

For 1000-node cluster with workload intensity, this might be insufficient. Controllers CPU spike when:

  • Rolling updates (reconciling Deployments)
  • Autoscaling (node pool scale up/down)
  • Large ConfigMap/Secret changes

Scaling option: Request more CPU/memory for controller-manager (custom GKE configuration, requires cluster API change).

Queue Depth ​

If queue depth growing unbounded:

controller_queue_depth{controller=deployment,quantile="0.99"}  // p99 queue size

Indicates controller can't keep up. Latency for new object reconciliation increases.

Typical resolution: Reduce workload churn (less rolling updates, batch autoscaling).

Large-Scale Pattern: Workload Distribution ​

At 1000+ nodes, single cluster controller manager can bottleneck. Solutions:

Pattern 1: Single Cluster (Accept Latency) ​

  • Single controller manager
  • Tolerate 30-60 second reconciliation latency for new Deployments
  • Monitor queue depth

Best for: Batch workloads (latency-sensitive apps need lower latency).

Pattern 2: Shard Across Namespaces ​

Custom controllers handle specific namespaces:

  • Main controller: watches all Deployments
  • Custom webhook: route namespace to specific "sub-controller" instance
  • Each sub-controller reconciles subset of Deployments

Complexity: Custom logic, testing burden.

Pattern 3: Multi-Cluster Orchestration ​

Instead of 1000-node GKE cluster:

  • 10 × 100-node clusters
  • Orchestrator distributes workload across clusters (e.g., Karmada, Flux)
  • Each cluster has own controller manager (lighter load)

Trade-off: Operational complexity (multi-cluster state management).

Diagnosis: Controller Manager Lag ​

Metrics ​

controller_reconcile_latency_seconds{controller=deployment}  # p99 reconciliation time
controller_queue_length{controller=deployment}               # queue backlog
controller_max_concurrent_reconciles                         # worker count

Warning Signs ​

  1. Growing queue depth: Dequeue rate < enqueue rate
  2. High reconciliation latency: Takes minutes to reconcile Deployment change
  3. Controller manager CPU maxed: Threads can't process queue faster

Remediation ​

  1. Immediate: Reduce workload churn (batch operations)
  2. Short-term: Increase controller manager resources (CPU/memory)
  3. Medium-term: Shard workload (multi-cluster, namespace-based routing)

Real-World Scenario: Batch Job Explosion ​

Case: 1000-node cluster, batch job framework submits 10,000 Jobs simultaneously.

Impact:

  1. Job controller watches all Jobs
  2. 10,000 Jobs enqueued
  3. Job controller workers (default 5) dequeue 5 at a time
  4. Each Job reconciliation: create Pods, monitor completion
  5. API server flooded with 10,000+ Pod create requests
  6. API server latency spikes
  7. Informers fall behind (watch lag)
  8. Controller manager falls behind

Resolution:

  • Rate limit job submission (e.g., 100 Jobs/sec instead of 10K/sec)
  • Increase Job controller workers (--concurrent-job-syncs=20)
  • Batch Pod creation within Job controller
  • Monitor queue depth, alert if >5K

References ​