API Server at Scale: Request Routing, Watch, Latency

Tại Sao API Server Là Bottleneck

API server là single point of coordination cho mọi cluster state change. Mọi object creation/update/delete, mọi watch subscription đều phải qua API server. Ở 1000+ nodes:

• 1000 kubelets, mỗi informer (pod, endpoint, configmap) watch → 3000+ watch connections
• 100+ controllers (Deployment, StatefulSet, Job, Custom) mỗi watch resources → additional 100+ watches
• Client applications (sidecar injectors, webhooks, operators) watch → 100+
• Total: 5000-10000 concurrent watch connections

Mỗi watch = stream connection giữ mở. API server phải maintain state, buffer events, serialized các changes.

Bottleneck: Streaming response serialization, watch event buffering, concurrent connection overhead.

Internal Model: API Server Request Flow

Request Path

Client (kubelet, controller, app)
  ↓ (HTTP/2 request)
GCP Cloud Load Balancer (TCP LB)
  ↓ (route to API server replica via etcd LB cookie persistence)
API Server (1 of 3-5 replicas, stateless)
  ↓ (authenticate, authorize via RBAC)
etcd (single leader)
  ↓ (read/write)
Response back through chain

Load balancer routing: API server replicas are stateless. LB routes request tới any replica. But watch connections are stateful (client must connect to replica handling watch stream). GKE uses session affinity (cookie) tới stick client to one replica untuk watch operations.

Watch Mechanism

Watch client (kubelet):

// Pseudocode
req := &watchRequest{
  resourceVersion: "12345",  // start watching from this revision
  fieldSelector: "status.phase=Pending",
  watch: true
}
response := apiServer.list(req)

API server:

1. Fetch current objects matching selector from etcd
2. Start watch stream (HTTP chunk encoding)
3. Buffer events from etcd watch stream
4. Send to client as they occur
5. Keep connection alive until client disconnects or timeout

Event types in stream:

• ADDED: object created or matching selector
• MODIFIED: object changed
• DELETED: object deleted
• ERROR: watch stream error (must restart)

Bottleneck mechanism:

If etcd is producing events faster than client consuming them:

• API server buffer grows (memory pressure)
• Watch lag increases (client sees old state)
• Eventually buffer exceeds limit → API server closes connection → client reconnects → thundering herd

Scalability Challenge: Watch Connections

Problem: Connection Overhead

Mỗi watch connection:

• OS socket (file descriptor)
• Goroutine (memory overhead ~2MB per goroutine)
• Event buffer (ring buffer, typically 100-1000 events)
• Network bandwidth (serialized JSON events)

Dengan 10,000 watches:

• 10,000 goroutines = 20GB memory just for goroutines (tidak feasible)
• 10,000 sockets = OS file descriptor limit (default 65K per process, dapat increase)
• Serialization CPU spike jika many events per second

GKE's Solution: Streaming List Response Encoding

Before (deprecated): List response buffered fully, serialized as one large JSON array:

{
  "items": [
    {Pod object 1},
    {Pod object 2},
    ...
    {Pod object 10000}
  ]
}

API server uses HTTP chunked encoding, but client waits for complete response sebelum processing. Memory intensive.

Now (GA in GKE 1.27+): Streaming list response:

200 OK
Transfer-Encoding: chunked

{"kind":"Pod",...}\n
{"kind":"Pod",...}\n
...

Each object sent individually, client can process as it arrives. API server doesn't need buffer huge list in memory.

Benefit: Can list 10,000 Pods without API server memory spike. Client processes streaming.

Downsides:

• Client must handle chunked stream correctly
• Older clients might not support (falling back to buffering)

Scalability: Request Rate & Latency

Request Rate Limits

API server components:

• apiserver goroutines: default 1000s (tunable), handles concurrent requests
• etcd leader throughput: ~5,000 writes/sec (fundamental limit)
• network bandwidth: GCP LB bandwidth = generous (not bottleneck typically)

Actual limit: etcd throughput when cluster heavy on writes (Deployment rollout, Pod creation storm).

Monitoring API server performance:

apiserver_request_duration_seconds{verb=POST}  // watch latency (p99)
apiserver_request_total                        // request rate
apiserver_inflight_requests                    // current in-flight
etcd_request_duration_seconds{operation=Put}   // etcd latency

Latency SLOs

GKE publishes SLOs for API server:

• p99 read latency (GET): <100ms for 1000-node cluster
• p99 mutation latency (POST, PUT, DELETE): <300ms
• watch latency (time for API server to send event to client): <5 seconds

When approaching saturation:

• Latency degrades non-linearly (p99 → p95 → p90)
• Eventually some requests timeout
• Clients retry → amplified load
• Cascading failure

Tuning API Server for Large Clusters

1. Max Requests Inflight

--max-requests-inflight=800  # HTTP/2 requests
--max-mutating-requests-inflight=600  # safe defaults

Too low → legitimate requests rejected Too high → memory spike, GC pauses

2. Watch Cache

--watch-cache=true  # default
--watch-cache-sizes=
  Pod*=100,
  ReplicationController=50,
  ...

Watch cache = in-memory index of recent versions (keyed by ResourceVersion). Allows fast list response for watches without hitting etcd every time.

3. Event Broadcast Buffer Size

--event-ttl=1h                    # how long events stay in system
--event-recorder-qps=50           # rate limit event recording

Events are spammy. Limiting event recording QPS reduces API load.

4. Streaming List Chunk Size

--streaming-list-chunk-size=5000  # objects per chunk

Trade-off: smaller chunks = more network packets, larger = more latency before first object. 5000 is balanced.

Addressing Watch Bottleneck

Problem: Informer Thundering Herd

When API server fails over or watch closes, all kubelets/controllers reconnect simultaneously:

1000 kubelets reconnect at same time
Each watch list (Pod, ConfigMap, Service, etc) = 3 list requests
Informer resync period = 30 minutes = full re-list every 30 min
→ 3000+ concurrent list requests
→ API server goroutines maxed
→ new watches get queued

Mitigations

1. Stagger informer sync times: Use different resync periods per informer:

kubeInformerFactory := kubeinformers.NewSharedInformerFactory(
  clientset,
  30*time.Minute,  // base
)
// Add jitter per informer
resyncPeriod := 30*time.Minute + time.Duration(rand.Intn(5))*time.Minute

2. Reduce watch scope: Use field selector, namespace selector instead of cluster-wide watch:

watchOpts := metav1.ListOptions{
  FieldSelector: "metadata.namespace=default",
  Watch: true,
}

3. Filtering at API server: GKE can apply server-side filtering before returning objects.

References

• Kubernetes API Server
• GKE Best Practices for Scalability
• Understanding Kubernetes Watch