GKE Scalability Limits: Ràng Buộc Kiến Trúc Ở Quy Mô Lớn

Tại Sao Bạn Cần Hiểu Limits

GKE clusters vượt 1000 nodes không phải là "bigger version" của clusters nhỏ — mỗi layer (control plane, data plane, networking) có hard limits do vật lý hệ thống (CPU, memory, API throughput) và thiết kế kiến trúc (etcd consistency model, API server watch mechanism, Kubernetes scheduler algorithm) quy định.

Khi cluster approaching limits:

• API latency tăng exponentially (watch connections overwhelming server)
• Scheduler convergence time blow up (predicates O(n) với tổng Pod count)
• etcd compaction lag, leading cluster instability
• Service routing latency (eBPF map lookups, endpoint cardinality)
• DNS queries timeout (CoreDNS overload, cache miss storm)

Điểm mấu chốt: Limits không phải negotiable constraints mà là physical boundaries — khi bạn exceed chúng, hệ thống không degrade gracefully, mà collapse hoặc misbehave với latency/availability impact tức thì.

Internal Model: Layers và Limits

GKE cluster có 5 layer chính, mỗi layer có distinct limits:

Layer 1: Kubernetes API Server

API server là gatekeeper của mọi Kubernetes object mutation và query.

Hard Limits:

Metric	Limit	Ghi Chú
Request rate (mutations + reads)	~10,000 req/s cho control plane	Thực tế phụ thuộc vào object size, complexity
Concurrent watch connections	~15,000 - 20,000 active watches	Mỗi kubelet mở watch tới API server, mỗi client app watch list resources
List response size	Unlimited (streaming list now GA)	Trước đây chunking bị limit, giờ streaming response encoding giải quyết
Object size	1.5 MB max per object	etcd limit, lớn hơn thì reject

Cơ Chế Vận Hành:

API server là stateless, replicated trên 3 masters (hoặc hơn nếu custom). Mỗi request:

1. Hit load balancer (GCP Cloud Load Balancer)
2. Route tới một API server replica
3. Replica contact etcd cho read/write
4. Return response

Bottleneck:

Không phải API server goroutines mà etcd throughput. API server có thể scale goroutines, nhưng backend etcd (dù là on-premise etcd hay GKE's Spanner-based etcd) có fixed throughput. Dưới 1000 nodes với normal workload (deployment, service, configmap), API throughput ổn. Nhưng:

• High Pod churn rate (preemption, autoscaling spike): Nếu 1000 Pods mỗi giây được tạo/xóa (ví dụ batch job preemption), mỗi Pod event (create, delete, run, success) tạo multiple API calls → API server struggles
• Large list operations: kubectl get all --all-namespaces or controller informers syncing on startup mỗi informer (RBAC, network policy, custom resources) làm list request → watch setup → dữ liệu khổng lồ serialize

Khi Approaching:

• API latency (p99) rises từ <10ms to 100ms+
• Scheduler watch stalls (can't see new Pods to schedule)
• Client retries amplify load
• Control plane CPU/memory spikes

Giải Pháp Design:

• Limit Pod churn: Batch operations instead of rolling updates
• Streaming list response: GKE 1.27+ uses streaming list response encoding, allows large responses chunked over time (thay vì buffer response toàn bộ)
• API server HPA: GKE tự động scale số API server replicas dựa vào load (newer GKE versions)

---

Layer 2: etcd — Distributed State Database

etcd lưu trữ mọi Kubernetes object (Pods, Services, etcd-specific ConfigMaps, CustomResources). Nó là consistency engine của cluster.

Hard Limits (Traditional etcd):

Metric	Limit	Ghi Chú
Total key-value pairs (objects)	~3 million	Vượt quá thì latency degrade, compaction fails
Database size	8 GB (default)	Có thể increase tới 16 GB with custom flag
Compaction lag	Ideally <5 minutes	Nếu lag >30min, stale data risk, watch lag
Write throughput	~5,000 writes/sec	Depends on write size

Trong GCP GKE 2025+:

GKE đang migrate từ traditional etcd tới Spanner-based key-value store. Spanner là distributed database Google dùng để scale arbitrary large (Google's internal state). Spanner-based etcd:

• Virtually unlimited scaling
• Better consistency semantics
• Managed by Google (no manual compaction headaches)

Nhưng ngay cả legacy GKE clusters vẫn dùng etcd, nên bạn cần hiểu limits.

Cơ Chế Vận Hành:

etcd là leader-based consensus system (Raft). Dữ liệu được replicated trên 3+ members:

• Leader nhận writes, synchronously replicate tới quorum (N/2+1 nodes)
• Reads có thể từ leader hoặc replicas (serializability guarantee)

Bottleneck:

1. Leader CPU bandwidth: Mỗi write phải:

   • Append tới Raft log
   • Replicate tới followers
   • Wait for quorum ACK
   • Apply tới state machine (in-memory tree)

   Ở 1000+ nodes với 100+ Pods/node, object count ~150K Pods + 50K Services + config = 300K objects. Nếu 10% Pod churn/minute = 1500 Pod events/sec = 3000+ etcd writes/sec (mỗi Pod create/delete là multiple operations). etcd leader nears saturation.

2. Database compaction: etcd giữ revision history (để support watch reconnection). Periodically (default 1 hour) compact = remove old revisions. Nếu compaction lag >30min, etcd memory bloat (revision history konsumsi RAM).

3. Watch backlog: Mỗi client (kubelet, controller) watch resources. etcd maintain per-client event queue. Nếu producer (object changes) faster than consumer (client reading), queue backlog, memory spike.

Khi Approaching:

• etcd leader CPU at 80%+
• Watch lag: client don't see object changes within milliseconds
• Compaction stalls (can't acquire mutex)
• Kubernetes objects become "invisible" for seconds → scheduling halts

Giải Pháp Design:

• Object count reduction: Use namespaces, delete old objects, avoid large ConfigMaps
• Pod churn batching: Preemption, rollouts, autoscaling → batch tối đa possible
• Watch optimization: Specify field selectors, namespace, reduce scope
• Spanner etcd migration: GKE automatically, không cần action (GA now)

---

Layer 3: Kubernetes Scheduler

Scheduler watches unscheduled Pods, filters nodes (predicates), scores nodes (priorities), binds Pod to node.

Hard Limits:

Metric	Limit	Ghi Chú
Pod scheduling rate	~100 Pods/second (traditional scheduler)	Phụ thuộc vào predicate complexity, node count
Predicate evaluation	O(n) where n = node count	Mỗi Pod → check tất cả nodes
Pending queue	~100,000 Pods	Soft limit, nếu exceed lâu scheduler fall behind

Cơ Chế Vận Hành:

Scheduler là single-threaded event loop (simplified):

1. Watch unscheduled Pods → add to queue
2. Dequeue Pod
3. For each node:
     - Run predicates (fit filters): resource request, affinity, taint/toleration, PDB
     - If pass → run priorities (scoring): zone spreading, image locality, etc
4. Bind Pod to best-scored node
5. Repeat

Bottleneck:

Nếu 1000 nodes, 10,000 Pods dalam queue:

• Masing-masing Pod scheduling = 1000 predicate evaluations (per-node)
• Predicate cost = resource request check (O(1)), affinity check (O(Pods on node)), taint/toleration (O(taints))
• Worst case: Pod dengan complex affinity (podAntiAffinity requiring topology spread) → O(10,000 * 1000) = 10M checks

Scheduler threads pada default hanya 1-2, sehingga scheduling latency bisa reach 10-30 seconds untuk Pod di ujung queue.

Kwan Approaching:

• Scheduler logs: Unable to schedule pod untuk Pods that seharusnya fit
• High latency (Pods sit Pending 30+ seconds) meskipun ada available capacity
• Affinity/anti-affinity rules conflict → Pods unschedulable indefinitely

Giải Pháp Design:

• Bin packing strategy: Prefer full-bin packing (dense placement) instead of spreading → fewer nodes checked
• Pod priority classes: Critical Pods scheduled first, non-critical defer
• Topology aware scheduling: TAS (Google Kubernetes Engine feature) optimizes scheduling untuk ML/distributed training workloads
• Custom scheduler: Untuk highly specialized workloads, custom scheduler atau external orchestration

---

Layer 4: Networking — Services, DNS, eBPF

Networking layer handles Service abstraction (load balancing) dan Pod-to-Pod communication.

Hard Limits:

Metric	Limit	Ghi Chú
Services per cluster	~5,000 (default)	Tăng possible nhưng trade-offs
Endpoints per Service (w/o eBPF)	~1,000	iptables becomes O(rules)
Endpoints per Service (w/ Dataplane V2 eBPF)	~260,000	Total across all services
Pod IPs (per node)	Limited by subnet	/24 = 256 IPs, /23 = 512 IPs, etc
DNS queries per second	~5,000 qps per CoreDNS replica	Saturates dengan high workload churn

Cơ Chế Vận Hành:

Khi Dataplane V2 OFF (legacy networking):

• Service routing via iptables rules
• Mỗi Service = iptables DNAT rule
• Mỗi endpoint = iptables rule variant
• Kernel traverses iptables chain (linear lookup) per packet

Khi Dataplane V2 ON (eBPF, Cilium):

• Service routing via eBPF programs (kernel JIT compiled)
• Service map = BPF hashmap (O(1) lookup)
• Endpoint list = compact array in BPF memory
• But: BPF map size limit = 260K total endpoints

Bottleneck:

1. eBPF endpoint limit (260K): Nếu 1000 nodes, 100 Pods/node, 50% Services dengan 50 replicas = 1000 * 50 = 50K endpoints. Scale further? Hitting limit. Workaround: frontend/backend Service pattern.

2. DNS saturation: CoreDNS forwards requests tới kube-dns. Nếu 1000 nodes, masing-masing node 100 Pods = 100K Pods, setiap Pod resolve service name = 100K DNS requests per change cycle. CoreDNS can't keep up → cache miss → latency spike.

3. IP exhaustion: Pod IP per node depends on subnet. Default secondary CIDR /21 = 2048 IPs. 1000 nodes = 1000 * 2048 = 2M IPs needed. VPC size matters.

Kwan Approaching:

• Service latency spikes (long iptables chain or BPF map stress)
• DNS resolution timeouts
• Pod creation fails: "no IP available"

Giải Pháp Design:

• Dataplane V2 mandatory untuk large clusters (eBPF efficiency)
• NodeLocal DNSCache: Reduce CoreDNS load with per-node DNS cache
• IP planning: Allocate generous secondary CIDR upfront
• Service aggregation: Limit total Services count (combine dengan backend routing)

---

Layer 5: Controller Manager — Reconciliation Loop

Controller manager runs controllers (Deployment, StatefulSet, DaemonSet, ReplicaSet, etc) which continuously reconcile desired state (spec) dengan actual state (status).

Hard Limits:

Metric	Limit	Ghi Chú
Controllers	~20 built-in controllers	Each watches resources, reconciles
Goroutines per controller	~10-20 (workers)	Tunable, but overhead increases
Work queue size	Unbounded (memory-limited)	If growing, controller falling behind
Reconciliation latency	Ideally <1 second	Nếu >10sec, object churn atau controller lag

Cơ Chế Vận Hành:

Mỗi controller:

1. Watch resource (Deployment) via informer
2. Add changed object ke work queue
3. Dequeue, reconcile (create/update/delete Pods to match spec)
4. Re-enqueue jika failed

Bottleneck:

Nếu 1000 Deployment, masing-masing Deployment spawn 100 Pods = 100K Pods. Deployment controller watches all Pods:

• Mutable field change (status.replicas, status.conditions) → 100K updates dalam queue
• Worker goroutines (say 16) dequeue dan process
• Masing-masing reconciliation = recompute Pods needed, diff, create/delete

Nếu churn rate high (rollout, autoscale up/down), queue overflow → lag → Deployments out-of-sync dengan spec.

Kwan Approaching:

• Work queue size growing unbounded
• Controller reconciliation latency increasing
• Rolled Deployments stuck in partial state
• Memory growth unchecked (queue backing up in RAM)

Giải Pháp Design:

• Rate limiting: Explicit rate limits on controller queue processing
• Batch reconciliation: GKE features like batch Pod creation
• Custom controllers: Reduce built-in controller load dengan uninstal unused ones

---

Tổng Hợp: Limits Across Layers

Khi designing 1000+ node cluster, think about cascade effect:

High Pod churn
    ↓ (multiple events per Pod)
API server throughput increases
    ↓ (etcd write storm)
etcd compaction lag, watch backlog
    ↓ (scheduler can't see Pods)
Scheduler queue buildup, scheduling latency
    ↓ (Pods pending longer)
Pod startup delayed, app SLO miss

Key Insights:

1. Chaining: Bottleneck at one layer cascades upstream
2. Predictability: Know your workload's Pod churn pattern → estimate API/etcd load → size cluster accordingly
3. Monitoring: Must observe each layer's metrics:
   • API server: request latency, goroutine count
   • etcd: leader CPU, write latency, compaction lag
   • Scheduler: pending Pod count, scheduling latency
   • Controller manager: work queue depth, reconciliation latency

Practical Guidance: Approach Limits Safely

If you're >80% of a limit:

1. Immediate (hours): Reduce workload intensity (batch operations, reduce churn)
2. Short-term (days): Shard cluster (split into multiple smaller clusters) or promote higher capacity pool (more nodes? Larger nodes?)
3. Medium-term (weeks): Architecture redesign:
   • Service aggregation (fewer but larger Services)
   • Pod consolidation (fewer replicas, over-provision compute)
   • Namespace isolation (distribute workload across namespaces with quotas)

If you're >95% of a limit:

Cluster is at critical stability risk. Operations like scaling, upgrades, failures could cause cascading failures. Consider emergency measures (temporary shard, workload migration).

References

• GKE Best Practices for Planning Large Clusters
• GKE Quotas and Limits
• GKE Scalability & Performance
• How We Built a 130,000-Node GKE Cluster
• Kubernetes API Conventions