Node Pool Planning: Sizing, Density, và Optimization

Tại Sao Node Pool Planning Là Quyết Định Kiến Trúc

Ở quy mô 1000+ nodes, node pool decisions được bảo hành tại cluster creation time không thể thay đổi lẻ loi. Thay đổi node type, machine shape, hoặc CIDR allocation sau này yêu cầu cluster recreation (hoặc khoá cluster trong tháng) — overhead không chấp nhận được.

Quyết định critical:

• Node machine type (CPU, memory) → direct impact đến Pod density, scheduling flexibility, upgrade window
• Pod density target (Pods/node) → networking, resource utilization, failure domain blast radius
• Node pool count (cộng lại size) → flexibility của scheduling, overprovisioning buffer, upgrade strategy

Sai lầm common: "Start small, scale as needed." Cách này làm việc tới ~200 nodes. Ở 1000+ nodes, tìm tối ưu cần toàn cục decision, không incremental tinkering.

Internal Model: Node Pool Mechanics

Kubernetes Node Abstraction

Mỗi node trong GKE cluster là Compute Engine virtual machine wrapped trong Kubernetes Node object. Node object store:

apiVersion: v1
kind: Node
metadata:
  name: gke-cluster-node-xxxxx
  labels:
    cloud.google.com/gke-nodepool: default-pool
    kubernetes.io/hostname: gke-cluster-node-xxxxx
spec:
  taints:
    - key: cloud.google.com/gke-preemptible
      value: "true"
      effect: NoSchedule
  allocatable:
    cpu: "3900m"           # requested CPU = machine CPU - reserved
    memory: "12884Mi"      # requested memory = machine memory - reserved - OS overhead
    ephemeral-storage: "..." 
    pods: "110"            # max Pods per node
status:
  conditions:
    - type: Ready          # node healthy
      status: "True"
  allocatable: ...
  capacity: ...

Allocatable ≠ Capacity:

• Capacity: Total resources (physical CPU/memory)
• Allocatable: Capacity - kubelet reserved resources (OS, system daemons, kubelet memory)

Scheduler respects allocatable khi placing Pods.

Pod Density Constraints

Max Pods per node = min of:

1. Resource constraint: Allocatable resources / Pod resource requests

   • Ví dụ: 4 CPU node, mỗi Pod request 100m = 40 Pods (worst case)
   • Nhưng thực tế: mixed workload, some Pods 50m, some 500m → actual ~20-30 Pods

2. IP constraint: Allocatable IPs per node (GKE secondary CIDR)

   • Node subnet allocation = /24 = 256 IPs per node (minus gateway, broadcast) = ~250 usable
   • Default GKE reserves ~50 for system Pods (kube-system, kube-node-lease) → ~200 for user Pods
   • Can increase subnet size (/23, /22) but comes with VPC CIDR overhead

3. Kubelet enforced limit: maxPods flag (default 110 for GKE)

   • Soft limit, can override pero not recommended (kubelet might OOM)

4. etcd object count: Total Pod count across cluster (mentioned in previous section)

   • Soft limit, but impacts API server/etcd performance

Formula for safe Pod density:

Max Pods per node = min(
  resource_allocatable / avg_pod_request,
  ip_per_node - system_pods,
  kubelet_max_pods,
  (cluster_etcd_object_limit / total_nodes) * safety_factor
)

Ví dụ calculation cho 1000-node cluster:

• Node machine: n2-standard-8 (8 CPU, 32 GB memory)
• Allocatable CPU: 7600m (reserve ~400m for OS)
• Allocatable memory: 29 GB (reserve ~3GB for OS)
• Pod resource request: avg 200m CPU, 500MB memory (mixed latency-sensitive + batch)
• IP per node: /24 = 200 usable for Pods
• Kubelet max: 110
• etcd object limit: ~3M objects, 1000 nodes → ~3000 objects/node, but Pods only ~70%, so ~2100 Pods/node cluster-wide

Calculation:

• CPU-limited: 7600m / 200m = 38 Pods
• Memory-limited: 29GB / 0.5GB = 58 Pods
• IP-limited: 200 Pods
• Kubelet: 110 Pods
• etcd: (3M * 70% / 1000) = 2100 Pods/node, way over
• Conservative pick: 50-60 Pods per node

Tại 50 Pods/node * 1000 nodes = 50K Pods — manageable etcd (well below 300K object limit).

Design Patterns: Node Pool Strategy

Pattern 1: Single Large Pool

Topology:

cluster
└── default-pool (1000 nodes, same machine type)

Pros:

• Simple: single pool = simpler scheduling, affinity rules
• Better bin-packing: scheduler sees 1000 nodes, can optimize placement
• Fewer failure domains: upgrade affects all nodes together, but faster overall

Cons:

• All-or-nothing: node type change requires cluster recreation
• No workload isolation: latency-critical + batch in same pool → priority scheduling conflicts
• Upgrade window: 1000 nodes = long upgrade, need high surge (many transient nodes)

Best for:

• Homogeneous workload (e.g., pure batch processing)
• High-frequency scaling (workload pattern varies, need flexibility)
• Cost-optimized: single machine type avoids premiuming for specialized hardware

Pattern 2: Multiple Specialized Pools

Topology:

cluster
├── latency-sensitive-pool (200 nodes, n2-standard-16, max 30 Pods/node)
├── batch-pool (600 nodes, n2-standard-4, max 80 Pods/node)
├── gpu-pool (50 nodes, n1-standard-8 + Tesla V100, max 8 Pods/node)
└── system-pool (150 nodes, n2-standard-2, system workloads + buffer)

Pros:

• Workload-aware: latency workload not starved by batch job burst
• Flexible scaling: batch-pool scale down when idle, latency-pool keep warm
• Blast radius: node pool failure affects subset of workload

Cons:

• Scheduling complexity: Anti-affinity rules, pod requests must match pool availability
• Fragmentation: single Pod size doesn't fit any pool → unschedulable (e.g., 32 CPU Pod in 4CPU batch pool)
• Over-provisioning: need minimum node per pool even if unused
• Expensive to rebalance: new workload type needs new pool

Best for:

• Diverse workload types (ML training + web serving + batch)
• Workload with strict SLO (isolate critical from bursty)
• Predictable usage pattern (workload type distribution known)

Pattern 3: Hybrid — Consolidation + Specialization

Topology:

cluster
├── general-pool (800 nodes, n2-standard-4, 70 Pods/node)
├── gpu-pool (50 nodes, GPU, needed by specific apps)
└── overload-burstable-pool (100 nodes, preemptible, non-critical)

Trade-off sweet spot:

• Primary workload dalam general-pool (good bin-packing, simple scheduling)
• Specialized workload (GPU, memory-heavy) dalam dedicated pools
• Burstable workload dalam preemptible pool (cost saving)

Sizing Strategy at 1000+ Nodes

Step 1: Estimate Workload Pod Count

Input:

• number of services
• target Pod replicas per service (median)
• system Pods overhead (~50 for kube-system, ~5 per app namespace)

Ví dụ:

• 200 services, 30 replicas each (avg) = 6000 Pods
• 100 app namespaces, 5 system Pods each = 500 Pods
• kube-system (40), kube-node-lease (3) = 43 Pods
• Total: ~6500 Pods

Step 2: Determine Pod Density

Based on resource requests + network constraint:

Pod density = min(
  available_cpu / avg_cpu_request,
  available_memory / avg_memory_request,
  ip_per_node,
  kubelet_max_pods
)

For 6500 Pods, test node density targets:

• 50 Pods/node → 130 nodes (safe, easy to manage)
• 65 Pods/node → 100 nodes (higher density, requires careful request setting)
• 100 Pods/node → 65 nodes (aggressive, only if workload very light or all batch)

Conservative choice for production: 50-60 Pods/node.

Step 3: Calculate Node Count, Add Overprovisioning Buffer

Base node count = total_pods / density

Add buffer for:

• Maintenance: 10% nodes reserved for upgrade rolling (surge nodes)
• Failure resilience: 5% extra (handle single node failure)
• Autoscaler headroom: 10% extra (avoid constant scaling thrashing)

Formula:

nodes_required = (total_pods / density) * 1.25

For 6500 Pods at 50 Pods/node:

• Base: 6500 / 50 = 130 nodes
• With buffer: 130 * 1.25 = ~163 nodes

But actual cluster: provision 200-250 nodes untuk 1000-node scale-out capacity later. This way, can grow to 5000-6000 Pods without node capacity issue.

Step 4: Network Planning

IP per node = secondary CIDR allocation. GKE default /24 per node (256 IPs).

For 1000 nodes:

• 1000 * 256 = 256K IPs needed
• VPC secondary CIDR must accommodate
• Google Cloud VPC allows multiple secondary CIDR ranges, each up to /9 (massive)
• Default: allocate /17 secondary CIDR for Pods (32K IPs), auto-expands as needed

But manual planning better: allocate /16 secondary CIDR (65K IPs) initially, covers up to 256 nodes comfortably, can expand via expansion request.

Resource Request Best Practices

CPU Request Setting

Problem: If Pod CPU request too low (e.g., 10m), scheduler thinks node can hold 400 Pods @ 4 CPU node. But at runtime, Pods consume 200m → node overloaded → evictions, context switch thrashing.

Better: Request reflects actual consumed CPU at peak, not minimum.

Guideline:

• Latency-sensitive (web services): request = p95 peak consumption
• Batch: request = average consumption (bursty ok, throttled at CGROUP limit)
• Burstable (logs aggregation): request = conservative, limit = actual peak

Memory Request Setting

Memory bottleneck critical ở large clusters. OOMKilled Pod = Kubelet kills + schedule eviction.

Guideline:

• Always set request ≠ limit
• Request = expected usage + 20% headroom
• Limit = absolute max (for burst tolerance)
• Too tight limits → frequent OOMKills, app instability

Example:

resources:
  requests:
    memory: "600Mi"   # expect ~500Mi, +20% headroom
  limits:
    memory: "1Gi"     # can burst, but hard cap

Bin Packing Strategy

Scheduler packing behavior affects node utilization, therefore affects cost (fewer nodes needed) vs resilience (finer failure domain).

Default (Balanced) Strategy

Scheduler spreads Pods across nodes (SelectorSpreadPriority, TaintToleration). Rationale: resilience, each node has Pods from multiple Deployments.

Pros: Fault-tolerant (single node failure doesn't take down service) Cons: Underutilized nodes (each node might have Pods from 10 Deployments, none fully packed)

Aggressive Bin-Packing

Use topology spread with maxSkew=1 + dnsPolicy=None (no affinity spreading):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: batch-worker
spec:
  replicas: 1000
  template:
    spec:
      affinity:
        podAntiAffinity: null  # no spreading
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
      terminationGracePeriodSeconds: 30

Behavior: Scheduler packs Pods densely (full one node before moving to next).

Pros: Fewer nodes, lower cost Cons: Single node failure impacts more Pods, longer recovery

Best for: Stateless batch workloads (can handle transient failure via retry).

References

• GKE Node Pools
• GKE Node Configuration
• Kubernetes Pod Resource Requests and Limits
• GKE Best Practices for Scalability