My Awesome Project

Appearance

Node Pool Strategy: Single vs Multiple Pools ​

Core Trade-off ​

Single pool (800 nodes, uniform machine type):

  • Simpler: no scheduling complexity
  • Better bin-packing: scheduler sees all 800 nodes
  • Faster upgrade: single strategy applies to all

Multiple pools (200 latency + 600 batch):

  • Flexible: workload isolation, independent scaling
  • Pod fragmentation: new workload might not fit any pool
  • Higher overhead: minimum nodes per pool even if unused

At 1000+ nodes, choice impacts operational burden significantly.

Pattern Analysis ​

Pattern 1: Single Large Pool (800-1000 nodes) ​

Topology:

cluster
└── default-pool (800 nodes, all n2-standard-8)

Scheduling behavior:

  • All Pods compete for same 800 nodes
  • Scheduler sees full node list (good for packing)
  • Affinity/anti-affinity rules simple (no pool selector needed)

Pros:

  • Maximum flexibility: any Pod can land on any node
  • Best bin-packing: scheduler optimizes across full 800 nodes
  • Simple upgrade: one upgrade procedure

Cons:

  • No workload isolation: batch job spike starves latency app (no reservation)
  • All-or-nothing scaling: scale down affects all workloads equally
  • Limited hardware diversity: can't run GPU jobs on CPU nodes (different machine types)

Best for:

  • Homogeneous workload (web services only, or batch only)
  • Cost-optimized (no over-provisioning per workload type)

Worst for:

  • Mixed workload (latency-critical + batch, GPU + CPU)

Pattern 2: Multiple Specialized Pools ​

Topology:

cluster
├── latency-sensitive-pool (150 nodes, n2-standard-16)
├── batch-pool (600 nodes, n2-standard-4)
└── gpu-pool (50 nodes, n1-standard-8 + GPU)

Scheduling behavior:

  • Deployment with GPU requirement → can't land on batch-pool (no GPU)
  • Latency app → should prefer latency-sensitive-pool (but can fallback if nodeSelectorTerms)
  • Bin-packing: each pool optimized independently (suboptimal globally)

Pros:

  • Workload isolation: batch can scale without affecting latency apps
  • Hardware specialization: different pools for different needs
  • Independent scaling: scale batch down (save cost), keep latency warm
  • Blast radius: one pool failure doesn't impact others

Cons:

  • Fragmentation: Pod with 8 CPU request in batch-pool (4-CPU machines) → can't fit
  • Over-provisioning: minimum node per pool (e.g., 1 GPU node minimum, even if unused)
  • Scheduling complexity: Pod spec must match pool (nodeSelector, affinity)

Best for:

  • Diverse workload (web + batch + ML + data pipeline)
  • Predictable usage patterns (know workload distribution)

Worst for:

  • Workload with unpredictable type distribution
  • High volume of custom Pod resource requirements

Topology:

cluster
├── general-pool (700 nodes, n2-standard-4)
├── compute-optimized-pool (100 nodes, c2-standard-4)
├── gpu-pool (50 nodes, GPU)
└── burstable-pool (100 nodes, preemptible)

Strategy:

  • General pool: 85% of workload (flexible, bin-packed)
  • Compute pool: IO-heavy workload that needs high-CPU (15%)
  • GPU pool: ML workload requiring GPU (specialized)
  • Burstable pool: non-critical batch (cost-saving)

Rationale:

  • Majority in general pool = good bin-packing
  • Specialized pools for outliers
  • Minimizes over-provisioning

Blast Radius Analysis ​

Blast radius: "If this pool fails, how many Pods affected?"

Single Pool Example ​

Scenario: network misconfiguration on 100 nodes in cluster
Impact: 100/800 = 12.5% of Pods evicted
Recovery: Pods re-schedule to remaining 700 nodes
Cost: 2-5 minute disruption

Multiple Pools Example ​

Scenario: network misconfiguration on latency-sensitive-pool (150 nodes)
Impact: 150 nodes lost
  - If latency workload ALL in this pool: 100% disruption (SLA breach)
  - If distributed across pools: 150/800 = 18.75% latency Pods
Cost: High (latency SLA miss) if workload concentrates in one pool

Learning: Multiple pools increase blast radius for concentrated workloads. Must distribute replicas across pools (PodTopologySpread).

Upgrade Strategy Per Pool ​

Single Pool: Rolling Upgrade ​

1. Cordon 10% nodes (80 nodes)
2. Drain Pods
3. Upgrade nodes
4. Uncordon
5. Repeat next 10%

Time: 10 waves × 15 minutes each = 150 minutes total (2.5 hours).

Multiple Pools: Parallel Upgrade ​

Parallel:
  - general-pool: 10% at a time (upgrade 70 nodes × 10 waves = 150 min)
  - gpu-pool: can upgrade faster (fewer nodes, less coordination)
  - burstable-pool: cheaply (non-critical, can disrupt)

Time: 150 minutes (same as single pool) but less operational overhead (batch pods can be disrupted freely).

Sizing Pools at 1000+ Nodes ​

Determine Workload Distribution ​

Input: Expected Pods by type over time.

Latency-sensitive (web APIs): 30% of Pods, peak 1500 Pods
Batch (job workers): 50%, peak 2500 Pods
ML (training jobs): 10%, peak 500 Pods
Data pipeline: 10%, peak 500 Pods

Total: 5000 Pods expected.

Allocate Pools ​

Conservative (avoid fragmentation):

general-pool: 70% of total capacity = 700 nodes (2.5K Pods @ 50/node)
latency-specific: 20% = 200 nodes (1K Pods @ 50/node)
ML/GPU: 10% = 100 nodes (dedicated)

Result: 1000 total, minimal fragmentation.

Cross-check with Scaling ​

  • If batch workload scales 10x (unusual), can it fit in general-pool?

    • 700 general + 200 latency = 900 total (can fit batch in latency during surge? No, different machine type.)
    • Need overflow pool or accept queue

  • Conservative approach: size general-pool to handle max expected + 30% margin

Real-World Scenario: Pool Saturation During Spike ​

Case: 1000-node cluster with strategy:

  • latency-pool: 200 nodes (web)
  • batch-pool: 600 nodes (job workers)
  • gpu-pool: 50 nodes
  • general: 150 nodes

Event: Batch job framework submits 10x normal load (5000 → 50K jobs).

Timeline:

  1. Job controller creates Pod for each job
  2. Scheduler tries to place Pods
  3. batch-pool saturated (600 nodes full)
  4. Remaining 49.4K Pods can't fit in latency-pool (wrong machine size)
  5. Pods stack in Pending

Resolution:

  • Auto-scale batch-pool (if using cluster autoscaler with node pool targets)
  • Or manually: gcloud container node-pools update batch-pool --num-nodes=800
  • Time to add nodes: 5-10 minutes
  • Pods still Pending for that duration

Better: Hybrid strategy with general-pool as overflow.

References ​