Cost Optimization — Spot VMs, Preemption Handling, Billing Models

Tại sao cost optimization quan trọng cho AI/ML

GPU cost dominates:

Training 70B model, 1 epoch (1 million steps, 1 hour):

On-demand pricing:
  8 × H100 GPU @ $3.06/hour = $24.48/hour
  1 hour training = $24.48
  
Spot pricing:
  8 × H100 GPU @ $0.91/hour = $7.28/hour
  1 hour training = $7.28
  
Savings: $17.20 per job (70% discount)

Over 1 year (365 training jobs):
  On-demand: $24.48 × 365 = $8,935
  Spot: $7.28 × 365 = $2,657
  Annual savings: $6,278 (71% reduction)

This is why Spot VMs are critical for cost-sensitive ML teams.

---

Spot VMs: pricing and characteristics

Pricing model

Spot VMs are "spare capacity" at reduced prices:

Pricing by region/machine type:
  n1-highmem-8 on-demand: $0.297/hour
  n1-highmem-8 spot:      $0.089/hour (70% discount)
  
  a2-highgpu-1g on-demand: $1.212/hour per GPU × 8 = $9.696/hour (VM only)
  + H100 GPU on-demand:    $3.06/hour × 8 = $24.48/hour
  Total: $34.18/hour
  
  With Spot VMs + Spot GPUs:
  a2-highgpu-1g spot:     $0.036/hour × 8 = $0.288/hour
  + H100 GPU spot:        $0.915/hour × 8 = $7.32/hour
  Total: $7.61/hour
  
Discount: 78% (from $34.18 to $7.61)

Preemption window

Spot VMs can be preempted at any time:

Preemption rate (historical):
  p50: runs for >1000 hours
  p10: runs for ~10 hours
  p1: runs for ~1 hour
  
Implication:
  Short jobs (< 10 min): low preemption risk
  Long jobs (> 10 hours): ~10% preemption probability
  Very long jobs (> 100 hours): ~50% preemption probability

Graceful shutdown

When preemption signal sent:

t=0: GCP sends ACPI shutdown signal
t=0-20s: OS receives signal, kubelet starts Pod eviction
         Container receives SIGTERM
         Application has ~30s to save state
t=20-30s: Kubelet force-kills container
t=30s: VM stops

Application must:
  1. Catch SIGTERM
  2. Save checkpoint (model state, training step)
  3. Exit gracefully

---

Cost models for AI/ML workloads

Model 1: Batch training (deadline-flexible)

Example: Train model for tomorrow's release

On-demand approach:
  8 GPUs × 10 hours = 80 GPU-hours = $240
  Complete by end of day

Spot approach:
  Expected preemptions: 1-2 over 10 hours
  Job 1: runs 2 hours, preempted → checkpoint saved
  Job 2: resumes from checkpoint, runs 3 hours, preempted
  Job 3: resumes, runs 5 hours, completes
  Total: ~10 hours wall time (or could be 20 hours if unlucky)
  
  Cost: 10 GPU-hours @ $0.91/hour = $9.10 (40x cheaper!)
  
  Trade-off:
    On-demand: predictable, fast, expensive
    Spot: variable latency, cheap, requires checkpointing

Recommendation: Use Spot for batch jobs with checkpointing.

Model 2: Serving (deadline-critical)

Example: Production LLM endpoint, 100ms SLA

On-demand approach:
  4 GPUs × 730 hours/month = 2920 GPU-hours
  Cost: 2920 × $3.06 = $8,943/month
  Availability: 99.99% (SLA guaranteed)

Spot approach:
  Preemption = SLA violation (Pod evicted, request fails)
  Cannot use Spot directly

Hybrid approach:
  2 Spot GPU + 2 on-demand GPU
  Spot (cheap baseline): 0-2ms latency
  On-demand (failover): guaranteed availability
  
  Cost: (1460 × $0.915 + 1460 × $3.06) = $5,900/month
  Availability: 99.95% (Spot preemption rare)
  Savings: $3,043/month (34% reduction)

Recommendation: Use hybrid for serving (Spot + on-demand failover).

Model 3: Hyperparameter tuning (embarrassingly parallel)

Example: Train 100 models (7B scale) in parallel

On-demand approach:
  100 × 1 GPU × 2 hours = 200 GPU-hours = $612
  
Spot approach:
  100 jobs, expected 10 preemptions
  Each preemption re-runs from scratch (small model, fast)
  
  Total GPU-hours: 200 + (10 × 2) = 220 GPU-hours
  Cost: 220 × $0.91 = $200
  
  Overhead: 10% (re-runs), but savings 67%
  
  Recommendation: Always use Spot for tuning

---

Handling preemption gracefully

Pod disruption budgets (PDB)

Without PDB:

Cluster gets preemption signal for 4 Pods
Kubelet evicts all 4 Pods immediately
Training job (needs 4 Pods) fails instantly

With PDB:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: training-job-pdb
spec:
  minAvailable: 3  # At least 3 Pods must stay running
  selector:
    matchLabels:
      job: training
  unhealthyPodEvictionPolicy: IfHealthyBudget

Kubelet behavior:

Preemption signal arrives
Kubelet checks: minAvailable=3, current Pods=4
Can evict: 4-3=1 Pod
Evicts 1 Pod, respects others
Training job survives (continues with 3 Pods)

Checkpoint/resume for training

import torch

# Training loop with checkpointing
best_loss = float('inf')
epoch_start = 0
step_start = 0

# Load checkpoint if exists
if os.path.exists('/checkpoints/latest.pt'):
    checkpoint = torch.load('/checkpoints/latest.pt')
    epoch_start = checkpoint['epoch']
    step_start = checkpoint['step']
    model.load_state_dict(checkpoint['model'])
    optimizer.load_state_dict(checkpoint['optimizer'])

# Install SIGTERM handler (preemption signal)
def handle_preemption(signum, frame):
    print("Preemption signal received, saving checkpoint...")
    torch.save({
        'epoch': epoch,
        'step': step,
        'model': model.state_dict(),
        'optimizer': optimizer.state_dict(),
        'loss': loss,
    }, '/checkpoints/latest.pt')
    exit(0)

signal.signal(signal.SIGTERM, handle_preemption)

# Training loop
for epoch in range(epoch_start, max_epochs):
    for step, batch in enumerate(dataloader, start=step_start):
        logits = model(batch)
        loss = compute_loss(logits)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        # Periodic checkpoint
        if step % 1000 == 0:
            torch.save({...}, f'/checkpoints/step-{step}.pt')

Pod restart policy

apiVersion: batch/v1
kind: Job
metadata:
  name: spot-training-job
spec:
  backoffLimit: 5  # Retry up to 5 times if Pod fails
  
  template:
    spec:
      restartPolicy: Never  # Don't restart Pod, let Job controller handle it
      
      tolerations:
      - key: cloud.google.com/gke-preemptible
        operator: Equal
        value: "true"
        effect: NoExecute
      
      containers:
      - name: trainer
        image: training:v1
        volumeMounts:
        - name: checkpoints
          mountPath: /checkpoints
      
      volumes:
      - name: checkpoints
        persistentVolumeClaim:
          claimName: training-checkpoints

Behavior:

Pod runs, preempted after 2 hours
Job controller: "Pod failed, retries < 5"
Recreates Pod (same image, volume mounts persisted)
New Pod: loads checkpoint from /checkpoints, resumes
Total: 10 hour job = 5 restarts × 2 hours avg

---

Hybrid strategies: on-demand + Spot

Strategy 1: Spot+OnDemand mix

# Create 2 node pools
gcloud container node-pools create spot-pool \
  --cluster=ml-cluster \
  --machine-type=a2-highgpu-8g \
  --spot \
  --num-nodes=8

gcloud container node-pools create on-demand-pool \
  --cluster=ml-cluster \
  --machine-type=a2-highgpu-8g \
  --num-nodes=2

Workload scheduling:

# Batch training job: prefer Spot
apiVersion: batch/v1
kind: Job
metadata:
  name: training
spec:
  template:
    spec:
      # Prefers Spot, falls back to on-demand if needed
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - key: cloud.google.com/gke-preemptible
                operator: In
                values: ["true"]
      
      containers:
      - name: trainer
        image: training:v1
        resources:
          limits:
            nvidia.com/gpu: 8

---
# Production serving: require on-demand
apiVersion: apps/v1
kind: Deployment
metadata:
  name: serving
spec:
  template:
    spec:
      # Requires on-demand (explicit rejection of Spot)
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: cloud.google.com/gke-preemptible
                operator: NotIn
                values: ["true"]
      
      containers:
      - name: server
        image: serving:v1
        resources:
          limits:
            nvidia.com/gpu: 4

Cost impact:

Cluster: 8 Spot (cheap) + 2 on-demand (expensive)

Scenario 1: Batch jobs only
  Use 8 Spot nodes → cost/GPU = $0.91/hour
  
Scenario 2: Batch + Serving mixed
  Batch: Spot nodes (cheap)
  Serving: on-demand nodes (guaranteed)
  Weighted cost: (8×0.91 + 2×3.06)/10 = $1.49/hour/GPU
  vs all on-demand: $3.06/hour/GPU → savings 51%

Strategy 2: Multi-cloud: Spot + Reserved

GCP Spot pricing: $0.91/hour (up to 91% discount)
Azure Spot pricing: $0.86/hour (up to 92% discount)

Multi-cloud cluster:
  Batch jobs: use cheapest Spot available
  If GCP preempted, auto-failover to Azure Spot
  Cost: average $0.88/hour (best of both)

---

Cost monitoring and optimization

Monitoring metrics

# Prometheus metrics for cost tracking
gpu_cost_per_job_total{job="training", status="completed"} 240.5
gpu_utilization_percent{node="gpu-0"} 45
preemption_rate_percent{pool="spot"} 2.1
cost_per_tflops{workload="inference"} 0.002

Optimization checklist

✓ Are you using Spot for batch jobs? (70-80% savings)
✓ Have you implemented checkpoint/resume? (enable Spot adoption)
✓ Is your GPU utilization >60%? (if not, over-provisioned)
✓ Are you using pod disruption budgets? (survive preemptions)
✓ Have you reserved capacity for predictable load? (commitment discount 30%)
✓ Are you time-slicing underutilized GPUs? (reduce over-provisioning)

---

Preemption handling failures

Failure 1: Lost checkpoint

Pod saves checkpoint every 1000 steps
Preemption happens at step 999
Restart loads last checkpoint (step 0)
Lost work: 999 steps × 2 minutes = 33 hours wasted

Fix: checkpoint every 100 steps (10x more frequent)
Trade-off: I/O overhead +5%, but prevents loss

Failure 2: Stale checkpoint

Pod saved checkpoint to /checkpoints (local node SSD)
Pod preempted, restarts on different node
New pod: /checkpoints is empty (local SSD, node-specific)

Fix: save checkpoint to persistent volume (Hyperdisk, GCS)
Cost: extra network I/O, but checkpoint survives node failure

Failure 3: Retries exhaust backoff limit

Job: backoffLimit: 5 (max 5 retries)
Spot pool: preemption rate 20% (every 5 runs ~1 preemption)

Job chain:
  Attempt 1: fails at 8 hours (preempted)
  Attempt 2: fails at 8 hours (preempted)
  Attempt 3: fails at 8 hours (preempted)
  Attempt 4: fails at 8 hours (preempted)
  Attempt 5: fails at 8 hours (preempted)
  Attempt 6: backoffLimit exceeded, job marked FAILED
  
Total time wasted: 40 hours of compute
Never completes

Fix: increase backoffLimit to 10-20, or increase Spot pool capacity
     (reduce preemption probability)

---

Mental model: When to use Spot vs on-demand

Decision tree:

Does workload tolerate preemption?
├─ YES (fault-tolerant, checkpointing)
│  ├─ Batch-like (deadline not tight)  → Use Spot (70% savings)
│  └─ Latency-sensitive (deadline tight) → Spot+OnDemand hybrid (50% savings)
└─ NO (cannot restart)
   └─ Use on-demand (guaranteed availability)

Examples:

✓ Spot suitable:
  - Model training (checkpointing, restart from epoch)
  - Batch inference (can reprocess)
  - Data processing (idempotent jobs)

✗ Spot unsuitable:
  - Real-time serving (cannot be interrupted)
  - Stateful databases (loss = data corruption)
  - Irreversible operations (e.g., fund transfer)

---

References

• GKE Spot VMs
• GKE Preemptible VMs
• GCP Spot VM Pricing
• Cost Optimization Best Practices
• Pod Disruption Budgets
• Compute Commitments (Reserved Capacity)