Spot VMs: Preemption Mechanics, Pricing, Workload Fit

Spot VMs là compute excess capacity mà Google bán rẻ — lên đến 91% discount so với on-demand. Nhưng đó là excess capacity, meaning Google có thể take it back anytime (preemption). Hiểu trade-off này là hiểu khi nào Spot VMs phù hợp, khi nào thì không.

Preemption Mechanics: Cách Spot VMs Bị Interrupt

Preemption Process

Khi Google cần tài nguyên back, process:

Signal phase (T):
- Metadata server updated với preemption signal
- Default: immediate (không có notice)
- Configurable: up to 30-second notice
Shutdown phase (T+notice):
- ACPI soft power-off signal sent
- Shutdown period: up to 30 seconds, best-effort
- Workload có cơ hội graceful shutdown
Final state (T+notice+shutdown):
- VM either stopped (default) hoặc deleted (configurable)
- Data on local SSD lost if deleted

Preemption Frequency & Regional Variance

Preemption frequency khác nhau per region và thời gian:

Region (typical preemption frequency):
  us-central1:        Low (< 5% per day)
  us-east1:           Low (< 5% per day)
  asia-southeast1:    Medium (10–20% per day)
  asia-south1:        High (30–50% per day)
  
Same region, different times:
  Peak hours (8 AM–6 PM):     High preemption (more demand)
  Off-peak (midnight–6 AM):   Low preemption (less demand)

Preemption frequency published per region on GCP pricing page — critical input for workload design.

No Billing for Short-Lived Interruptions

Key advantage: Nếu VM preempted < 1 phút, bạn không bị tính phí. Điều này encourage aggressive placement strategies.

Pricing Model: Spot VM Cost Structure

Spot Price Variability

Spot VM giá không fixed — nó thay đổi:

Pricing update: up to once per day (typically)

Price range: 60–91% off on-demand

Example (n2-standard-4, us-central1):
  On-demand:   $0.19/giờ
  Spot (low):  $0.02/giờ (89% discount)
  Spot (high): $0.06/giờ (68% discount)
  
Price changes tùy theo demand, supply, time of day

Price information available via:

bash

gcloud compute machine-types describe n2-standard-4 \
  --zone us-central1-a \
  --format="value(spotPrice)"

Cost Components Not Discounted

Spot pricing apply chỉ cho vCPU + memory. Không cover:

OS licenses (Windows, RHEL) — tính full price
Persistent disk — tính standard on-demand
GPUs/TPUs — tính Spot price (if available)
Network egress — tính full price

Ví dụ:

Windows n2-standard-4 Spot VM:
  vCPU/memory (Spot, 80% discount): $0.04/giờ
  Windows license (not discounted): $0.32/giờ
  Total: $0.36/giờ (vs $0.52 on-demand)
  Effective discount: 30% (not 80%!)

Workload Suitability: When Spot Makes Sense

Ideal Spot Workloads

Batch processing: Jobs có retry logic, checkpoint support
Data analysis: Spark/BigQuery jobs tolerant of task failure
ML training: Resumable from checkpoints
Testing / CI: Non-critical test runs
Background jobs: Cron tasks tolerant of delay
Batch rendering: Media processing, video encoding

Unsuitable for Spot VMs

Databases / stateful services: Preemption = data loss
Web frontends: Preemption = customer visible outage
Real-time systems: Latency-sensitive applications
Single points of failure: No redundancy to absorb preemption
Long-running interactive jobs: User waiting for result

Deployment Patterns: Maximizing Spot VM Uptime

Pattern 1: Managed Instance Groups with Auto-Healing

MIG configuration:
  - Spot VM instances (config)
  - Auto-healing enabled
  - Replacement policy: on-demand if Spot unavailable
  
Result: If Spot preempted, MIG auto-creates replacement
        (might be Spot or on-demand depending on capacity)

Pattern 2: Spot Fleet with Diversification

Diversify across:
  - Multiple regions (us-central1, us-east1, us-west1)
  - Multiple machine types (n2-standard-4, n2-standard-8, c2-standard-4)
  
Preemption in one region/type doesn't affect others

Pattern 3: Graceful Shutdown Handling

Configure shutdown script:
  - Save state to Cloud Storage
  - Notify load balancer (remove from backend)
  - Close DB connections cleanly
  - Upload logs
  
gcloud compute instances create spot-vm \
  --provisioning-model=SPOT \
  --metadata shutdown-script-url=gs://my-bucket/shutdown.sh

Pattern 4: Spot + On-Demand Mix

Target: 100 instances
  - 80 Spot instances (cheap)
  - 20 on-demand instances (guaranteed)

If Spot preemption happens:
  - On-demand handles minimum SLA
  - Lost Spot gradually replaced (next day, off-peak)

GKE Integration: Spot VMs in Kubernetes

GKE Spot VM Node Pools

gcloud container node-pools create spot-pool \
  --cluster=my-cluster \
  --machine-type=n2-standard-4 \
  --enable-gke-spot

GKE handling:

Pods on Spot nodes get graceful eviction (30s deadline)
Pod disruption budgets allow controlled drain
Workload identity continues working during preemption
DaemonSets (logging, monitoring) get re-created on replacement nodes

Pod Disruption Budgets

yaml

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 3  # Always keep 3 replicas
  selector:
    matchLabels:
      app: my-app

With PDB:

GKE drain Spot nodes gracefully
Pods evicted orderly, not abruptly
New pods scheduled on other nodes (Spot or on-demand)

Cost Comparison: Spot vs On-Demand vs CUD

100x n2-standard-4 running 730 hours/month × 12 months

Scenario 1: All on-demand
  Cost: 100 × 730 × 12 × $0.19 = $166,320/year

Scenario 2: All Spot (80% discount avg)
  Cost: 100 × 730 × 12 × $0.19 × 20% = $33,264/year
  Savings: $133,056 (80%)
  Caveat: Preemption, no SLA

Scenario 3: 80 Spot + 20 on-demand (multi-tier)
  Spot cost: 80 × 730 × 12 × $0.19 × 20% = $26,611/year
  On-demand: 20 × 730 × 12 × $0.19 = $33,264/year
  Total: $59,875/year
  Savings: $106,445 (64%), better SLA than pure Spot

Scenario 4: All 1-year CUD (55% discount)
  Cost: 100 × 730 × 12 × $0.19 × 45% = $74,844/year
  Savings: $91,476 (55%), guaranteed uptime

Takeaway: Spot VM amazing savings, but CUD safer for predictable workload. Hybrid approach best.

Anti-Patterns & Failure Modes

Anti-Pattern 1: Spot VM for Stateful Workload

Mistake: Using Spot for database, cache, or session store.

Issue: Preemption causes data loss, corruption, or downtime.

Fix: Use on-demand for stateful, Spot for stateless/batch.

Anti-Pattern 2: Ignoring Regional Preemption Variance

Mistake: Deploying latency-critical app on asia-south1 Spot (50% daily preemption).

Issue: High churn = poor UX.

Fix: Use preemption frequency data to select region. Query Spot pricing history to understand variance.

Anti-Pattern 3: No Graceful Shutdown

Mistake: Spot VM with no shutdown script, just SIGKILL.

Issue: In-flight requests lost, DB connections unclosed, logs unsaved.

Fix: Implement shutdown script, respect 30s grace period, use preemption signal for cleanup.

Monitoring Spot VM Health

Track Preemption Events

gcloud logging read \
  'resource.type="gce_instance" AND \
   protoPayload.reason="compute.instances.preempted"' \
  --limit 100

GKE Node Preemption Metrics

# In Prometheus/Cloud Monitoring:
kube_node_labels{label_cloud_google_com_gke_provisioning="SPOT"} = 1

# Count preemptions per day
rate(container_node_preemption_total[24h])

Cost Alerts

sql

-- Alert if actual Spot cost > expected (price spike?)
SELECT
  DATE_TRUNC(usage_start_time, DAY) as day,
  SUM(cost) as spot_cost,
  SUM(cost) / COUNT(*) as cost_per_instance_hour
FROM `project.billing_dataset.gcp_billing_export_v1`
WHERE labels.gke_io_provisioning = "spot"
GROUP BY day
HAVING spot_cost > (SELECT AVG(daily_cost) FROM ... WHERE date >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)) * 1.5

Spot VMs: Preemption Mechanics, Pricing, Workload Fit ​

Preemption Mechanics: Cách Spot VMs Bị Interrupt ​

Preemption Process ​

Preemption Frequency & Regional Variance ​

No Billing for Short-Lived Interruptions ​

Pricing Model: Spot VM Cost Structure ​

Spot Price Variability ​

Cost Components Not Discounted ​

Workload Suitability: When Spot Makes Sense ​

Ideal Spot Workloads ​

Unsuitable for Spot VMs ​

Deployment Patterns: Maximizing Spot VM Uptime ​

Pattern 1: Managed Instance Groups with Auto-Healing ​

Pattern 2: Spot Fleet with Diversification ​

Pattern 3: Graceful Shutdown Handling ​

Pattern 4: Spot + On-Demand Mix ​

GKE Integration: Spot VMs in Kubernetes ​

GKE Spot VM Node Pools ​

Pod Disruption Budgets ​

Cost Comparison: Spot vs On-Demand vs CUD ​

Anti-Patterns & Failure Modes ​

Anti-Pattern 1: Spot VM for Stateful Workload ​

Anti-Pattern 2: Ignoring Regional Preemption Variance ​

Anti-Pattern 3: No Graceful Shutdown ​

Monitoring Spot VM Health ​

Track Preemption Events ​

GKE Node Preemption Metrics ​

Cost Alerts ​

References ​