GPU Node Pool Architecture & Device Plugin

Tại sao quan trọng

GPU trong Kubernetes không phải là "resource như CPU/memory". Chúng yêu cầu:

Device driver cài sẵn trên node
Device plugin expose GPU resources cho Kubernetes
Taints/tolerations để chỉ GPU workloads được schedule
Custom scheduling logic để tránh fragmenting GPU memory

Nếu hiểu sai mental model, bạn sẽ gặp vấn đề: Pod pending dù có GPU available, driver mismatch, workloads không được isolate properly.

Cơ chế GPU Node Pools trong GKE

Node Pool creation và GPU attachment

Khi bạn tạo node pool với GPU:

bash

gcloud container node-pools create gpu-pool \
  --cluster=my-cluster \
  --machine-type=n1-highmem-8 \
  --accelerator=type=nvidia-tesla-a100,count=2 \
  --num-nodes=3

GKE thực hiện các bước sau:

Validate quota: Kiểm tra bạn có đủ GPU quota cho region/zone (2 GPUs × 3 nodes = 6 GPUs)
Create VMs: Provision 3 VMs với 2 A100 GPUs mỗi VM
Install driver: Tùy theo cấu hình (mặc định: GKE tự động install NVIDIA driver)
Tag nodes: Thêm label cloud.google.com/gke-accelerator=nvidia-tesla-a100
Apply taints: Thêm taint nvidia.com/gpu=true:NoSchedule để chỉ GPU Pods được schedule

Taints and tolerations: Isolating GPU workloads

GKE tự động thêm taint:

yaml

taints:
- key: nvidia.com/gpu
  value: true
  effect: NoSchedule

Pod yêu cầu GPU phải tolerant taint này:

yaml

spec:
  tolerations:
  - key: nvidia.com/gpu
    operator: Equal
    value: "true"
    effect: NoSchedule
  containers:
  - name: training-job
    resources:
      limits:
        nvidia.com/gpu: 2  # Request 2 GPUs

Mental model: Taint là "bộ lọc cứng" — GPU node sẽ từ chối Pod không có tolerations, ngay cả nếu có GPU free.

NVIDIA Device Plugin: Resource exposure

Cơ chế hoạt động

NVIDIA device plugin là Daemonset chạy trên mỗi GPU node:

kubelet → device plugin grpc endpoint → enumerate GPUs
                                          ↓
          expose nvidia.com/gpu resource type
                                          ↓
          kubelet → resource available on node

Quá trình chi tiết:

Device discovery (startup): Plugin quét node và tìm tất cả physical GPUs
```
nvidia-smi → 2 GPUs (GPU 0: H100, GPU 1: H100)
```

Capacity export (kubelet sync):

yaml

Node status:
allocatable:
  nvidia.com/gpu: 2

Pod admission:

yaml

Pod requests nvidia.com/gpu: 1
kubelet → plugin: allocate GPU 0
plugin → kubelet: device allocated, volume mounts /dev/nvidia0

Container execution:

bash

# kubelet launches container with:
# - /dev/nvidia0, /dev/nvidia1, /dev/nvidiactl mounted
# - LD_LIBRARY_PATH=/usr/local/nvidia/lib64
# - GPU driver libraries + CUDA runtime available

Device plugin lifecycle

Khi node khởi động:

1. kubelet starts → loads default device plugins
2. nvidia-device-plugin daemonset pod starts
3. plugin discovers GPUs → exports capacity
4. kubelet calls plugin's ListAndWatch endpoint
5. On GPU failure: plugin updates capacity (e.g., 2 → 1 GPU)
6. kubelet reschedules pending Pods

Key constraint: Device plugin là stateless. Nó chỉ track physical availability, không track actual usage (VRAM, compute utilization). Kubernetes scheduler không biết Pod A dùng 80% VRAM của GPU — từ perspective của kubelet, "GPU được allocated" và đó là tất cả thông tin nó có.

GPU Driver Management: Automatic vs Manual

GKE-Managed Driver (Default, Recommended)

Khi tạo GPU node pool, GKE tự động:

Detect GPU type: Query Compute Engine metadata
Select driver version: Dựa trên GPU model + node OS (COS)
Download + install: From GKE bucket (network-local, fast)
Monitor + auto-repair: NVIDIA DCGM on node monitors driver health

Node startup sequence:
1. Boot COS image
2. GKE agent detects: GPU H100 present
3. Install NVIDIA driver (latest stable for H100 + COS)
4. Start DCGM daemon (monitoring + error detection)
5. Notify kubelet: Ready

Advantage: Zero operational burden. GKE patches driver automatically.

Limitation: Driver version is fixed per GPU type. Bạn không thể force driver 550 nếu GKE phân phối 545.

NVIDIA GPU Operator (User-Managed)

Nếu bạn cần control driver lifecycle (multi-cloud consistency, custom CUDA versions):

bash

gcloud container node-pools create gpu-pool \
  --cluster=my-cluster \
  --enable-ip-alias \
  --accelerator=type=nvidia-tesla-h100,count=8 \
  --labels=gke-no-default-nvidia-gpu-device-plugin=true  # Disable GKE plugin

Sau đó install NVIDIA GPU Operator qua Helm:

bash

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator-system \
  --create-namespace

GPU Operator quản lý:

NVIDIA driver installation + updates
NVIDIA device plugin (same as GKE's, but user-managed)
CUDA libraries
DCGM (monitoring)
NVIDIA MIG manager (if enabled)

Trade-off:

✓ Control driver version
✓ Consistent across clouds
✗ More operational overhead (you manage updates)
✗ Longer debugging chain if GPU issues (is it driver? operator? plugin?)

GPU Scheduling: Constraints and Fragmentation

Bin-packing behavior

Kubernetes scheduler không hiểu "GPU memory fragmentation". Nó chỉ track:

GPU count (nvidia.com/gpu: integer)

Scenario: Node có 2 H100 GPUs (80GB mỗi cái)

Pod A: requests nvidia.com/gpu: 1 → allocated GPU 0
Pod B: requests nvidia.com/gpu: 1 → allocated GPU 1

Now Pod C: requests nvidia.com/gpu: 1 → PENDING (no GPU)

Nhưng nếu Pod A chỉ dùng 30GB VRAM và Pod B dùng 20GB, vẫn có 110GB miễn phí tổng. Scheduler không thấy điều này. Đó là lý do cần MIG hoặc Time-Slicing (chương tiếp).

Zone pinning

GPU availability là zone-specific. Nếu bạn tạo GPU node pool chỉ ở zone us-central1-a:

Cluster có 3 zones: a, b, c
GPU nodes chỉ ở zone a

→ Tất cả GPU workloads được pinned vào zone a
→ Khi zone a resource pool saturated, pending Pod sẽ block (cluster autoscaler scale up nhưng chỉ ở zone a)

Mitigation:

Tạo GPU node pool ở tất cả zones (hoặc 3 node pools, mỗi cái 1 zone)
Hoặc sử dụng NodeAffinity để workload có thể move giữa zones

Device plugin failure modes

Scenario 1: Device plugin crash

GPU node running 4 Pods (each with GPU)
nvidia-device-plugin crashes (OOM, kernel panic, etc.)

kubelet notices device plugin dead → marks GPU capacity = 0
All 4 Pods still running (kubelet doesn't evict)
But no new GPU Pod can be scheduled on that node

Recovery: kubelet auto-restarts device plugin daemonset. Nếu DaemonSet crashed, manual intervention cần.

Scenario 2: Driver mismatch

Node A: Driver 550 + CUDA 12.0
Pod requests: cuda:11.8 (incompatible)

Container tries to load libcuda.so.11.8 → ENOENT
Pod crashes with OOM-like symptoms (vague error)

Prevention: Version-specific image tagging, nodeSelector ở CUDA version label (nếu track).

GPU Quota và Billing

Quota enforcement

Setiap region punya quota untuk GPU count:

project my-project:
  us-central1:
    nvidia-tesla-a100: 50 (limit)
                        8 (in-use)
                       42 (available)

Ketika membuat node pool:

bash

gcloud container node-pools create gpu-pool \
  --accelerator=type=nvidia-tesla-a100,count=8 \
  --num-nodes=10

# Total: 8 × 10 = 80 A100s
# But quota = 50 → CREATE FAILS

Quota request pattern: Most teams request in increments of 8, 16, 32 (matching popular batching).

Billing

GPU ditagih per-minute (seperti VM):

1 A100 GPU = $1.95/hour (us-central1, on-demand)

Node pool 10 nodes × 8 GPUs/node = 80 GPUs
Monthly cost: 80 × $1.95 × 730 hours = ~$114,120

Dengan 10% utilization (actual VRAM usage 8GB dari 80GB), cost-per-unit bisa meningkat 10x. Itulah mengapa resource efficiency (batching, model parallelism, Spot VMs) kritis.

GKE-specific GPU behaviors

Autopilot vs Standard modes

GKE Autopilot:

User hanya declare resource needs: nvidia.com/gpu: 4
GKE provision node automatically
GPU driver auto-installed + auto-patched
Pricing: per-Pod resource reservation (tidak per-node)

GKE Standard:

User create node pool + manage scaling
Full control over machine type, driver version, autoscaling
Pricing: per-node (GPU cost bundled with VM)

GPU node pool specifics in Autopilot: Pod CPU/memory request harus match GPU memory capacity roughly. Jika request 4 GPUs (320GB) tapi hanya 64GB CPU memory, kubelet reject (mismatch).

Graceful shutdown

When node drain (update, scale-down):

1. kubelet sends SIGTERM to GPU Pod
2. GPU code must checkpoint state (loss = restart training)
3. If no graceful shutdown hooks → Pod forcibly killed after 30s
4. New Pod schedule on another GPU node (potentially losing epochs)

For long-running training: implement PreStop hook to checkpoint state, atau pakai ProvisioningRequest (chapter 4) untuk gang-schedule resumable batches.

Mental Model Summary

GPU node pools dalam GKE:

Device plugin: stateless inventory + resource export mechanism, does NOT track actual utilization
Taints: hard filter ensuring non-GPU-tolerant workloads can't accidentally schedule
Driver: usually GKE-managed (safe, automatic), optionally user-managed (GPU Operator)
Fragmentation: scheduler bins by count, not by memory → underlies need for MIG/time-slicing
Quota: regional, not project-global → potential bottleneck at specific region
Graceful shutdown: workload must handle SIGTERM + checkpoint, else data loss

Production Anti-patterns

❌ Anti-pattern 1: No taint toleration

yaml

# ❌ WRONG: GPU Pod without toleration
spec:
  containers:
  - name: inference
    resources:
      limits:
        nvidia.com/gpu: 1
  # Missing: tolerations

GPU taint ngăn chặn Pod này → PENDING forever.

Fix:

yaml

tolerations:
- key: nvidia.com/gpu
  operator: Equal
  value: "true"
  effect: NoSchedule

❌ Anti-pattern 2: Ignoring fragmentation

Running 10 training Pods, mỗi cái request 1 GPU, trên cluster có 16 GPUs. Mỗi Pod dùng ~60GB VRAM → 6GB spare per GPU.

Pod ke-11 pending (no free GPU), nhưng tổng cluster có 96GB spare VRAM.

→ Solution: MIG/Time-Slicing (chapter 2), không scaling up more GPUs.

❌ Anti-pattern 3: Assuming driver version doesn't matter

dockerfile

FROM nvidia/cuda:11.8
COPY model.pth /app/
RUN python -m pip install torch==2.0

Node chạy CUDA 12.0 (GKE standard H100 default) → library mismatch → cryptic error.

Fix: Pin CUDA version trong image tag hay sử dụng base image matching node's driver. Atau track driver version ở node label + nodeSelector.

GPU Node Pool Architecture & Device Plugin ​

Tại sao quan trọng ​

Cơ chế GPU Node Pools trong GKE ​

Node Pool creation và GPU attachment ​

Taints and tolerations: Isolating GPU workloads ​

NVIDIA Device Plugin: Resource exposure ​

Cơ chế hoạt động ​

Device plugin lifecycle ​

GPU Driver Management: Automatic vs Manual ​

GKE-Managed Driver (Default, Recommended) ​

NVIDIA GPU Operator (User-Managed) ​

GPU Scheduling: Constraints and Fragmentation ​

Bin-packing behavior ​

Zone pinning ​

Device plugin failure modes ​

GPU Quota và Billing ​

Quota enforcement ​

Billing ​

GKE-specific GPU behaviors ​

Autopilot vs Standard modes ​

Graceful shutdown ​

Mental Model Summary ​

Production Anti-patterns ​

❌ Anti-pattern 1: No taint toleration ​

❌ Anti-pattern 2: Ignoring fragmentation ​

❌ Anti-pattern 3: Assuming driver version doesn't matter ​

References ​

GPU Node Pool Architecture & Device Plugin

Tại sao quan trọng

Cơ chế GPU Node Pools trong GKE

Node Pool creation và GPU attachment

Taints and tolerations: Isolating GPU workloads

NVIDIA Device Plugin: Resource exposure

Cơ chế hoạt động

Device plugin lifecycle

GPU Driver Management: Automatic vs Manual

GKE-Managed Driver (Default, Recommended)

NVIDIA GPU Operator (User-Managed)

GPU Scheduling: Constraints and Fragmentation

Bin-packing behavior

Zone pinning

Device plugin failure modes

GPU Quota và Billing

Quota enforcement

Billing

GKE-specific GPU behaviors

Autopilot vs Standard modes

Graceful shutdown

Mental Model Summary

Production Anti-patterns

❌ Anti-pattern 1: No taint toleration

❌ Anti-pattern 2: Ignoring fragmentation

❌ Anti-pattern 3: Assuming driver version doesn't matter

References