GPU Memory Strategies — MIG, Time-Slicing, Tradeoffs

Tại sao cần multiplexing

GPU cost là bottleneck lớn. Một A100 (80GB) chạy single workload là lãng phí:

Node: 8 × H100 GPUs (640GB VRAM)
Workload A: vLLM serving, uses 50GB VRAM, 30% compute
Workload B: batch inference, uses 40GB VRAM, busy 2 hours/day

→ Tổng 90GB VRAM, 640GB available
→ Cost per unit VRAM: 640GB ÷ $1000/month = $1.56/GB-month
→ Actual usage VRAM: 90GB ÷ $1000/month = $11.11/GB-month

Multiplexing giảm effective cost per workload, nhưng đặt ra trade-off: latency variance, isolation gaps, complexity tăng.

GKE cung cấp 3 chiến lược:

Multi-Instance GPU (MIG) — hardware partitioning
GPU Time-Slicing — software context-switching
NVIDIA Multi-Process Service (MPS) — kernel share (legacy)

Multi-Instance GPU (MIG): Hardware Partitioning

Cơ chế hoạt động

MIG chia single GPU thành tối đa 7 isolated partitions (slices). Mỗi slice có:

Dedicated compute cores
Dedicated memory bank
Independent error isolation (fault trong slice A không ảnh hưởng slice B)

Physical H100 GPU (80GB, 14080 SMs)
│
├─ MIG Slice 0 (1g.10gb): 1408 SMs, 10GB VRAM, GPU 0
├─ MIG Slice 1 (1g.10gb): 1408 SMs, 10GB VRAM, GPU 1
├─ MIG Slice 2 (1g.10gb): 1408 SMs, 10GB VRAM, GPU 2
├─ MIG Slice 3 (1g.10gb): 1408 SMs, 10GB VRAM, GPU 3
├─ MIG Slice 4 (1g.10gb): 1408 SMs, 10GB VRAM, GPU 4
├─ MIG Slice 5 (1g.10gb): 1408 SMs, 10GB VRAM, GPU 5
├─ MIG Slice 6 (1g.10gb): 1408 SMs, 10GB VRAM, GPU 6
└─ Spare capacity: ~10GB VRAM

MIG partition types

GKE/NVIDIA cung cấp preset partitions:

GPU             Total  1g.10gb  1g.20gb  2g.20gb  3g.40gb  4g.40gb  7g.80gb
────────────────────────────────────────────────────────────────────────────
H100 (80GB)       80     7x10    3x20+    2x20+    1x40     1x40     1x80
A100 (40GB)       40     3x10    1x20+    1x20+    -        -        -
H200 (141GB)     141    11x10    6x20+    -        3x40     -        1x141

Note: + means fractional. E.g., 1g.20gb on H100 leaves 60GB; can't fit another 1g.20gb (need 20GB), but can fit 1g.10gb (10GB).

Setup MIG trên GKE

Enable MIG mode (node-level):

bash

# Node pool creation dengan MIG enabled
gcloud container node-pools create mig-pool \
  --cluster=my-cluster \
  --machine-type=a2-highgpu-1g \
  --accelerator=type=nvidia-tesla-a100,count=1 \
  --enable-mig=true \
  --mig-config=1g.10gb:7  # 7 slices of 1g.10gb each

Kubernetes side: GKE plugin exposes each slice as separate "GPU":

yaml

Node status:
allocatable:
  nvidia.com/gpu: 7  # 7 slices, mỗi cái addressable như separate GPU

# Pod scheduling:
spec:
  containers:
  - name: app-a
    resources:
      limits:
        nvidia.com/gpu: 1  # Gets 1 slice (10GB)
  - name: app-b
    resources:
      limits:
        nvidia.com/gpu: 1  # Gets another slice (10GB)

Constraints và failure modes

❌ Memory enforcement gap: MIG provides hard memory limit — nếu Pod request 10GB nhưng cố dùng 11GB, kernel OOM-kill Pod.

Container: NVIDIA GPU driver enforces memory limit
→ Reads VRAM above limit: GPU memory error
→ Process killed by GPU kernel (not kubelet)

❌ CUDA Compute Capability mismatch: MIG slices của H100 (CC 9.0) khác A100 (CC 8.0). Code compiled cho A100 có thể fail/slow trên H100 slice.

❌ Fragmentation across nodes: Bạn request 2 slices nhưng 2 nodes khác nhau — inter-GPU communication thông qua network (1000x slower).

MIG vs Time-Slicing trade-off:

Feature	MIG	Time-Slicing
Memory isolation	✓ Hard limit	✗ No enforcement
Compute isolation	✓ Dedicated cores	✗ Shared cores
Latency predictability	✓ High	✗ Variable
Cost per unit	Higher (less utilization)	Lower (higher utilization)
Setup complexity	Medium	Low
Scheduling flexibility	Lower (fixed slices)	Higher (can request <1 GPU)

GPU Time-Slicing: Software Context-Switching

Cơ chế

GPU time-slicing sử dụng NVIDIA GPU instruction-level preemption feature:

Timeline:
[--- Pod A process ---] [--- Pod B process ---] [--- Pod A process ---]
       100ms                  100ms                   100ms
       (timeslice)           (timeslice)            (timeslice)

Kernel rapidly context-switch between processes
→ Process A registers saved, process B registers loaded
→ Fair CPU-like scheduling nhưng ở GPU level

Key difference from MPS: Time-slicing tất cả processes nhìn thấy full GPU VRAM, nhưng từ perspective của GPU hardware, chỉ 1 process chạy tại 1 lần.

Setup trên GKE

Node pool creation:

bash

gcloud container node-pools create timeshare-pool \
  --cluster=my-cluster \
  --accelerator=type=nvidia-tesla-h100,count=1 \
  --enable-gpu-time-sharing=true \
  --max-shared-clients-per-gpu=10  # Up to 10 containers share 1 GPU

Pod spec: Yêu cầu GPU như bình thường:

yaml

spec:
  containers:
  - name: batch-inference-1
    resources:
      limits:
        nvidia.com/gpu: 1  # Actually 0.1 GPU (1/10)
  - name: batch-inference-2
    resources:
      limits:
        nvidia.com/gpu: 1  # Actually 0.1 GPU (1/10)

Kubernetes kubelet accepts 0.1 fractions (with time-sharing enabled).

Constraints

❌ Memory enforcement: Không có. Kernel không enforce memory limit. Nếu process A cố allocate 60GB trên 80GB GPU:

GPU memory allocation: successful (no kernel check)
But when process B runs (context-switch):
→ GPU runs out of memory during B's execution
→ GPU memory pressure → unpredictable behavior

→ Mental model: Time-slicing assumes cooperative workloads — mỗi workload allocates bounded memory, không hành xử tham lam.

❌ Performance variance:

Container A: latency p50=100ms (when unshared)
Container A: latency p50=150ms, p99=400ms (when sharing with 9 others)

Context-switching overhead và cache pressure → 2-5x latency increase.

❌ One GPU per container: Kubernetes admission controller rejects multi-GPU time-slicing:

yaml

# ❌ REJECTED
resources:
  limits:
    nvidia.com/gpu: 2  # Multi-GPU + time-slicing = not supported

Reason: multi-GPU workloads typically communication-bound; time-slicing overhead becomes prohibitive.

❌ No kernel-level preemption guarantees: Meskipun instruction-level preemption ada, kernel operations (long-running kernels) dapat delay context-switch.

Process A: launch_kernel() → kernel runs 500ms (without preemption)
Process B: waiting 500ms (latency variance)

MPS adalah legacy approach (before MIG/time-slicing):

Single GPU kernel + shared memory allocator
→ Multiple processes share same kernel scheduler
→ Hyper-lightweight isolation (less overhead than time-slicing)

Disadvantages:

No memory isolation (single allocator, cooperative only)
Deprecated in favor of MIG + time-slicing
GKE discourages MPS (prefer time-slicing)

GKE support: Available via GPU Operator configuration, nhưng not recommended for new deployments.

Choosing multiplexing strategy

Decision tree

Does workload need predictable latency?
├─ YES → Need hard memory isolation?
│        ├─ YES → MIG (dedicated cores + memory)
│        └─ NO → Time-slicing + careful memory budgeting
└─ NO (batch, offline)
   └─ Time-slicing (lowest cost)

Workload patterns

Pattern 1: LLM serving (vLLM)

Few large models (1-2 per node)
Latency-sensitive
Each model pre-allocates large VRAM (40-60GB)

→ Recommendation: No multiplexing (1 GPU per model). If multiple smaller models, use MIG slices.

Pattern 2: Batch inference (many small models)

100s of inference requests, 10-100ms latency tolerance
Small model size (1-5GB each)
Burstiness (not continuous traffic)

→ Recommendation: Time-slicing (10-15 clients per GPU). Cooperative memory allocation.

Pattern 3: Training jobs (data parallelism)

GPU-to-GPU synchronization required (NCCL allreduce)
Bandwidth-sensitive (inter-GPU comm)
Long-running (hours to days)

→ Recommendation: No multiplexing. Dedicated GPU per training job. Use gang scheduling (chapter 4) to ensure all GPUs in training sync'ed together.

Pattern 4: Development/experimentation

Variable workload intensity
Cost-sensitive
Low-latency requirement: none

→ Recommendation: Time-slicing (up to 10 clients). Tolerates variance.

Combining MIG + Time-Slicing

GKE memungkinkan kombinasi:

bash

gcloud container node-pools create hybrid-pool \
  --accelerator=type=nvidia-tesla-h100,count=1 \
  --enable-mig=true \
  --mig-config=2g.20gb:1,1g.10gb:4 \  # 1 × 2g.20gb + 4 × 1g.10gb
  --enable-gpu-time-sharing=true \
  --max-shared-clients-per-gpu=3

Result:

1 × H100 GPU (80GB)
├─ MIG slice 1g.10gb #0
│  ├─ Time-slice tenant A
│  ├─ Time-slice tenant B
│  └─ Time-slice tenant C (up to 3 per partition)
├─ MIG slice 1g.10gb #1
│  ├─ Time-slice tenant D
│  ├─ Time-slice tenant E
│  └─ Time-slice tenant F
├─ MIG slice 2g.20gb #0
│  ├─ Time-slice tenant G
│  ├─ Time-slice tenant H
│  └─ Time-slice tenant I

Advantage: Granular resource allocation, better isolation (MIG) + higher utilization (time-slicing).

Disadvantage: Scheduling complexity tăng. Tenant G yêu cầu "large GPU" (20GB), cần explicit scheduling onto 2g.20gb slice, không thể fallback để 1g.10gb.

Monitoring and troubleshooting

Utilization tracking

Kubelet exposes metrics per GPU, nhưng not per container:

node_gpu_utilization: 60%
node_gpu_memory_used: 48GB / 80GB

Cannot correlate: "container A uses 25GB" (không phải kubelet job).

Monitoring solution: Sidecar container + NVIDIA DCGM (query /dev/nvidia-uvm stats):

yaml

spec:
  containers:
  - name: app
  - name: gpu-monitor
    image: nvidia/dcgm-exporter:latest
    volumeMounts:
    - name: device
      mountPath: /dev/nvidia0
  volumes:
  - name: device
    hostPath:
      path: /dev/nvidia0

Common issues

Issue 1: Pod pending, GPU available

Status: Pod pending
nvidia.com/gpu: 0/16 available (all used)
But: nvidia-smi shows 40% utilization

→ Cause: GPU memory fragmented, no contiguous 10GB for 1g.10gb MIG slice
→ Solution: Time-slicing (fractional GPU), hoặc manual pod eviction + defrag

Issue 2: OOM on time-slicing GPU

Container: request nvidia.com/gpu: 1
Runtime: allocates 70GB from shared GPU
Process B tries to allocate 20GB
→ GPU OOM (no memory enforcement)

→ Cause: Overlapping VRAM allocation (both processes think they can use 0-80GB)
→ Solution: Explicit memory limits in container (ENV, config), monitor at container level

Mental model synthesis

MIG vs Time-Slicing is not binary:

MIG: Hardware-isolated slices, predictable (latency, memory), lower utilization
Time-slicing: Software-multiplexed, flexible (fractional GPU), higher utilization, variance

Pilihan tergantung pada isolation requirement + workload burstiness:

Serving LLMs (unpredictable traffic, latency SLA) → MIG + no multiplexing
Batch jobs (offline, deadline-tolerant) → Time-slicing + high multiplexing

Production Anti-patterns

❌ Anti-pattern 1: Assuming time-slicing memory enforcement

yaml

# ❌ WRONG
spec:
  containers:
  - name: hog
    resources:
      limits:
        nvidia.com/gpu: 1
        memory: 64Gi  # CPU memory, NOT GPU memory
    # Allocates all 80GB GPU VRAM, no limits

→ Causes OOM for sibling containers.

Fix: Track GPU memory externally (DCGM monitoring), implement application-level memory budget.

❌ Anti-pattern 2: Mixing training + inference on time-slicing

yaml

# Node: time-slicing enabled, max 4 clients per GPU
Pods:
- training-job (needs 50GB, communication latency-sensitive)
- inference-server (needs 20GB, latency SLA 100ms)

→ Context-switching between training kernel + inference kernel
→ Training blocks inference (long kernel), SLA violated

Fix: Dedicated GPU for training. Inference on separate time-sliced pool.

❌ Anti-pattern 3: Ignoring MIG slice availability

bash

gcloud container node-pools create mig-pool \
  --mig-config=2g.20gb:1  # Only 1 slice of 20GB

# Then Pod requests:
#  - 3 × (2g.20gb) slices
# → Pod pending (only 1 available)

→ Needs horizontal scaling (add more nodes).

Fix: Plan MIG layout matching expected workload distribution. Or use time-slicing for variable-size workloads.

GPU Memory Strategies — MIG, Time-Slicing, Tradeoffs ​

Tại sao cần multiplexing ​

Multi-Instance GPU (MIG): Hardware Partitioning ​

Cơ chế hoạt động ​

MIG partition types ​

Setup MIG trên GKE ​

Constraints và failure modes ​

GPU Time-Slicing: Software Context-Switching ​

Cơ chế ​

Setup trên GKE ​

Constraints ​

NVIDIA Multi-Process Service (MPS): Process Kernel Sharing ​

Choosing multiplexing strategy ​

Decision tree ​

Workload patterns ​

Combining MIG + Time-Slicing ​

Monitoring and troubleshooting ​

Utilization tracking ​

Common issues ​

Mental model synthesis ​

Production Anti-patterns ​

❌ Anti-pattern 1: Assuming time-slicing memory enforcement ​

❌ Anti-pattern 2: Mixing training + inference on time-slicing ​

❌ Anti-pattern 3: Ignoring MIG slice availability ​

References ​

GPU Memory Strategies — MIG, Time-Slicing, Tradeoffs

Tại sao cần multiplexing

Multi-Instance GPU (MIG): Hardware Partitioning

Cơ chế hoạt động

MIG partition types

Setup MIG trên GKE

Constraints và failure modes

GPU Time-Slicing: Software Context-Switching

Cơ chế

Setup trên GKE

Constraints

NVIDIA Multi-Process Service (MPS): Process Kernel Sharing

Choosing multiplexing strategy

Decision tree

Workload patterns

Combining MIG + Time-Slicing

Monitoring and troubleshooting

Utilization tracking

Common issues

Mental model synthesis

Production Anti-patterns

❌ Anti-pattern 1: Assuming time-slicing memory enforcement

❌ Anti-pattern 2: Mixing training + inference on time-slicing

❌ Anti-pattern 3: Ignoring MIG slice availability

References