Dynamic Resource Allocation (DRA) — Next-Gen GPU Scheduling

Tại sao DRA quan trọng (so với device plugins)

Device plugins (GPU, TPU):

GPU node pool (A100 × 8):
  nvidia.com/gpu: 8

Pod spec:
  limits:
    nvidia.com/gpu: 2

Scheduler:
  Allocates "2 units of generic GPU resource"
  Doesn't know: GPU type (A100 vs H100), memory, topology
  Result: Pod might get A100 (8GB mem, slow) or H100 (80GB mem, fast) — unpredictable

DRA (Dynamic Resource Allocation):

DeviceClass "high-mem-gpu":
  - GPU type: A100 or newer
  - VRAM: ≥40GB
  - Topology: no constraint

Pod spec:
  resourceClaim:
    name: training-gpu
  
  claims:
  - name: training-gpu
    resourceClassName: high-mem-gpu
    
Scheduler:
  Inspects cluster ResourceSlice (device inventory)
  Finds A100 with 40GB VRAM
  Allocates precisely matched GPU
  Result: Pod gets expected GPU

Mental model: Device plugins expose static resource counts. DRA exposes device attributes — scheduler can match on attributes, not just count.

Core concepts

ResourceSlice: Device inventory

Each GPU node exposes ResourceSlice:

yaml

apiVersion: resource.k8s.io/v1alpha3
kind: ResourceSlice
metadata:
  name: node-a-gpu-inventory
spec:
  nodeName: node-a
  devices:
  - name: gpu-0
    type: nvidia-a100-80gb
    attributes:
      memory-gb: "80"
      compute-capability: "8.0"
      topology: "pcie-slot-0"
  - name: gpu-1
    type: nvidia-a100-80gb
    attributes:
      memory-gb: "80"
      compute-capability: "8.0"
      topology: "pcie-slot-1"

Kubelet automatically generates ResourceSlice based on device plugin discovery.

DeviceClass: Device blueprint

Admin defines device categories:

yaml

apiVersion: resource.k8s.io/v1alpha3
kind: DeviceClass
metadata:
  name: high-mem-gpu
spec:
  selectors:
  - CELExpression: |
      device.attributes['memory-gb'] >= '40' &&
      device.attributes['compute-capability'] >= '8.0'
      
---
apiVersion: resource.k8s.io/v1alpha3
kind: DeviceClass
metadata:
  name: low-latency-fpga
spec:
  selectors:
  - CELExpression: |
      device.type == 'intel-fpga-low-latency' ||
      device.type == 'xilinx-fpga-low-latency'

ResourceClaim: Pod request

Pod requests device via claim:

yaml

apiVersion: resource.k8s.io/v1alpha3
kind: ResourceClaim
metadata:
  name: training-gpu-0
spec:
  resourceClassName: high-mem-gpu
  
---
apiVersion: v1
kind: Pod
metadata:
  name: training-job
spec:
  resourceClaims:
  - name: gpu-claim
    source:
      resourceClaimName: training-gpu-0
  
  containers:
  - name: trainer
    resources:
      claims:
      - name: gpu-claim

Scheduler:

1. Find all ResourceSlice with devices matching 'high-mem-gpu' class
2. Filter devices: memory >= 40GB AND compute >= 8.0
3. Check availability (not already allocated)
4. Bind ResourceClaim to device
5. Schedule Pod
6. kubelet exposes device to container (via device plugin)

ResourceClaimTemplate: Per-Pod claims

For multi-replica workloads:

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: distributed-training
spec:
  replicas: 4
  template:
    spec:
      resourceClaimTemplates:
      - metadata:
          name: gpu-claim
        spec:
          resourceClassName: high-mem-gpu
      
      containers:
      - name: trainer
        resources:
          claims:
          - name: gpu-claim

# Kubernetes auto-generates:
# - Pod 0: ResourceClaim training-distributed-training-0
# - Pod 1: ResourceClaim training-distributed-training-1
# - Pod 2: ResourceClaim training-distributed-training-2
# - Pod 3: ResourceClaim training-distributed-training-3

Scheduler improvements with DRA

Global device view

Without DRA:

Kubelet (Node A): GPU 0 free, GPU 1 allocated
Kubelet (Node B): GPU 0 allocated, GPU 1 free
Kubelet (Node C): GPU 0 free, GPU 1 free

Scheduler sees:
  allocatable.nvidia.com/gpu: 4 (on nodes A, B, C)
  But doesn't know which specific GPUs are free
  
Pod requests gpu: 2
  Scheduler: "I'll schedule on Node A + Node B (2 GPUs available)"
  But Node A GPU 0 busy, Node B GPU 1 busy
  → Pod pending (fragmentation)

With DRA:

Scheduler sees ResourceSlice:
  Node A: GPU 0 [busy], GPU 1 [free]
  Node B: GPU 0 [busy], GPU 1 [free]
  Node C: GPU 0 [free], GPU 1 [free]

Pod requests 2 GPUs of class 'high-mem-gpu':
  Scheduler: "allocate Node C GPU 0 + GPU 1"
  → Perfect fit, no fragmentation

Topology-aware placement

yaml

# DRA can filter by topology
spec:
  selectors:
  - CELExpression: |
      device.attributes['nvlink-capable'] == 'true' &&
      device.attributes['same-pcie-switch'] == 'true'
      # Only allocate GPUs connected via NVLink on same switch

Cross-cluster scheduling

Different clusters, different GPU types:

yaml

# Cluster A has A100 GPUs
# Cluster B has H100 GPUs

# Pod spec: just request 'high-mem-gpu'
resourceClassName: high-mem-gpu

# Can deploy same manifest to both clusters
# Scheduler matches to available GPU in each cluster

DRANET: Networking resource allocation

DRA extended to network interfaces (preview feature):

yaml

apiVersion: resource.k8s.io/v1alpha3
kind: NetworkDeviceClass
metadata:
  name: high-bandwidth-nic
spec:
  selectors:
  - CELExpression: |
      device.attributes['bandwidth-gbps'] >= '200' &&
      device.attributes['low-latency'] == 'true'

---
apiVersion: v1
kind: Pod
metadata:
  name: training-job
spec:
  resourceClaims:
  - name: gpu-claim
    source:
      resourceClaimName: training-gpu
  - name: nic-claim
    source:
      resourceClaimName: training-nic
  
  containers:
  - name: trainer
    resources:
      claims:
      - name: gpu-claim
      - name: nic-claim  # Allocate matched NIC

Use case: High-performance clusters where you need to co-allocate:

GPU (compute device)
NIC (network device) connected to same NUMA node

Comparison: device plugins vs DRA

Aspect	Device Plugin	DRA
Resource model	Static count (e.g., `gpu: 2`)	Dynamic attributes (e.g., memory, type)
Scheduler knowledge	Device count only	Full device attributes
Fragmentation	Possible (blind allocation)	Minimized (informed placement)
Setup complexity	Simple	More complex
Flexibility	Fixed topology	Flexible filtering
Multi-vendor support	Device-specific	Vendor-agnostic (CEL expressions)

Migration from device plugins to DRA

Phase 1: Coexistence

Both can run simultaneously:

Cluster with DRA + device plugins:
  GPUs exposed via:
  - ResourceSlice (DRA)
  - nvidia.com/gpu (device plugin)
  
  Pods can request either:
  - resourceClaim: high-mem-gpu (DRA)
  - limits: nvidia.com/gpu: 2 (device plugin)

Phase 2: Gradual adoption

New workloads use DRA, existing use device plugins:

yaml

# New training job (DRA)
apiVersion: batch/v1
kind: Job
metadata:
  name: new-training
spec:
  template:
    spec:
      resourceClaimTemplates:
      - name: gpu
        spec:
          resourceClassName: high-mem-gpu

---
# Existing serving job (device plugin)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-serving
spec:
  template:
    spec:
      containers:
      - name: server
        resources:
          limits:
            nvidia.com/gpu: 4  # Old-style request

Phase 3: Full migration

All workloads use DRA, device plugins deprecated.

GKE-specific DRA support

Enabling DRA

bash

gcloud container clusters create dra-cluster \
  --cluster-version=1.34+ \
  --enable-dynamic-resource-allocation

GKE automatically:

Deploys DRA scheduler plugin
Enables ResourceClaim API
Configures device plugin → ResourceSlice bridge

GKE DeviceClass examples

yaml

# Pre-built by Google Cloud
apiVersion: resource.k8s.io/v1alpha3
kind: DeviceClass
metadata:
  name: nvidia-h100-80gb
spec:
  selectors:
  - CELExpression: |
      device.type == 'nvidia-tesla-h100' &&
      device.attributes['memory-gb'] == '80'

---
apiVersion: resource.k8s.io/v1alpha3
kind: DeviceClass
metadata:
  name: tpu-v5p-8
spec:
  selectors:
  - CELExpression: |
      device.type == 'tpu-v5p' &&
      device.attributes['chip-count'] == '8'

---
apiVersion: resource.k8s.io/v1alpha3
kind: DeviceClass
metadata:
  name: gpu-with-nccl-fast-socket
spec:
  selectors:
  - CELExpression: |
      device.attributes['nccl-fast-socket'] == 'true' &&
      device.attributes['memory-gb'] >= '40'

Practical patterns

Pattern 1: Workload-specific device matching

yaml

apiVersion: resource.k8s.io/v1alpha3
kind: DeviceClass
metadata:
  name: inference-optimized
spec:
  selectors:
  - CELExpression: |
      device.attributes['inference-optimized'] == 'true' &&
      device.attributes['memory-gb'] >= '20'

Pattern 2: Multi-tier resource classes

yaml

# For different SLA tiers
apiVersion: resource.k8s.io/v1alpha3
kind: DeviceClass
metadata:
  name: premium-gpu  # SLA: <50ms latency
spec:
  selectors:
  - CELExpression: |
      device.attributes['latency-p99-ms'] < '50' &&
      device.attributes['memory-gb'] >= '80'

---
apiVersion: resource.k8s.io/v1alpha3
kind: DeviceClass
metadata:
  name: standard-gpu  # SLA: <200ms latency
spec:
  selectors:
  - CELExpression: |
      device.attributes['latency-p99-ms'] < '200'

---
apiVersion: resource.k8s.io/v1alpha3
kind: DeviceClass
metadata:
  name: batch-gpu  # SLA: best effort
spec:
  selectors:
  - CELExpression: |
      device.attributes['type'] == 'gpu'

Pattern 3: Reserved resources

yaml

# Reserve GPUs for critical workloads
apiVersion: resource.k8s.io/v1alpha3
kind: DeviceClass
metadata:
  name: reserved-critical
spec:
  selectors:
  - CELExpression: |
      device.attributes['reserved-for'] == 'critical' &&
      device.attributes['node'] == 'gpu-reserved-1'

Failure modes

Claim starvation

5 Pod requests for DeviceClass 'high-mem-gpu'
Only 2 GPUs match the class

3 Pods pending indefinitely
(DRA scheduler waiting, no backoff)

Solution:
  1. Create broader DeviceClass
  2. Add more matching GPUs
  3. Use priority/preemption (lower priority Pod evicted)

Device type churn

Admin changes DeviceClass definition:
  Before: memory-gb >= 40
  After: memory-gb >= 80

Existing ResourceClaim (allocated to 40GB GPU):
  Can still use it (claim binding doesn't change)
  
New ResourceClaim:
  Only matches 80GB GPUs
  
Result: inconsistent behavior across old/new claims

Mental model: DRA as resource matching engine

Device plugins:
  "Give me 2 GPUs"
  (scheduler: "here's 2 units")

DRA:
  "Give me 2 devices matching: high-memory AND low-latency"
  (scheduler: "checking inventory... found 2 matching devices on Node X")

DRA is not a replacement for device plugins (simpler to understand). It's an upgrade for complex heterogeneous clusters where attribute matching matters.

Dynamic Resource Allocation (DRA) — Next-Gen GPU Scheduling ​

Tại sao DRA quan trọng (so với device plugins) ​

Core concepts ​

ResourceSlice: Device inventory ​

DeviceClass: Device blueprint ​

ResourceClaim: Pod request ​

ResourceClaimTemplate: Per-Pod claims ​

Scheduler improvements with DRA ​

Global device view ​

Topology-aware placement ​

Cross-cluster scheduling ​

DRANET: Networking resource allocation ​

Comparison: device plugins vs DRA ​

Migration from device plugins to DRA ​

Phase 1: Coexistence ​

Phase 2: Gradual adoption ​

Phase 3: Full migration ​

GKE-specific DRA support ​

Enabling DRA ​

GKE DeviceClass examples ​

Practical patterns ​

Pattern 1: Workload-specific device matching ​

Pattern 2: Multi-tier resource classes ​

Pattern 3: Reserved resources ​

Failure modes ​

Claim starvation ​

Device type churn ​

Mental model: DRA as resource matching engine ​

References ​

Dynamic Resource Allocation (DRA) — Next-Gen GPU Scheduling

Tại sao DRA quan trọng (so với device plugins)

Core concepts

ResourceSlice: Device inventory

DeviceClass: Device blueprint

ResourceClaim: Pod request

ResourceClaimTemplate: Per-Pod claims

Scheduler improvements with DRA

Global device view

Topology-aware placement

Cross-cluster scheduling

DRANET: Networking resource allocation

Comparison: device plugins vs DRA

Migration from device plugins to DRA

Phase 1: Coexistence

Phase 2: Gradual adoption

Phase 3: Full migration

GKE-specific DRA support

Enabling DRA

GKE DeviceClass examples

Practical patterns

Pattern 1: Workload-specific device matching

Pattern 2: Multi-tier resource classes

Pattern 3: Reserved resources

Failure modes

Claim starvation

Device type churn

Mental model: DRA as resource matching engine

References