Storage Issues: Volume Attachment Failures

Symptoms Recognition

Storage issues xuất hiện khi:

Pod stuck Pending (waiting for volume mount)
Pod status MountFailure hoặc UnmountFailure
Event: "Unable to attach or mount volumes"
Pod Running nhưng mount timeout hung indefinitely
Pod CrashLoop sau mount success

Why This Matters

Storage failures có high production impact. Database pods, stateful workloads, cache layers đều phụ thuộc vào volume. Một mount timeout có thể cause cascading failures (pod restart loop, data corruption). Understanding volume attachment mechanics ở GKE là critical.

Information Gathering — Quick Diagnostics

Step 1: Check PVC & PV Status

bash

# View PVC status
kubectl get pvc -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>

# View PV status
kubectl get pv
kubectl describe pv <pv-name>

# View storage class
kubectl get storageclass
kubectl describe storageclass <sc-name>

Interpretasi:

PVC status Bound = successfully provisioned
PVC status Pending = provisioning stuck (check CSI driver)
PV status Bound = attached to node
PV status Released = available để claim (correct state)

Step 2: Check Pod Volume Mount Status

bash

# Describe pod để volume info
kubectl describe pod <pod-name> -n <namespace>
# Cari section "Mounts" và "Volumes"

# View mount point di pod
kubectl exec <pod-name> -n <namespace> -- mount | grep <pvc-name>

# View mount failures trong logs
kubectl logs <pod-name> -n <namespace> | grep -i "mount\|volume"

# Check kernel logs để mount errors
kubectl debug node/<node-name> -it --image=ubuntu
# Trong debug shell: dmesg | grep -i mount

Cek để:

Mount point exists? (/data or path specified)
Permissions correct? (ls -la /data)
Filesystem type expected? (df -T)
Out of space? (df -h)

Step 3: Check CSI Driver Status

bash

# GKE uses Google Persistent Disk CSI Driver
kubectl get pods -n kube-system -l app=gce-pd-csi-driver

# Check driver logs
kubectl logs -n kube-system -l app=gce-pd-csi-driver -c csi-driver

# View CSI driver events
kubectl get events -n kube-system --field-selector reason=AttachFailed,reason=MountFailed

Daftar để dipahami:

Attachment errors → disk quota exceeded?
Provisioning errors → storage class issue?
Timeout errors → CSI driver hung?

Step 4: Check Node Disk/Volume Attachment Limits

bash

# View attached volumes per node
kubectl describe node <node-name> | grep -A10 "VolumesAttached"

# Cek persistent disk quota
gcloud compute disks list --filter="zone:<zone>" | wc -l

# Max disks per instance trong GCP
# Standard: 128 disks per instance (soft limit)
# High-priority: quota có thể increase via request

Step 5: Check fsGroup & Permission Issues

bash

# View pod securityContext
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A10 "securityContext:"

# View volume mountPath permissions
kubectl exec <pod-name> -n <namespace> -- ls -la <mount-path>

# View file count trong volume (nếu mount success)
kubectl exec <pod-name> -n <namespace> -- find <mount-path> | wc -l

Diagnostic Decision Tree

Storage Issue
│
├─ PVC status?
│  ├─ Pending → PV provisioning stuck
│  │  ├─ Check storage class exists
│  │  ├─ Check CSI driver pod running
│  │  ├─ Check zone/region constraint
│  │  └─ Check quota
│  │
│  └─ Bound → continue
│
├─ Pod scheduled?
│  ├─ NO → Pending (see Scheduling Failures section)
│  │
│  └─ YES → continue
│
├─ Volume attached to node?
│  ├─ NO → node.status.volumesAttached empty
│  │  └─ Wait (attach in progress) hoặc check CSI logs
│  │
│  └─ YES → continue
│
├─ Volume mounted inside pod?
│  ├─ NO → "Unable to mount" event
│  │  ├─ filesystem type mismatch?
│  │  ├─ fsGroup permission issue?
│  │  ├─ mount options incompatible?
│  │  └─ Check CSI driver logs
│  │
│  └─ YES → pod should running
│     └─ Continue to app-level issues
│
├─ fsGroup change taking long time?
│  ├─ YES → fsGroup chmod recursively
│  │  └─ Wait hoặc reduce file count
│  │
│  └─ NO → continue
│
└─ Other storage-related errors?
   ├─ Out of space → expand PV
   ├─ Permission denied → check securityContext
   ├─ Stale NFS mount → umount force + remount

Common Root Causes & Fixes

Root Cause 1: Storage Class Not Found

Dấu hiệu:

PVC event: "storageclass does not exist"
PVC stuck Pending indefinitely
CSI driver logs show storage class <name> not found

Nguyên nhân: PVC storage class không tồn tại hoặc bị gõ sai.

Diagnostic:

bash

# View storage classes
kubectl get storageclass

# View PVC spec
kubectl get pvc <pvc-name> -n <namespace> -o yaml | grep storageClassName

# View default storage class
kubectl get storageclass | grep default

Immediate Fix:

bash

# Option 1: Use existing storage class
EXISTING_SC=$(kubectl get storageclass -o jsonpath='{.items[0].metadata.name}')
kubectl patch pvc <pvc-name> -n <namespace> \
  -p '{"spec":{"storageClassName":"'$EXISTING_SC'"}}'

# Option 2: Create missing storage class
kubectl create -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <sc-name>
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-standard
  replication-type: regional-pd
EOF

# Option 3: Update PVC to use existing class
kubectl edit pvc <pvc-name> -n <namespace>
# Change storageClassName field

Permanent Fix:

GKE phải sở hữu default storage class
Document available storage classes để developers
Create common storage classes vào cluster initialization

Root Cause 2: PersistentVolume Quota Exceeded

Dấu hiệu:

CSI driver logs: "quota exceeded" hoặc "limit reached"
Event: "FailedCreation: ... quota ... "
Unable to create new PV

Nguyên nhân: GCP quota để Persistent Disks đạt đến limit. Per zone/region quota có batasan.

Diagnostic:

bash

# View current PD count di project
gcloud compute disks list --filter="zone:<zone>" | tail -1

# View quota limit
gcloud compute project-info describe --project=<project> | grep -A1 "quota.*SSD_TOTAL_GB\|DISKS_TOTAL_GB"

# Alternative: Cloud Console → Quotas & System Limits

Immediate Fix:

bash

# Option 1: Increase quota
# Cloud Console → Quotas & System Limits → search "Persistent Disks per zone"
# Click → Edit Quotas → Increase

# Option 2: Delete unused PVs (if safe)
kubectl get pv -o wide
# Identify unused PVs
kubectl delete pv <pv-name>

# Option 3: Use smaller PV size
# Edit PVC: reduce resources.requests.storage

Permanent Fix:

Regular audit PV usage
Implement PV retention policies (auto-delete old unused)
Pre-request quota increase berdasarkan growth plan
Monitor quota usage qua Cloud Monitoring

Root Cause 3: fsGroup Permission Timeout

Dấu hiệu:

Pod stuck Init:0/N hoặc PodInitializing để 5+ menit
Event: "chown the volume" hoặc similar
Pod finally runs nhưng delayed 10+ minutes
Volume punya millions of files

Nguyên nhân: GKE kubelet thực hiện recursive chown để fsGroup (thay đổi quyền sở hữu tất cả file). Với millions of files, này rất lambat.

Diagnostic:

bash

# Check pod spec để fsGroup
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A5 "securityContext:"

# View kubelet logs để fsGroup operation
# On node (via SSH hoặc debug pod):
dmesg | grep -i "chown\|fsgroup"
journalctl -u kubelet | grep -i "chown"

# View file count trong volume (if accessible)
kubectl exec <pod> -n <namespace> -- find <mount-path> | wc -l

Immediate Fix:

bash

# Option 1: Remove fsGroup (if app doesn't need it)
kubectl patch pod <pod-name> -n <namespace> --type=json \
  -p='[{"op":"remove","path":"/spec/securityContext/fsGroup"}]'
# Better: update deployment spec

# Option 2: Change fsGroupChangePolicy to OnRootMismatch
kubectl patch deployment <dep> -n <namespace> --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/securityContext/fsGroupChangePolicy","value":"OnRootMismatch"}]'

# Option 3: Reduce volume size hoặc cleanup files
# Clean old data trong volume
kubectl exec <pod> -n <namespace> -- find <mount-path> -type f -mtime +30 -delete

Permanent Fix:

Avoid fsGroup nếu không diperlukan (app can run với current permissions)
Use fsGroupChangePolicy: OnRootMismatch (GKE 1.20+)
- Only changes ownership nếu needed
- Signifikan faster để large volumes
Educate developers: Understand fsGroup overhead

Implement monitoring:

bash

# Alert if pod initialization > 5 minutes

Root Cause 4: Filesystem Type Mismatch

Dấu hiệu:

Event: "MountFailure: ... wrong fs type ..."
mount: /mnt: unknown filesystem type 'ext4'" (hoặc lainnya)
Pod cannot start

Nguyên nhân: PVC spec sở hữu fsType: ext4 nhưng disk belum diformat, hoặc kernel không support filesystem type.

Diagnostic:

bash

# View PV spec
kubectl get pv <pv-name> -o yaml | grep -i "fstype\|format"

# View mount error trong pod
kubectl logs <pod-name> -n <namespace> | head -50

# Check trong node (via debug pod)
kubectl debug node/<node-name> -it --image=ubuntu
# Trong shell: lsblk -f (View filesystem types)

Immediate Fix:

bash

# Option 1: Nếu disk not formatted, GCP SDK không auto-format
# Manual format (HATI-HATI: data loss):
# 1. Delete PVC + PV
# 2. Create PV với fsType: ext4
# 3. Manually format disk: mkfs.ext4 /dev/sdf (on node)
# 4. Recreate PVC

# Option 2: Change fsType đến supported
# Default GCP: ext4 (preferred)
kubectl patch pv <pv-name> -p '{"spec":{"storageClassName":"<sc>"}}'
# Update PVC fsType: ext4

# Option 3: Use pre-formatted disk
# GCP Persistent Disk phải pre-formatted manually

Permanent Fix:

Document supported filesystem types
Pre-format disks nếu custom type
Use standard ext4 (most compatible)
Testing mounting với various fs types trong staging

Root Cause 5: PersistentVolume Attachment Limit Exceeded

Dấu hiệu:

Event: "max limit of 128 disks per instance reached"
New PVC không attach đến node
Attachment stuck pending

Nguyên nhân: GCP instance sở hữu max disk attachment limit (default 128 per instance). Cluster sudah at maximum.

Diagnostic:

bash

# View attached disks per node
kubectl describe node <node-name> | grep "VolumesAttached" -A5

# Count total volumes
kubectl get pv -o json | jq '.items | length'

# Per-node breakdown
for node in $(kubectl get nodes -o name); do
  count=$(kubectl describe $node | grep "VolumesAttached" | wc -l)
  echo "$node: $count volumes"
done

Immediate Fix:

bash

# Option 1: Delete unused PVs
kubectl get pv --sort-by=.metadata.creationTimestamp
# Identify old/unused PVs, delete:
kubectl delete pv <pv-name>

# Option 2: Create new node pool với higher machine type
# Larger machines có thể attach lebih nhiều disks:
gcloud container node-pools create <new-pool> \
  --cluster=<cluster> --zone=<zone> \
  --machine-type=n1-highmem-2 \
  --num-nodes=1

# Option 3: Use regional disks (có thể share across zones)
# Reduce total disk count với consolidation

# Option 4: Request quota increase để higher limit
# (up to 1000 disks possible với request)

Permanent Fix:

Architecture planning: consolidate data
Use database layers (Cloud SQL, Firestore) instead of per-pod volumes
Implement PV cleanup (soft-delete after age)
Monitoring: track volume per-node ratio

Prevention & Monitoring

Monitoring Setup

bash

# Cloud Monitoring metrics để storage:
# 1. kubernetes.io/pvc/used_bytes
# 2. kubernetes.io/pv/available_bytes
# 3. gce.resource/disk/read_bytes_count
# 4. gce.resource/disk/write_bytes_count

# Alert rules:
# - PVC usage > 80% capacity
# - Mount latency > 30s
# - Disk attachment failures

Preventive Configuration

yaml

# Recommended: StorageClass với best practices
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard-rwo
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-standard
  replication-type: regional-pd
volumeBindingMode: WaitForFirstConsumer  # Bind nếu pod schedule
allowVolumeExpansion: true  # Allow grow size
---
# Set resource requests để PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: standard-rwo
  resources:
    requests:
      storage: 10Gi
---
# Pod với fsGroupChangePolicy
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  securityContext:
    fsGroup: 1000
    fsGroupChangePolicy: OnRootMismatch  # Fast mode
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-pvc

Escalation Criteria

Escalate if:

Multiple volumes failing attachment (infrastructure issue)
CSI driver CrashLoop (requires GKE support)
Disk quota unexpectedly exceeded
Data corruption suspected (do NOT delete, escalate immediately)

Pod Creation Failures — Nếu PVC không provision
Scheduling Failures — Nếu PVC blocking pod schedule
Control Plane Issues — Nếu API slow (affects CSI controller)

Storage Issues: Volume Attachment Failures ​

Symptoms Recognition ​

Why This Matters ​

Information Gathering — Quick Diagnostics ​

Step 1: Check PVC & PV Status ​

Step 2: Check Pod Volume Mount Status ​

Step 3: Check CSI Driver Status ​

Step 4: Check Node Disk/Volume Attachment Limits ​

Step 5: Check fsGroup & Permission Issues ​

Diagnostic Decision Tree ​

Common Root Causes & Fixes ​

Root Cause 1: Storage Class Not Found ​

Root Cause 2: PersistentVolume Quota Exceeded ​

Root Cause 3: fsGroup Permission Timeout ​

Root Cause 4: Filesystem Type Mismatch ​

Root Cause 5: PersistentVolume Attachment Limit Exceeded ​

Prevention & Monitoring ​

Monitoring Setup ​

Preventive Configuration ​

Escalation Criteria ​

Related Sections ​

References ​

Storage Issues: Volume Attachment Failures

Symptoms Recognition

Why This Matters

Information Gathering — Quick Diagnostics

Step 1: Check PVC & PV Status

Step 2: Check Pod Volume Mount Status

Step 3: Check CSI Driver Status

Step 4: Check Node Disk/Volume Attachment Limits

Step 5: Check fsGroup & Permission Issues

Diagnostic Decision Tree

Common Root Causes & Fixes

Root Cause 1: Storage Class Not Found

Root Cause 2: PersistentVolume Quota Exceeded

Root Cause 3: fsGroup Permission Timeout

Root Cause 4: Filesystem Type Mismatch

Root Cause 5: PersistentVolume Attachment Limit Exceeded

Prevention & Monitoring

Monitoring Setup

Preventive Configuration

Escalation Criteria

Related Sections

References