My Awesome Project

Appearance

Admission Control Pipeline — MutatingAdmissionWebhook, ValidatingAdmissionWebhook, etcd Write ​

Tại Sao Admission Control Quan Trọng ​

Admission webhooks là last line of defense trước khi data ghi vào etcd. Misconfigured webhook có thể:

  • Reject tất cả requests → cluster unavailable
  • Mangle data → inconsistent state
  • Timeout → hanging API calls

Hiểu admission pipeline → design safe webhooks, debug webhook issues.

Request Pipeline Tầng ​

1. TLS Termination
2. Authentication
3. Authorization  
4. ┌──────────────────────┐
   │ Admission Control    │ ← We are here
   │  ├─ Mutating         │
   │  └─ Validating       │
   └──────────────────────┘
5. Resource Quota Check
6. etcd Write
7. Response Serialization

Mutating Admission Webhooks ​

Role ​

Mutating webhooks modify request object trước ghi ke etcd:

yaml
# User submits
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  labels: {}  # empty labels
spec:
  containers:
  - name: app

# Mutating webhook (e.g., istio-injection):
# Adds sidecar container

# After mutation
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  labels:
    istio-injection: enabled
spec:
  containers:
  - name: app
  - name: istio-proxy  # ← Added by webhook

Execution Order ​

Multiple mutating webhooks execute sequentially dalam ordered specified:

yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: my-mutators
webhooks:
- name: webhook-a.example.com
  admissionReviewVersions: ["v1"]
  timeoutSeconds: 5
- name: webhook-b.example.com
  admissionReviewVersions: ["v1"]
  timeoutSeconds: 5
- name: webhook-c.example.com
  admissionReviewVersions: ["v1"]
  timeoutSeconds: 5

# Execution order:
# webhook-a executes first
#   ↓ (output becomes input untuk next)
# webhook-b executes
#   ↓ (output becomes input untuk next)
# webhook-c executes
#   ↓
# Final object sent ke validating webhooks

Mutation Example: Service Account Injection ​

yaml
apiVersion: v1
kind: Pod
metadata:
  name: app-pod
  namespace: default
spec:
  # No serviceAccount specified
  containers:
  - name: app

# Mutating webhook adds default service account
---
# After mutation:
apiVersion: v1
kind: Pod
metadata:
  name: app-pod
  namespace: default
spec:
  serviceAccountName: default  # ← Added
  serviceAccount: default      # ← Added
  containers:
  - name: app

Validating Admission Webhooks ​

Role ​

Validating webhooks check object correctness sebelum persistence:

yaml
# User submits
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: app
    securityContext:
      privileged: true

# Validating webhook (policy enforcer):
# Rule: pods tidak boleh privileged
# Decision: DENY

# Result: Request rejected
# User error: "Privileged containers not allowed"

Execution Order ​

All validating webhooks execute in parallel (after all mutating webhooks complete):

Mutating Webhooks (sequential):
webhook-a → webhook-b → webhook-c

Final object → Validating Webhooks (parallel):
           ├─ validate-a (check policy)
           ├─ validate-b (check resource limits)
           └─ validate-c (check labels)

If ANY reject → Request denied
If ALL approve → Object persisted

Webhook Configuration ​

Failure Policies ​

yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
webhooks:
- name: webhook.example.com
  failurePolicy: Fail      # ← Important!
  # OR: failurePolicy: Ignore
  
  # Fail = reject request if webhook fails
  # Ignore = allow request if webhook fails

Timeout Configuration ​

yaml
webhooks:
- name: webhook.example.com
  timeoutSeconds: 5  # ← Default: 10 seconds
  # After 5 seconds, if webhook not respond:
  # failurePolicy determines: accept or reject

Scope & Resources ​

yaml
webhooks:
- name: webhook.example.com
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
    scope: "Namespaced"  # Only namespaced pods

Failure Modes ​

Failure Mode 1: Webhook Timeout ​

Webhook:  5-second processing
Config:   2-second timeout
     ↓
API Server waits 2 seconds
     ↓
Timeout → failurePolicy logic
  ├─ Fail: Request rejected → User error 504
  └─ Ignore: Request allowed → Data potentially corrupted

Prevention: Set reasonable timeouts, optimize webhook performance

Failure Mode 2: Webhook Crash ​

Webhook endpoint down
User submits Pod
     ↓
API Server tries connect webhook
     ↓
Connection refused
     ↓
failurePolicy decision:
  ├─ Fail: Pod rejected
  └─ Ignore: Pod created without validation

Prevention: Monitor webhook availability, use HA webhook setup

Failure Mode 3: Cascading Failures ​

Webhook A mutates Pod
Webhook B expects Pod in certain format
     ↓
Pod format from webhook A breaks webhook B assumptions
     ↓
Webhook B rejects Pod
     ↓
User cannot create any Pods

Prevention: Careful webhook ordering, comprehensive testing

Common Webhook Patterns ​

Pattern 1: Pod Security Policy Enforcement ​

yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-security
webhooks:
- name: validate.pod-security.io
  clientConfig:
    service:
      name: policy-webhook
      namespace: kube-system
      path: "/validate-pods"
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: [""]
    resources: ["pods"]
  failurePolicy: Fail  # Strict enforcement
  sideEffects: None

Pattern 2: Label Injection ​

yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: label-injector
webhooks:
- name: inject.labels.io
  clientConfig:
    service:
      name: label-injector
      namespace: kube-system
      path: "/mutate-labels"
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    resources: ["pods", "services"]
  failurePolicy: Ignore  # Non-critical, allow if fails

Admission Webhook Best Practices ​

Anti-Pattern 1: Heavy Computation in Webhook ​

go
// ❌ BAD - Slow webhook blocks all Pod creation
func ValidateHandler(w http.ResponseWriter, r *http.Request) {
    // Network calls
    result := callSlowExternalAPI()
    
    // Computation
    for i := 0; i < 1000000; i++ {
        complexCalculation()
    }
    
    // Returns after 5+ seconds
}

// ✅ GOOD - Fast validation
func ValidateHandler(w http.ResponseWriter, r *http.Request) {
    // Local policy check only
    allowed := pod.Spec.Privileged == false
    
    // Returns in milliseconds
}

Anti-Pattern 2: Webhook That Crashes on Edge Cases ​

go
// ❌ BAD
func MutateHandler(w http.ResponseWriter, r *http.Request) {
    var review admissionv1.AdmissionReview
    json.NewDecoder(r.Body).Decode(&review)
    
    pod := review.Request.Object // What if nil?
    containers := pod.Spec.Containers // What if empty?
    
    containers[0].Image = "mutated"  // Panic if len(containers)==0!
}

// ✅ GOOD
func MutateHandler(w http.ResponseWriter, r *http.Request) {
    var review admissionv1.AdmissionReview
    json.NewDecoder(r.Body).Decode(&review)
    
    pod := review.Request.Object
    if pod == nil {
        denyWithError("Invalid pod object")
        return
    }
    
    if len(pod.Spec.Containers) == 0 {
        // Handle gracefully
        return
    }
    
    pod.Spec.Containers[0].Image = "mutated"
}

Troubleshooting Webhooks ​

Debug: Check Webhook is Registered ​

bash
kubectl get mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations

kubectl describe mutatingwebhookconfigurations my-webhook

Debug: Check Webhook is Reachable ​

bash
# From API Server perspective, webhook must be reachable
# Typical setup:
clientConfig:
  service:
    name: webhook-service
    namespace: kube-system
    path: "/validate"
  caBundle: <base64-encoded-CA>

Debug: Enable Audit Logging ​

bash
# Check which webhooks are being called
kubectl logs -n kube-system kube-apiserver | grep webhook

Debug: Manual Test ​

bash
# Create test pod, check if webhook called
kubectl create -f test-pod.yaml -v 9

# Should see webhook call dalam logs

Cluster Stability Patterns ​

Pattern: Multiple Webhook Instances ​

yaml
# Don't rely on single webhook instance
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
webhooks:
- name: webhook.example.com
  failurePolicy: Ignore  # If ONE instance down, allow request
  clientConfig:
    service:
      name: webhook-lb  # Load balanced service

Pattern: Webhook Timeout ​

yaml
webhooks:
- name: webhook.example.com
  timeoutSeconds: 3  # Err on side of fast failure
  failurePolicy: Ignore  # Allow if timeout

Reference Documentation ​

Summary ​

  • Mutating webhooks: Modify objects (sequential execution)
  • Validating webhooks: Check compliance (parallel execution)
  • Failure policies: Fail = strict, Ignore = lenient
  • Timeout important: Default 10s, consider shorter
  • Performance critical: Webhooks serialize/deserialize overhead
  • HA pattern: Multiple webhook instances behind load balancer
  • Debugging: Enable audit, check connectivity, test manually