RBAC Audit Logging & Compliance

Vì Sao Quantum Trọng

Audit trail là only evidence để tracing who did what, when, and why trong cluster. Nó critical cho:

• Compliance — SOC2, HIPAA, PCI-DSS require complete authorization audit trail
• Incident investigation — "How did attacker escalate privileges?" answer chỉ via audit
• Change tracking — "Who created this ClusterRole?" requires audit
• Access review — "Who has permission to production?" audit-based answer
• Accountability — "Who deployed this to prod?" must be traceable

Nhưng audit logging sering misconfigured hoặc ignored. Nó "expensive" — lots of data, storage overhead, compliance burden. Nhưng nếu RBAC breach xảy ra, lack of audit trail == unrecoverable.

Memahami audit structure, filtering, retention crucial để maintain security posture.

Kubernetes Audit Logging Architecture

How Audit Works

API Server
  ↓
Authentication (set user identity)
  ↓
Authorization (RBAC check)
  ↓
Audit Handler (BEFORE mutation/admission)
  ↓
Mutation Admission (modify request)
  ↓
Validating Admission (check constraints)
  ↓
etcd Write
  ↓
Audit Handler (AFTER response)

Kubernetes audit captures request/response at two stages:

1. RequestReceived — after authentication, before authorization
2. ResponseComplete — after request processed, response available

Audit Log Structure

Mỗi audit record là JSON:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "12345-67890",
  "stage": "ResponseComplete",
  "requestReceivedTimestamp": "2026-06-26T12:34:56.789Z",
  "stageTimestamp": "2026-06-26T12:34:56.890Z",
  "user": {
    "username": "alice",
    "uid": "uid-123",
    "groups": ["developers", "system:authenticated"],
    "extra": {}
  },
  "impersonatedUser": {
    "username": "system:serviceaccount:default:my-app",
    "groups": ["system:serviceaccounts", "system:serviceaccounts:default"],
    "uid": "..."
  },
  "sourceIPs": ["192.168.1.100"],
  "userAgent": "kubectl/v1.30",
  "verb": "create",
  "apiGroup": "apps",
  "apiVersion": "v1",
  "objectRef": {
    "resource": "deployments",
    "namespace": "production",
    "name": "my-app",
    "uid": "uid-456",
    "apiVersion": "apps/v1"
  },
  "requestObject": {
    "apiVersion": "apps/v1",
    "kind": "Deployment",
    "metadata": {...},
    "spec": {...}
  },
  "responseStatus": {
    "code": 201,
    "message": "Created"
  },
  "responseObject": {
    "apiVersion": "apps/v1",
    ...
  },
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"deployer\""
  }
}

Key authorization-related fields:

Field	Purpose
user.username / user.groups	Original requester identity
impersonatedUser	If request impersonated another subject
verb	Authorization verb (get, create, delete, etc.)
apiGroup, resource, namespace	Resource being accessed
annotations.authorization.k8s.io/decision	ALLOW / DENY
annotations.authorization.k8s.io/reason	Why decision (RBAC rule, webhook response, etc.)
sourceIPs	Request origin
stage	RequestReceived hoặc ResponseComplete

Authorization Decision Tracking

ALLOW Decision

Request với successful authorization:

{
  "user": {"username": "alice"},
  "verb": "create",
  "objectRef": {"resource": "pods", "namespace": "default"},
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by RoleBinding \"pod-creator\""
  },
  "responseStatus": {"code": 201}
}

Tells: alice created pod trong default namespace, ALLOWED via RoleBinding pod-creator.

DENY Decision

Request denied authorization:

{
  "user": {"username": "alice"},
  "verb": "delete",
  "objectRef": {"resource": "secrets", "namespace": "production"},
  "annotations": {
    "authorization.k8s.io/decision": "deny",
    "authorization.k8s.io/reason": "RBAC: no RBAC policy matched"
  },
  "responseStatus": {"code": 403, "message": "Forbidden"}
}

Tells: alice attempted delete secrets trong production, DENIED (no matching RBAC rule).

Authorizer Decision

Nếu multiple authorizers active, annotation mencatat which authorizer decided:

{
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "webhook authorized: ALLOWED"
  }
}

Hoặc RBAC:

{
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"admin\""
  }
}

Clear trace về which authorizer responsible để decision.

Impersonation Audit Trail

Delegation Tracking

Khi user impersonate, BOTH identities logged:

{
  "user": {
    "username": "alice",
    "groups": ["developers"]
  },
  "impersonatedUser": {
    "username": "system:serviceaccount:default:deployer",
    "groups": ["system:serviceaccounts"]
  },
  "verb": "create",
  "objectRef": {"resource": "deployments"},
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by Role \"deployer\""
  }
}

Audit clearly shows:

1. Who (alice) issued command
2. As whom (deployer SA) it executed
3. What (create deployment) happened
4. Permission source (deployer role)

Detecting Impersonation Abuse

# Find all impersonation attempts
grep '"impersonatedUser"' /var/log/audit.log | wc -l

# Find impersonations that were DENIED
grep '"impersonatedUser"' /var/log/audit.log | \
  grep '"decision": "deny"'

# Find unusual delegation chains (alice → bob → charlie)
# (More complex analysis needed)

Complete audit trail prevent hidden privilege escalation.

Audit Policy Configuration

Audit Policy YAML

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log all requests at RequestResponse level
- level: RequestResponse
  omitStages:
  - RequestReceived

# Higher verbosity để dangerous verbs
- level: RequestResponse
  verbs: ["create", "update", "patch", "delete", "deletecollection"]
  omitStages:
  - RequestReceived

# Log RBAC changes
- level: RequestResponse
  resources:
  - group: "rbac.authorization.k8s.io"
    resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
  omitStages:
  - RequestReceived

# Log impersonation
- level: RequestResponse
  verbs: ["impersonate"]
  omitStages:
  - RequestReceived

# Log DENY decisions
- level: RequestResponse
  matchPolicy: "Any"
  rules:
  - resources: ["*"]
  # (Need to filter trong handler; audit policy doesn't support denial filtering directly)

# Default: log metadata level để high-volume requests
- level: Metadata
  omitStages:
  - RequestReceived

Audit Levels

Level	Logged	Use
None	Nothing	Disabled
Metadata	Request metadata, no body	High-volume (default)
RequestResponse	Full request + response	Sensitive operations
Request	Full request, no response	Debug

Trade-off: RequestResponse = most detail, nhưng massive storage overhead.

API Server Configuration

kube-apiserver \
  --audit-log-path=/var/log/audit.log \
  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100

Configuration options:

• --audit-log-path — output file
• --audit-policy-file — policy rules
• --audit-log-maxage — delete logs older than N days
• --audit-log-maxbackup — keep max N backup files
• --audit-log-maxsize — rotate nếu file > N MB

Compliance Patterns

RBAC Authorization Audit

Regulations require:

• Who authorized access
• What permissions granted
• When granted/revoked
• Why (business justification)

Kubernetes audit capture WHAT (permissions), WHEN, WHO. WHY requires external tracking (Jira, ServiceNow):

RBAC: Create RoleBinding admin binding cluster-admin để alice
AUDIT LOG: Records user=admin, action=create rolebinding
WHY: (Not trong audit log)
  → Need external ticket: TICKET-1234: "Grant alice cluster-admin nếu incident"

Authorization Decision Tracking

Để compliance, must track:

1. Request timestamp
2. Requester identity
3. Resource requested
4. Action (verb)
5. Decision (ALLOW/DENY)
6. Authorization rule that caused decision

All captured trong audit record; retention 30+ days standard.

Cross-Organization Access

Audit must track impersonation để delegate operations:

RECORD 1: User=org-admin, Action=impersonate org2-admin
RECORD 2: User=org2-admin (impersonated), Action=create user

Clearly shows org-admin delegated access để org2 admin task.

Audit Log Analysis

Query Examples

Find all cluster-admin grants:

grep "clusterrole.*cluster-admin" /var/log/audit.log | \
  grep "create" | jq '.user.username'

Find all DENY decisions:

grep '"decision": "deny"' /var/log/audit.log | \
  jq '{user: .user.username, verb: .verb, reason: .annotations["authorization.k8s.io/reason"]}'

Find all cross-namespace access attempts:

grep -E 'serviceaccount.*:' /var/log/audit.log | \
  jq 'select(.user.username | contains("serviceaccount")) | 
      select(.objectRef.namespace != null) |
      select((.user.username | split(":")[1]) != .objectRef.namespace)'

Find all impersonation:

grep '"impersonatedUser"' /var/log/audit.log | \
  jq '{original: .user.username, impersonated: .impersonatedUser.username}'

Automated Alerting

Setup webhooks/sidecar để real-time detection:

# Monitor audit log, alert trên suspicious patterns
tail -f /var/log/audit.log | \
  jq 'select(.annotations["authorization.k8s.io/decision"] == "deny") |
      select(.verb == "impersonate")' | \
  xargs -I {} curl -X POST http://alertmanager/api/v1/alerts \
    -d '{"reason": "Impersonation DENIED", "detail": {}}'

Retention & Storage

Audit logs large (1-10GB/day depending verbosity). Typical retention:

Regulation	Retention
SOC2	90 days
HIPAA	6 years
PCI-DSS	1 year

Storage strategies:

1. Local SSD — limited; rotate weekly
2. Cloud Storage (GCS, S3) — long-term, queryable
3. Log aggregation (ELK, Splunk, Google Cloud Logging) — centralized, searchable

GCP Logging Integration

GKE có thể send audit logs đến Cloud Logging:

gcloud container clusters create my-cluster \
  --enable-cloud-logging \
  --logging-service=logging.googleapis.com/kubernetes

Logs automatically sent tới Cloud Logging để long-term retention, searchability.

Best Practices

Best Practice 1: Log Authorization Decisions

Enable RequestResponse audit level để authorization changes:

- level: RequestResponse
  resources:
  - group: "rbac.authorization.k8s.io"
    resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
  omitStages:
  - RequestReceived

Every RBAC change fully logged với request + response.

Best Practice 2: Monitor DENY Decisions

Setup alert để DENY decisions; often indicate:

• Misconfigured RBAC
• Unauthorized access attempts
• Privilege escalation attempts

# Alert nếu >10 DENY decisions trong 1 minute
grep '"decision": "deny"' /var/log/audit.log | \
  wc -l | xargs -I {} test {} -gt 10 && \
  alert "High DENY count: potential attack"

Best Practice 3: Audit Log Integrity

Protect audit logs từ tampering:

# Make audit log append-only
chmod a-w /var/log/audit.log
chattr +a /var/log/audit.log  # append-only attribute

# Send logs tới remote storage (cannot be deleted locally)

Best Practice 4: Regular Access Reviews

Quarterly review authorization changes:

# Extract all RBAC changes trong quarter
jq -s 'map(select(.objectRef.resource | contains("role"))) |
       map({date: .stageTimestamp, user: .user.username, action: .verb, target: .objectRef.name}) |
       sort_by(.date)' audit-quarter-*.log > access-review.json

Compare với approved request tickets.

Best Practice 5: Document Decision Reason

Always include WHY nếu granting permission:

# In ConfigMap hoặc Jira ticket
RULE: deployer-prod (ClusterRoleBinding)
WHY: "Grant CI system ability to deploy prod applications"
WHO APPROVED: platform-team lead
AUDIT: Jira ticket DEPLOY-1234

Audit log shows WHAT and WHEN; documentation shows WHY.

Summary

Audit logging essential để RBAC security:

• Authorization decisions fully logged (ALLOW/DENY)
• Impersonation tracked (both original và delegated user)
• Complete audit trail prevents hidden escalation
• Compliance-ready — retention policies configurable

Setup auditing:

1. Enable RequestResponse level audit
2. Configure policy để sensitive operations
3. Setup long-term storage (GCS, Cloud Logging)
4. Regular analysis và alerting
5. Protect audit logs từ tampering

Tham Khảo

• Kubernetes Audit Logging
• Audit Policy Format
• GCP Cloud Logging for GKE
• Security Best Practices