RBAC for Multi-Tenancy & Namespace Isolation

Vì Sao Quantum Trọng

Multi-tenant Kubernetes clusters — shared infrastructure cho multiple teams/customers — common pattern ở production. Nhưng design RBAC sai có thể result trong:

Cross-tenant access — Team A inadvertently (or maliciously) access Team B's resources, data
Privilege escalation — Tenant escalate tới cluster-admin hoặc other tenants
Audit opacity — Impossible tracking per-tenant activity
Blast radius — Compromise của one tenant affect lainnya

RBAC là first line of defense để tenant isolation. Understanding how RBAC enforce boundaries, and failure modes, critical để safe multi-tenancy.

Multi-Tenancy Model

Namespace-Per-Tenant Isolation

Standard model: masing-masing tenant gets dedicated namespace(s):

Cluster
├── Namespace: tenant-a
│   ├── Pods (tenant-a only)
│   ├── Services (tenant-a only)
│   └── Secrets (tenant-a only)
├── Namespace: tenant-b
│   ├── Pods (tenant-b only)
│   └── ...
├── Namespace: tenant-c
│   └── ...
└── Namespace: kube-system (shared, restricted)

RBAC enforces:

Tenant-a ServiceAccounts có thể access resources chỉ trong tenant-a namespace
Tenant-a không có thể access resources trong tenant-b
Cross-namespace access explicitly denied

Implementation: RoleBinding Scoping

Each tenant's RBAC strictly namespaced:

yaml

# Tenant A's RBAC
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: tenant-a         # SCOPED
  name: tenant-admin
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: tenant-a         # SCOPED: only tenant-a
  name: admin-binding
roleRef:
  kind: Role
  name: tenant-admin
subjects:
- kind: ServiceAccount
  namespace: tenant-a
  name: admin

---
# Tenant B's RBAC (completely separate)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: tenant-b
  name: tenant-admin
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: tenant-b
  name: admin-binding
roleRef:
  kind: Role
  name: tenant-admin
subjects:
- kind: ServiceAccount
  namespace: tenant-b
  name: admin

tenant-a:admin authorized chỉ trong tenant-a. tenant-b:admin authorized chỉ trong tenant-b.

Cross-namespace không possible via standard RoleBinding.

Cross-Namespace Access Patterns

Scenario 1: Shared Service (Logs)

Centralized logging service trong kube-system namespace. Tenants ingin grant logging SA access đến logs từ pods trong tenant namespaces.

Pattern: ClusterRole + RoleBinding in each tenant namespace

yaml

# ClusterRole: read logs globally
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: log-reader
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list"]
---
# Bind trong kube-system namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: kube-system
  name: logging-sa-reader
roleRef:
  kind: ClusterRole
  name: log-reader
subjects:
- kind: ServiceAccount
  namespace: kube-system
  name: logger

---
# THEN: Bind same ClusterRole trong EACH tenant namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: tenant-a
  name: logger-access
roleRef:
  kind: ClusterRole
  name: log-reader
subjects:
- kind: ServiceAccount
  namespace: kube-system
  name: logger

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: tenant-b
  name: logger-access
roleRef:
  kind: ClusterRole
  name: log-reader
subjects:
- kind: ServiceAccount
  namespace: kube-system
  name: logger

Result: kube-system:logger có thể read pods/logs trong mỗi tenant namespace (via explicit RoleBindings trong each namespace).

Important: Logger SA không có thể read things lainnya (secrets, configmaps) — kaya restricted via ClusterRole definition.

Scenario 2: Monitoring Access

Similar pattern để monitoring stack mà needs access đến metrics seluruh cluster:

yaml

# Central monitoring SA
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: monitoring
  name: prometheus
---
# ClusterRole: metrics read-only
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metrics-reader
rules:
- apiGroups: [""]
  resources: ["nodes", "nodes/metrics", "services", "pods"]
  verbs: ["get", "list"]
---
# Bind trong EVERY tenant namespace
# (For large clusters, use automation để create these bindings)

Automation:

bash

for tenant in tenant-a tenant-b tenant-c; do
  kubectl create rolebinding monitoring-access \
    -n $tenant \
    --clusterrole=metrics-reader \
    --serviceaccount=monitoring:prometheus
done

Escape Prevention

Threat Model

Threat: Tenant user escalate privileges beyond namespace.

Attack vectors:

Create role/rolebinding trong cluster để escalate

bash

# Tenant A tries to create cluster-admin for self
kubectl create clusterrole new-admin --verb='*' --resource='*'
# Denied: tenant A không punya create verb trên clusterroles

Impersonate cluster-admin user

bash

# Tenant A tries impersonate cluster-admin
kubectl create role admin --verb='*' --resource='*'
kubectl create rolebinding self-admin --role=admin
# Denied: tenant A không punya impersonate verb

Write admission webhooks

bash

# Tenant tries create validating/mutating webhooks
kubectl apply -f webhook.yaml
# Should be blocked: webhooks cluster-scoped, require admin

Prevention Mechanisms

Mechanism 1: RBAC Boundaries

Explicit RBAC configuration prevent escalation:

yaml

# Each tenant role EXPLICITLY limited
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: tenant-a
  name: tenant-admin
rules:
# Only resource trong tenant-a namespace
- apiGroups: ["", "apps", "batch", "extensions"]
  resources: ["pods", "services", "deployments", "jobs", "replicasets"]
  verbs: ["*"]

# NO permission để create roles/rolebindings
# (implicit denial)

Tenant-a cannot:

Create new roles
Modify existing RBAC
Impersonate other tenants
Create webhooks

Mechanism 2: Network Policies

RBAC + Network Policy prevents cross-tenant traffic:

yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  namespace: tenant-a
  name: deny-cross-tenant
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          tenant: tenant-a

Even nếu RBAC misconfigured, network policy prevent pod-to-pod communication.

Mechanism 3: Admission Controllers

Webhook admission controller có thể enforce policies:

Request: Tenant A create ClusterRole
        ↓
Admission Webhook check: "Is requester cluster-admin?"
        ↓
        NO → Reject (even if RBAC incorrectly allow)

Defense in depth: RBAC + admission enforce tenant boundaries.

Audit Per Tenant

Tracking Tenant Activity

Audit trail must clearly identify tenant:

json

{
  "user": {
    "username": "system:serviceaccount:tenant-a:app",
    "groups": ["system:serviceaccounts", "system:serviceaccounts:tenant-a"]
  },
  "objectRef": {
    "namespace": "tenant-a",
    "resource": "pods",
    "name": "my-pod"
  },
  "verb": "create",
  "sourceIPs": ["10.0.0.5"]
}

Key fields để tenant tracking:

user.username — serviceaccount identity indicates tenant
objectRef.namespace — action trong which tenant
verb — what action
sourceIPs — where action originate (can indicate rogue activity)

Per-Tenant Audit Log Filtering

bash

# Extract tenant-a activity
grep "system:serviceaccount:tenant-a" /var/log/audit.log | \
  jq '. | select(.objectRef.namespace == "tenant-a")'

# Find suspicious activity (cross-tenant attempt)
grep "system:serviceaccount:tenant-a" /var/log/audit.log | \
  jq '. | select(.objectRef.namespace != "tenant-a")'

Last query cho thấy "Tenant-a resource access attempt đến non-tenant-a namespace" → suspicious.

Tenant-Specific Audit Policies

yaml

# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log all tenant RBAC modifications
- level: RequestResponse
  verbs: ["create", "update", "patch", "delete"]
  resources: ["rolebindings", "clusterrolebindings", "roles", "clusterroles"]
  omitStages:
  - RequestReceived

# Log all cross-namespace access (suspicious)
- level: RequestResponse
  omitStages:
  - RequestReceived
  matchPolicy: "Any"
  rules:
  - resources: ["*"]
    # (Would need custom logic here to detect cross-namespace)

Setup separate audit log per tenant để easier analysis.

Multi-Tenancy Design Patterns

Pattern 1: Isolated Namespace Per Team

Cluster
├── namespace: team-backend
│   ├── ServiceAccount: team-backend-admin
│   ├── RBAC: can manage resources trong team-backend
├── namespace: team-frontend
│   ├── ServiceAccount: team-frontend-admin
│   ├── RBAC: can manage resources trong team-frontend
└── namespace: shared-services
    ├── ServiceAccount: shared-svc (read-only)

Each team separate RBAC, cannot interfere với lainnya.

Pattern 2: Shared Cluster, Tenant Quotas

yaml

# Tenant A ResourceQuota
apiVersion: v1
kind: ResourceQuota
metadata:
  namespace: tenant-a
  name: tenant-a-quota
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    pods: "100"

# Tenant B ResourceQuota
apiVersion: v1
kind: ResourceQuota
metadata:
  namespace: tenant-b
  name: tenant-b-quota
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    pods: "100"

Combined với namespace RBAC: tenants share hardware nhưng isolated via namespaces + quotas.

Pattern 3: Hub-and-Spoke (Shared Services)

┌─────────────────────┐
│   kube-system       │  ← Shared services (logging, monitoring)
│  ServiceAccounts    │  ← Carefully scoped
└─────────────────────┘
     ↑      ↑      ↑
     │      │      │
┌────┴─┐ ┌──┴──┐ ┌─┴─────┐
│tenant│ │tenant│ │tenant │
│  -a  │ │  -b  │ │  -c   │
└──────┘ └──────┘ └───────┘

Central services (logging, monitoring) trong kube-system; tenant namespaces grant read-only access via RoleBindings.

Operational Patterns

Onboarding New Tenant

bash

# 1. Create namespace
kubectl create namespace tenant-new

# 2. Label để identification
kubectl label namespace tenant-new \
  tenant=tenant-new \
  tenant-admin-contact=admin@tenant-new.com

# 3. Create tenant SA
kubectl create sa tenant-admin -n tenant-new

# 4. Create tenant-admin role
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: tenant-new
  name: tenant-admin
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
EOF

# 5. Bind role
kubectl create rolebinding admin-binding \
  -n tenant-new \
  --role=tenant-admin \
  --serviceaccount=tenant-new:tenant-admin

# 6. Create service account token
# (for external RBAC systems)

# 7. Grant cross-namespace access if needed
# (e.g., for shared logging service)
kubectl create rolebinding logging-access \
  -n tenant-new \
  --clusterrole=log-reader \
  --serviceaccount=kube-system:logger

Repeatable onboarding prevent misconfiguration.

Offboarding Tenant

bash

# 1. Verify all resources cleaned
kubectl delete namespace tenant-to-remove
# (cascade deletion removes all resources)

# 2. Remove any ClusterRoleBindings
kubectl delete clusterrolebinding \
  -l tenant=tenant-to-remove

# 3. Verify removed từ audit
grep "tenant-to-remove" /var/log/audit.log | tail -20

Troubleshooting Multi-Tenant RBAC

Issue 1: Tenant Cross-Access

bash

# Alert: Tenant-a accessed tenant-b resource
# Check audit log
grep "system:serviceaccount:tenant-a" /var/log/audit.log | \
  jq 'select(.objectRef.namespace == "tenant-b")'

# Likely causes:
# 1. Overly broad ClusterRoleBinding
# 2. Shared SA credentials
# 3. Impersonation permission incorrectly granted

# Fix: Review and narrow RBAC

Issue 2: Tenant Cannot Access Shared Service

bash

# Tenant A needs read logs nhưng denied
# Check: Does kube-system:logger have RoleBinding trong tenant-a?
kubectl get rolebinding -n tenant-a -o wide | grep logger

# If missing, create:
kubectl create rolebinding logger-access \
  -n tenant-a \
  --clusterrole=log-reader \
  --serviceaccount=kube-system:logger

Issue 3: Escalation Attempts Detected

bash

# Alert: Tenant-a tried create ClusterRole
# Check if attack successful:
kubectl auth can-i create clusterroles --as=system:serviceaccount:tenant-a:app

# Should be NO (denied)
# If YES, immediate remediation required:
# 1. Identify RBAC granting permission
# 2. Remove escalation permission
# 3. Audit để damage

Summary

RBAC để multi-tenancy:

Namespace boundaries — primary isolation mechanism
RoleBinding scoping — permissions restricted per namespace
Explicit cross-namespace — shared services via explicit RoleBindings
Escape prevention — limit dangerous verbs (create roles, impersonate)
Audit tracking — per-tenant activity fully logged

Safe multi-tenancy requires:

Strict namespace-per-tenant RBAC
Explicit cross-namespace permissions
Regular audit review
Defense in depth (RBAC + network policy + admission)
Repeatable onboarding/offboarding procedures

RBAC for Multi-Tenancy & Namespace Isolation ​

Vì Sao Quantum Trọng ​

Multi-Tenancy Model ​

Namespace-Per-Tenant Isolation ​

Implementation: RoleBinding Scoping ​

Cross-Namespace Access Patterns ​

Scenario 1: Shared Service (Logs) ​

Scenario 2: Monitoring Access ​

Escape Prevention ​

Threat Model ​

Prevention Mechanisms ​

Audit Per Tenant ​

Tracking Tenant Activity ​

Per-Tenant Audit Log Filtering ​

Tenant-Specific Audit Policies ​

Multi-Tenancy Design Patterns ​

Pattern 1: Isolated Namespace Per Team ​

Pattern 2: Shared Cluster, Tenant Quotas ​

Pattern 3: Hub-and-Spoke (Shared Services) ​

Operational Patterns ​

Onboarding New Tenant ​

Offboarding Tenant ​

Troubleshooting Multi-Tenant RBAC ​

Issue 1: Tenant Cross-Access ​

Issue 2: Tenant Cannot Access Shared Service ​

Issue 3: Escalation Attempts Detected ​

Summary ​

Tham Khảo ​

RBAC for Multi-Tenancy & Namespace Isolation

Vì Sao Quantum Trọng

Multi-Tenancy Model

Namespace-Per-Tenant Isolation

Implementation: RoleBinding Scoping

Cross-Namespace Access Patterns

Scenario 1: Shared Service (Logs)

Scenario 2: Monitoring Access

Escape Prevention

Threat Model

Prevention Mechanisms

Audit Per Tenant

Tracking Tenant Activity

Per-Tenant Audit Log Filtering

Tenant-Specific Audit Policies

Multi-Tenancy Design Patterns

Pattern 1: Isolated Namespace Per Team

Pattern 2: Shared Cluster, Tenant Quotas

Pattern 3: Hub-and-Spoke (Shared Services)

Operational Patterns

Onboarding New Tenant

Offboarding Tenant

Troubleshooting Multi-Tenant RBAC

Issue 1: Tenant Cross-Access

Issue 2: Tenant Cannot Access Shared Service

Issue 3: Escalation Attempts Detected

Summary

Tham Khảo