Fleet RBAC & Fleet Observability — Governance Xuyên Cluster

Tại Sao Quan Trọng Trong Production

Kubernetes RBAC là per-cluster — role bindings được define riêng trên từng cluster. Khi fleet có 50 clusters, setup RBAC identically xuyên tất cả là operational burden lớn. Platform teams cần mechanism để:

1. Define RBAC policy centrally
2. Distribute policies đến tất cả fleet members
3. Ensure namespaces có consistent permission model xuyên clusters

Fleet Observability là flip side của governance — không chỉ control ai làm gì, mà cần thấy những gì đang xảy ra xuyên clusters. Làm sao biết có pod nào crash ở cluster B? Có service latency increase ở region C? Làm sao troubleshoot issues khi bạn có 100+ clusters?

Fleet IAM → Kubernetes RBAC Mapping

Fleet IAM Roles

Fleet cung cấp tập hợp Fleet-specific IAM roles định nghĩa quyền ở fleet level:

roles/gkehub.admin                    → Admin tất cả fleet features
roles/gkehub.editor                   → Edit fleet feature configs
roles/gkehub.viewer                   → Read-only access
roles/gkehub.scopeViewer              → Read team scopes
roles/gkehub.scopeEditor              → Manage team scopes

Ví dụ: grant engineer membership "fleet viewer" permission:

gcloud projects add-iam-policy-binding FLEET_PROJECT \
  --member=user:engineer@company.com \
  --role=roles/gkehub.viewer

Điều này allow engineer xem membership status, fleet configuration, nhưng không modify.

Fleet RBAC Binding

Để grant permissions trên Kubernetes (namespace access, deploy apps), Fleet RBAC Binding map GCP IAM users/groups đến Kubernetes RBAC:

# Fleet RBAC Binding (chạy trên fleet hub project)
apiVersion: rbac.fleetscope.cnrm.cloud.google.com/v1beta1
kind: FleetRBACRoleBinding
metadata:
  name: payments-team-admin
  namespace: fleet-rbacbindings
spec:
  scope: team-payments  # Reference team scope
  role: admin          # Kubernetes ClusterRole
  principals:
  - gcloud:group:payments-engineers@company.com

Khi binding apply:

1. Fleet controller identify team scope team-payments
2. Query namespaces belong đến team scope
3. Create RoleBinding/ClusterRoleBinding trong member clusters mapping:
   • GCP group payments-engineers@company.com → Kubernetes ServiceAccount dùng OIDC assertion từ group membership → ClusterRole admin trong namespaces của team

# Automatically created trong member cluster
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-team-admin
  namespace: payments
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admin
subjects:
- kind: User
  name: "payments-engineers@company.com"  # OIDC identity
  apiGroup: rbac.authorization.k8s.io

Khi engineer từ group payments-engineers@company.com authenticate với cluster (via Connect Gateway), Google Cloud trace group membership, issue token, Kubernetes RBAC evaluate token claim.

Team Scopes — Narrowed Permissions

Team scope là collection của namespaces + member clusters mà team có access:

apiVersion: gkehub.cnrm.cloud.google.com/v1beta1
kind: GkeHubScope
metadata:
  name: team-payments
spec:
  displayName: "Payments Team"
  namespaceLabels:
    matchLabels:
      scope: payments    # Namespaces với label scope=payments
  clusters:
  - name: prod-us-east1
  - name: prod-eu-west1

Team scope approach:

• Platform admin define scopes
• Grant team access qua scope RBAC binding (not cluster-level RBAC)
• Team có permissions chỉ trong namespace+cluster combo thuộc scope

Này giải quyết bài toán: "Payments team có full admin ở namespaces payments/payments-infra trên prod clusters, nhưng không ở auth/security namespaces."

Với scope, RBAC binding tự động create bindings cho tất cả namespaces trong scope xuyên tất cả member clusters.

OIDC-Based Authentication

Fleet RBAC dùng OIDC-based authentication:

1. Engineer authenticate with Google Cloud (gcloud auth login)
2. Google issue OIDC token (Bearer token) với claims:
   • sub: user identity
   • groups: list of GCP groups user belongs
3. Engineer run kubectl sau gcloud authentication
4. kubectl send OIDC token đến cluster
5. Kubernetes API server validate token với Google OIDC provider
6. Create user principal từ OIDC claims

Advantage: không cần kubeconfig files — authentication delegate đến Google Cloud identity.

Disadvantage: engineer must authenticate với Google Cloud trước kubectl work. Nếu Google Cloud authentication expire, kubectl commands fail.

Fleet Observability

Architecture

Fleet Observability là feature collect metrics, logs, traces từ member clusters vào Fleet-level dashboards trên Cloud Monitoring.

Cluster A (GKE with Monitoring enabled)
├── Metrics exporter (Prometheus-compatible)
├── Logs router (Cloud Logging agent)
└── Traces collector (Google Cloud Trace)
    ↓
    Cloud Monitoring Project (Fleet Host Project)
├── Cloud Monitoring dashboards
├── Alerting policies
└── Log aggregation
    ↓ (Backwards link)
Cluster B
Cluster C

Fleet Observability aggregates metrics từ tất cả clusters vào single project. Không cần create dashboards trong individual cluster projects — central observability cho toàn fleet.

Metrics Collection

Khi Fleet Observability enable:

1. GKE Monitoring agent deploy trên member clusters (as DaemonSet)
2. Agent scrape metrics từ:
   • Kubelet (node CPU, memory, disk)
   • API server (request latencies, error rates)
   • Container runtime (container CPU, memory)
   • User-defined Prometheus metrics (if enabled)
3. Metrics ship đến fleet host project's Cloud Monitoring
4. Metrics label với cluster identity:
   • resource.label.cluster_name: cluster identifier
   • resource.label.namespace_name: namespace
   • resource.label.pod_name: pod name

Cross-Cluster Dashboards

Điều tạo giá trị là cross-cluster queries: aggregate metrics xuyên clusters, nhìn pattern xuyên fleet.

# Example Cloud Monitoring query (MQL syntax)
# Find P99 latency của API server xuyên tất cả clusters
fetch k8s_cluster
| metric 'kubernetes.io/api_server/request_latencies'
| group_by [resource.cluster_name], [percentile_agg(value, 99)]

Query này return metric xuyên tất cả fleet members, grouped by cluster. Thay vì click từng cluster dashboard, nhìn pattern toàn fleet.

Log Aggregation

Logs từ member clusters forward đến Cloud Logging project của fleet host. Logs labeled với:

• resource.labels.cluster_name
• resource.labels.namespace_name
• jsonPayload.pod_name, jsonPayload.container_name

Log query:

resource.type="k8s_container"
resource.labels.cluster_name=("cluster-prod-us-east1" OR "cluster-prod-eu-west1")
resource.labels.namespace_name="payments"
severity="ERROR"

Câu query này find ERROR logs từ payments namespaces trên tất cả production clusters.

Alerting Across Fleet

Alert policy dùng cross-cluster metrics trigger alerts based on fleet-wide conditions:

# Alert nếu bất kỳ cluster nào có API server latency P99 > 1s
apiVersion: monitoring.googleapis.com/v3
kind: AlertPolicy
metadata:
  name: high-api-latency-fleet
spec:
  displayName: "High API Server Latency"
  conditions:
  - displayName: "P99 latency > 1s"
    conditionThreshold:
      filter: 'metric.type="kubernetes.io/api_server/request_latencies"'
      comparison: COMPARISON_GT
      thresholdValue: 1000  # milliseconds
      aggregations:
      - alignmentPeriod: 60s
        perSeriesAligner: ALIGN_PERCENTILE_99
      - groupByFields:
        - resource.cluster_name

Alert trigger nếu any cluster violate condition. Nếu 50 clusters và một cluster spike, alert fires.

Constraints Và Operational Considerations

RBAC Propagation Latency: Fleet RBAC bindings sync asynchronously. Khi tạo new FleetRBACRoleBinding, có delay trước khi RoleBindings created trên member clusters (thường < 5 minutes, đôi khi lâu hơn tuỳ cluster count).

Scope-to-Namespace Mapping: Team scope match namespaces by labels. Nếu namespace created mà label chưa được add, nó không part của team scope. Automation cần ensure namespace labels set khi tạo namespace (hoặc Config Sync apply labels sau).

Observability not real-time: Metrics collection có latency — point-in-time data có thể 1–2 phút behind actual state. Để debugging real-time issues, vẫn cần direct cluster access hoặc logs.

Metrics Storage Cost: Fleet Observability aggregate metrics vào single project. Số lượng metrics series (cluster × metric type × label combinations) có thể huge. Pricing based on volume — cost có thể surprising lớn cho huge fleets.

Log Retention: Cloud Logging retention policy apply tới tất cả members. Không thể per-cluster log retention — hoặc tất cả clusters giữ logs 30 days, hoặc tất cả giữ 90 days.

OIDC Token Expiry: Kubernetes OIDC token expiry shorter than session lifetime. Nếu engineer idle > 1 hour, kubectl token expired. kubectl auto-refresh bằng Google Cloud SDK, nhưng nếu SDK expire, engineer phải gcloud auth login lại.

Best Practices

Fleet RBAC:

• Use team scopes instead of cluster-level RBAC — reduce policy duplication
• Regular audit FleetRBACRoleBindings — ensure granularity aligned với team organization
• Test scope membership after creating/modifying scopes — verify correct namespaces included

Fleet Observability:

• Define dashboards cho common troubleshooting scenarios (API latency spike, pod crash, disk pressure)
• Set up alerts cho critical metrics before fire-fighting — data-driven decisions vs reactive
• Export metrics để long-term trend analysis (do BI tools) — Cloud Monitoring retention limited

References

• Fleet RBAC
• Team Scopes
• Fleet Observability
• Cloud Monitoring MQL