GKE Cost Allocation: Namespace-level Breakdown & Chargeback

Khi chạy GKE cluster dùng chung, bạn cần biết team A dùng bao nhiêu, team B dùng bao nhiêu. Cost allocation là dự đoán chi phí per namespace/team/application, từ đó chargeback hoặc inform optimization decisions.

Khó khăn: GCP billing không natively track per-namespace. Nó track per-project, per-VM, per-disk. Việc chiếu GKE cost xuống namespace level yêu cầu data enrichment qua labels, BigQuery analysis.

---

Cost Components in GKE

GKE cluster cost gồm:

1. Compute (GCE nodes):        60–70%
   - GCE VMs (master/worker)
   - Persistent disks (node OS)

2. Networking:                 10–15%
   - Load balancers (Ingress)
   - Cloud NAT (egress)
   - Network peering

3. Storage:                    10–20%
   - PersistentVolume (GCE PD, Filestore)
   - Cloud Storage (logs, artifacts)

4. Observability:              5–10%
   - Cloud Logging (container logs)
   - Cloud Monitoring (metrics)

5. Other (KMS, etc):           1–5%

Tracking Problem

• Compute cost: Tied to node pool. If pool shared, cannot attribute to single namespace
• Networking cost: Tied to VPC, load balancer. Hard to split per namespace
• Storage cost: Tied to PV, can be labeled per namespace. Easiest to track
• Observability cost: Log volume per namespace visible, metric volume per namespace visible

---

Cost Allocation Strategy

Strategy 1: Node Pool Segregation

Setup:

Cluster "prod":
  - node-pool-team-a: 5 VMs, labels: team=a
  - node-pool-team-b: 3 VMs, labels: team=b
  - node-pool-shared: 10 VMs (shared by multiple teams)

Cost attribution:

• Team A: Full cost of node-pool-team-a nodes + allocated share of node-pool-shared
• Team B: Full cost of node-pool-team-b nodes + allocated share of node-pool-shared
• Shared pool: Allocate by namespace request (CPU/memory)

Billing query:

SELECT
  node_pool,
  SUM(cost) as node_pool_cost
FROM `project.billing_dataset.gcp_billing_export_v1`
WHERE service.description = 'Compute Engine'
  AND labels.node_pool IN ('node-pool-team-a', 'node-pool-team-b')
GROUP BY node_pool;

Strategy 2: Namespace Cost Attribution (Advanced)

Input: Node pool cost + namespace CPU/memory allocation

Process:

For each namespace:
  namespace_cost = (
    (namespace_cpu_request / node_pool_cpu_total) × node_pool_cost +
    (namespace_memory_request / node_pool_memory_total) × node_pool_memory_cost +
    namespace_storage_cost +
    namespace_logging_cost +
    namespace_ingress_cost
  )

Implementation:

1. Extract node pool costs from billing export
2. Query Kubernetes API server: get namespace CPU/memory requests
3. BigQuery: join billing + K8s metrics, calculate attribution
4. Output: per-namespace cost report

---

Data Collection & Enrichment

Step 1: Label Nodes & Workloads

# Node pool config
gcloud container node-pools create team-a-pool \
  --cluster=my-cluster \
  --labels=team=a,cost-center=eng

# Pod labels (enforced via admission webhook)
apiVersion: v1
kind: Pod
metadata:
  labels:
    team: a
    app: my-app
    cost-center: engineering

Step 2: GCP Billing Labels

# Add billing labels to resources
gcloud compute instances add-labels $INSTANCE_ID \
  --labels=team=a,application=web-backend

Step 3: Kubernetes Metrics Export

Query Kubernetes for actual CPU/memory:

kubectl top pods -A --containers
# Output: namespace, pod, cpu, memory

Export to Cloud Monitoring or BigQuery:

Namespace    | Pod              | CPU (m) | Mem (Mi)
kube-system  | kube-dns         | 50      | 128
team-a       | web-backend-1    | 500     | 512
team-b       | api-server-1     | 300     | 256

---

BigQuery Cost Analysis

Query 1: Node Pool Costs

SELECT
  COALESCE(labels.node_pool, 'unknown') as node_pool,
  COALESCE(labels.team, 'unknown') as team,
  SUM(cost) as total_cost,
  COUNT(*) as line_items
FROM `project.billing_dataset.gcp_billing_export_v1`
WHERE service.description = 'Compute Engine'
  AND DATE(usage_start_time) >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
GROUP BY node_pool, team
ORDER BY total_cost DESC;

Query 2: Namespace CPU/Memory Attribution

-- Assume 'kube_namespace' table exists (exported from K8s)
-- and 'gcp_billing' table has node pool cost

WITH namespace_usage AS (
  SELECT
    namespace,
    SUM(cpu_millicores) as total_cpu,
    SUM(memory_bytes) as total_mem
  FROM `project.k8s_metrics.pod_metrics`
  WHERE DATE(timestamp) >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
  GROUP BY namespace
),

node_pool_costs AS (
  SELECT
    labels.node_pool,
    SUM(cost) as pool_cost
  FROM `project.billing_dataset.gcp_billing_export_v1`
  WHERE service.description = 'Compute Engine'
  GROUP BY node_pool
)

SELECT
  n.namespace,
  (n.total_cpu / SUM(n.total_cpu) OVER ()) * 
    (SELECT pool_cost FROM node_pool_costs WHERE node_pool = 'default') as allocated_cost
FROM namespace_usage n;

Query 3: Per-Namespace Total Cost

-- Combine compute + storage + logging
SELECT
  namespace,
  SUM(CASE 
    WHEN service.description = 'Compute Engine' 
      THEN cost * (SELECT ... allocation_factor ...)
    WHEN service.description = 'Cloud Storage'
      THEN cost
    WHEN service.description = 'Cloud Logging'
      THEN cost
  END) as total_cost
FROM `project.billing_dataset.gcp_billing_export_v1`
WHERE DATE(usage_start_time) >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
GROUP BY namespace
ORDER BY total_cost DESC;

---

Multi-Tenant Billing Models

Model 1: Show-Back (Awareness)

Approach: Calculate per-namespace cost, share with team, no actual charge.

Goal: Cost awareness, encourage optimization.

Pros: Simple, no chargeback overhead, builds awareness Cons: Teams ignore unless tied to incentive

Model 2: Charge-Back (Accounting)

Approach: Allocate namespace cost to team's P&L, via internal billing.

Implementation:

Team A monthly charge = namespace_cost_a
Team B monthly charge = namespace_cost_b
Shared cost = shared_pool_cost × utilization_ratio

Pros: Drives cost consciousness, fair allocation Cons: Complex accounting, requires governance

Model 3: Hybrid (Show-Back + Incentive)

Approach: Show-back cost + budget alert + quarterly reviews.

Implementation:

Q1 Baseline: Team A baseline cost = $10k
Q2 Budget: Team A allocated $10k (keep-it-same incentive)
Q3 Review: If spending $9k, return $1k budget credit
           If spending $12k, bill $2k overage

---

Namespace Segregation for Cost Control

Separate Namespaces by Team/Cost Center

# Team A workloads
apiVersion: v1
kind: Namespace
metadata:
  name: team-a
  labels:
    team: a
    cost-center: eng-a

# Team B workloads
apiVersion: v1
kind: Namespace
metadata:
  name: team-b
  labels:
    team: b
    cost-center: eng-b

# Shared services (monitored separately)
apiVersion: v1
kind: Namespace
metadata:
  name: platform
  labels:
    team: platform
    cost-center: platform-eng

Resource Quotas per Namespace

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "100"
    requests.memory: "200Gi"
    limits.cpu: "200"
    limits.memory: "400Gi"

This prevents one team from consuming unbounded resources.

---

Optimization via Cost Allocation

Pattern 1: Identify High-Cost Namespaces

SELECT
  namespace,
  total_cost,
  RANK() OVER (ORDER BY total_cost DESC) as rank
FROM namespace_costs
WHERE month = CURRENT_DATE()
ORDER BY total_cost DESC
LIMIT 10;

Action: Review top 3 namespaces for optimization opportunities.

Pattern 2: Cost Trend Analysis

SELECT
  DATE_TRUNC(date, MONTH) as month,
  namespace,
  SUM(cost) as monthly_cost,
  LAG(SUM(cost)) OVER (PARTITION BY namespace ORDER BY DATE_TRUNC(date, MONTH)) as prev_month_cost
FROM namespace_costs
GROUP BY month, namespace
HAVING month >= DATE_SUB(CURRENT_DATE(), INTERVAL 12 MONTH);

Action: If cost growing 10%+ MoM, investigate cause (new workload? no optimization?).

Pattern 3: Right-Sizing Recommendation

Map namespace CPU request vs actual usage:

SELECT
  namespace,
  SUM(requested_cpu) as requested_cpu,
  SUM(actual_cpu_p95) as actual_cpu_p95,
  ROUND(SUM(actual_cpu_p95) / SUM(requested_cpu), 2) as utilization_ratio
FROM (namespace request metrics)
GROUP BY namespace
HAVING utilization_ratio < 0.5;

Action: Reduce resource requests for namespaces with < 50% utilization.

---

Anti-Patterns

Anti-Pattern 1: No Cost Attribution

Mistake: Running shared cluster, no breakdown per team.

Impact: Teams unaware of true cost, no incentive to optimize.

Fix: Implement minimal show-back (query node pool cost, divide by team count).

Anti-Pattern 2: Incorrect Allocation Formula

Mistake: Allocate node cost by CPU request, but namespace uses 1% CPU, 50% memory.

Impact: Over-bill CPU-light, under-bill memory-heavy workloads.

Fix: Allocate by weighted score: cost = (cpu_ratio × cpu_weight) + (memory_ratio × memory_weight), tuned per workload.

Anti-Pattern 3: Ignoring Shared Services Cost

Mistake: Charge team A for pod cost, but ignore cluster ingress, logging cost (shared).

Impact: Team A thinks cost $2k, actual $5k (shared unaccounted).

Fix: Track shared service cost separately, allocate proportionally.

---

References

• GKE Cost Optimization Best Practices
• Cloud Billing Export Schema
• Kubernetes Resource Quotas