My Awesome Project

Appearance

Config Controller — Quản Lý GCP Resources Qua Kubernetes CRDs ​

Tại Sao Quan Trọng Trong Production ​

Infrastructure-as-Code (IaC) có hai approaches chính:

  1. Terraform/CloudFormation: Dedicated DSL, separate state management, separate tool lifecycle
  2. Kubernetes as IaC: Manage infrastructure qua Kubernetes manifests, RBAC, Config Sync, GitOps

Config Controller là Google's answer cho approach thứ hai — fully-managed GKE cluster cài sẵn Config Connector (KCC) cho phép quản lý Google Cloud resources (GCS buckets, Cloud SQL, Pub/Sub topics, Service Accounts, v.v.) bằng Kubernetes CRDs.

Lợi ích:

  • Single source of truth: Infra definitions trong Git, apply qua Config Sync giống như application code
  • Unified RBAC: Ai được tạo GCS bucket = Kubernetes RBAC roles, không cần separate IAM management
  • Declarative: Dùng declarative manifest thay vì imperative scripts
  • GitOps workflow: Review infra changes qua PRs, audit trail từ Git commits

Config Controller vs Standalone KCC ​

Config Connector (KCC) là open-source Kubernetes addon cho phép CRDs quản lý GCP resources.

Config Controller là GCP's fully-managed service:

  • Provisioned GKE cluster (Autopilot, không quản lý nodes)
  • KCC pre-installed
  • Built-in Config Sync
  • Connected vào fleet (optional)
  • Simplified operations
Standalone KCC:
  You provision GKE cluster → install KCC addon → manage yourself

Config Controller:
  Google provision GKE cluster → KCC pre-installed → you provision resources

Config Controller là recommended approach cho production vì:

  • No node management burden
  • Automatic KCC updates
  • Built-in high availability
  • Direct integration with fleet
  • Audit logging enabled by default

Internal Architecture — CRD to GCP Resource Reconciliation ​

Config Connector Custom Resources ​

KCC define CRDs cho mỗi GCP resource type. Ví dụ: StorageBucket CRD cho GCS bucket:

yaml
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  name: my-bucket
  namespace: default
  annotations:
    cnrm.cloud.google.com/project-id: my-project
spec:
  location: US
  versioning:
    enabled: true
  lifecycleRule:
  - action:
      type: Delete
    condition:
      age: 30
  - action:
      type: SetStorageClass
      storageClass: ARCHIVE
    condition:
      numNewerVersions: 3

Khi resource apply, KCC controller:

  1. Validate spec — StorageBucket spec dùng custom validation rules
  2. Call GCP APIs — storage.googleapis.com CreateBucket
  3. Set ownership — GCP resource được tagged với annotations cho traceability
  4. Update status — .status.selfLink, .status.conditions, GCP resource ID
  5. Watch for external changes — nếu bucket bị modified vào GCP console, KCC detect drift

Example: Service Account Management ​

yaml
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMServiceAccount
metadata:
  name: payments-service-account
  namespace: default
  annotations:
    cnrm.cloud.google.com/project-id: my-project
spec:
  displayName: "Payments Service Account"
---
# Grant payments-service-account role editor trên GCS bucket
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucketIAMBinding
metadata:
  name: bucket-payments-access
  namespace: default
spec:
  bucketRef:
    name: my-bucket
  role: roles/storage.objectViewer
  members:
  - serviceAccount:payments-service-account@my-project.iam.gserviceaccount.com

Khi apply:

  1. IAMServiceAccount controller create GCP Service Account
  2. StorageBucketIAMBinding controller apply IAM binding tới bucket
  3. Nếu delete resource từ Kubernetes, GCP resource default được delete (tùy theo resource type)

Deletion Protection ​

Xóa Kubernetes resource default xóa GCP resource — nguy hiểm. KCC support deletion protection:

yaml
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  name: prod-bucket
  annotations:
    cnrm.cloud.google.com/deletion-policy: abandon
spec:
  ...

Deletion policies:

  • delete (default): Delete GCP resource khi KCC resource deleted
  • abandon: Keep GCP resource, remove từ KCC management
  • retain: Prevent deletion (raise error nếu try delete)

For production resources: luôn set deletion-policy: abandon để prevent accidental deletion.

Khác với Terraform prevent_destroy, abandon cho phép resource tồn tại independent sau khi delete Kubernetes resource. Tức là rollback không impossible — GCP resource vẫn sống, chỉ KCC quản lý bị remove.

Dependency Management ​

KCC handle resource dependencies via references:

yaml
apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLInstance
metadata:
  name: prod-database
  namespace: default
spec:
  databaseVersion: MYSQL_8_0
  settings:
    backupConfiguration:
      enabled: true
---
apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLDatabase
metadata:
  name: payments-db
  namespace: default
spec:
  instanceRef:
    name: prod-database  # Reference to SQLInstance above
  charset: utf8mb4
---
apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLUser
metadata:
  name: payments-user
  namespace: default
spec:
  instanceRef:
    name: prod-database
  password:
    valueFrom:
      secretKeyRef:
        name: db-password
        key: password

KCC controller order applies: SQLInstance trước SQLDatabase trước SQLUser. Đồng thời, nếu try delete SQLInstance nhưng SQLDatabase vẫn exist (tham chiếu), delete sẽ fail.

Config Controller + Config Sync = Meta-IaC ​

Config Controller + Config Sync tạo nên meta-IaC pattern: dùng Git → Config Sync → Config Connector để manage GCP infrastructure.

Git Repository: fleet-infrastructure/
├── namespaces/
│   ├── default/
│   │   ├── storage-bucket.yaml          # GCS bucket
│   │   ├── database.yaml                # Cloud SQL
│   │   └── service-account.yaml         # Service Account
│   └── platform/
│       └── monitoring.yaml              # Monitoring setup
├── kustomization.yaml
└── RootSync (deployed on Config Controller cluster)

Workflow:

  1. Platform engineer write GCP resource manifests
  2. Commit đến Git
  3. Config Sync pull từ Git, apply đến Config Controller
  4. Config Connector create GCP resources
  5. Audit trail từ Git commits

Tất cả infrastructure versions tracked trong Git. Rollback = revert commit + config sync pull.

Resource Dependency Resolution ​

KCC handle transitive dependencies — nếu StorageBucket depend trên Network, Network depend trên Subnetwork, KCC tạo trong order đúng.

Nhưng circular dependencies === error:

yaml
# ERROR: A depends on B, B depends on A
apiVersion: ...
kind: ServiceA
spec:
  refB:
    name: service-b
---
apiVersion: ...
kind: ServiceB
spec:
  refA:
    name: service-a

Apply sẽ fail. Designer phải break dependency cycle.

Constraints Và Limitations ​

Not all GCP services supported: KCC support ~140 resource types, không phải mọi GCP service. Phải check KCC documentation xem resource type supported trước dùng. Unsupported resource phải quản lý qua Terraform hoặc GCP console.

Eventual consistency: Khi update resource spec, change apply asynchronously — không instant. Status field update khi change complete.

GCP API rate limiting: Nếu apply hàng trăm resources cùng lúc, GCP API rate limiting có thể kick in. KCC retry theo exponential backoff, nhưng large-scale resource provisioning cần patient waiting.

Secret management: Sensitive data (database passwords, API keys) phải store trong Kubernetes Secrets. Không làm encryption at rest cho Secrets — phải setup Workload Identity hoặc managed KMS để protect secrets.

Resource recreation on spec change: Một số GCP resource properties immutable — change property yêu cầu delete + recreate. KCC xử lý này, nhưng user phải aware downtime.

Multi-project resources: Config Connector có thể manage resources trong multiple GCP projects (via annotations), nhưng RBAC ở project level. Nếu create resources ở project A và project B, need IAM permissions trên cả hai.

Example: Platform Team Setup ​

Typical Config Controller usage cho platform team:

Platform Team:
├── Config Controller cluster (GKE Autopilot)
│   └── Config Connector enabled
├── Fleet membership (member của platform fleet)
├── Config Sync RootSync pointing to fleet-infrastructure Git repo
└── All infra defined as CRDs

Namespaces in Config Controller:
├── default/
│   ├── GCP project VPCs
│   ├── Service accounts
│   └── IAM bindings
├── monitoring/
│   ├── Cloud Monitoring alert policies
│   └── Log sinks
└── networking/
    ├── Cloud Armor policies
    └── Private Service Connections

When new GKE cluster needs provisioning:

  1. Platform engineer create GKECluster CRD in Config Controller Git repo
  2. Commit + push
  3. Config Sync sync lên Config Controller
  4. Config Connector create GKE cluster resource
  5. Cluster ready, register vào fleet
  6. Config Sync distribute workloads từ fleet-workload-config repo

Entire provisioning tracked in Git, reproducible, audited.

References ​