Configuration Backup: GitOps, Terraform State, Resource Manifests

Tại Sao Điều Này Quan Trọng

Disaster recovery không chỉ là data backup. Nó cũng bao gồm infrastructure configuration — Kubernetes manifests, Terraform modules, secrets, network policies, load balancer rules. Nếu mất configuration, thậm chí có data backup cũng không thể rebuild service.

Ví dụ:

Data backup OK, Kubernetes manifests gone → không biết container image nào, resource limit nào, ngân sách scaling nào
Terraform state lost → không biết cloud infrastructure là gì, không thể update/destroy resources safely
Secrets không backup → không thể authenticate với external services (databases, APIs)

Internal Model: Configuration Management in DR

Three Types of Configuration

1. Infrastructure Configuration (Terraform)

resources:
├─ VPC networks
├─ Subnet CIDR ranges
├─ Cloud SQL instances
├─ Load balancers
├─ GKE clusters (node pools, addons)
└─ IAM roles, permissions

Stored in: Git repository + Terraform state file
Critical for: Recreating infrastructure from scratch

2. Application Configuration (Kubernetes Manifests + GitOps)

resources:
├─ Deployments (image, replicas, resources)
├─ ConfigMaps (application configuration)
├─ Services (networking)
├─ Ingress (routing)
├─ StatefulSets (persistent workloads)
└─ Policies (network policies, RBAC)

Stored in: Git repository + Kubernetes etcd
Critical for: Recreating applications and workloads

3. Secrets (Credentials, Keys)

resources:
├─ Database passwords
├─ API keys
├─ TLS certificates
├─ SSH keys
└─ Encryption keys

Stored in: Google Secret Manager + sealed-secrets (in Git)
Critical for: Authentication and encryption

GitOps as Source-of-Truth

Principle: All infrastructure and application state defined in Git. Git is single source-of-truth.

Git repository (GitHub, GitLab)
├─ terraform/
│  ├─ main.tf (GKE cluster definition)
│  ├─ networking.tf (VPC, subnets)
│  └─ cloud-sql.tf (database definition)
├─ k8s/
│  ├─ deployments.yaml
│  ├─ services.yaml
│  └─ configmaps.yaml
└─ secrets/ (sealed secrets)
   └─ production-secrets.yaml.enc

GitOps Operator (ArgoCD, Flux):
├─ Watches Git repository
├─ Detects changes
├─ Applies to target infrastructure
└─ Ensures infrastructure matches Git state

Disaster recovery flow:

Step 1: Cluster destroyed
Step 2: Create new cluster (from Terraform in Git)
Step 3: Deploy GitOps operator
Step 4: GitOps operator pulls from Git
Step 5: All applications auto-deployed to new cluster
Step 6: Recovery complete

Advantage: Recovery is reproducible and automated (not manual, error-prone).

Terraform State File: Critical But Often Forgotten

Terraform state file tracks what resources exist:

hcl

# terraform/main.tf
resource "google_container_cluster" "gke_cluster" {
  name = "production-cluster"
  ...
}

# After terraform apply:
# State file captures:
# ├─ Cluster ID
# ├─ Node pool configuration
# ├─ Cluster version
# └─ Other metadata

If state file lost:
├─ Terraform forgets about resources
├─ Next `terraform apply` tries to create NEW resources
├─ Result: duplicate resources or import errors

State file location:

Option 1: Local (dangerous)
├─ terraform.tfstate on engineer's machine
├─ If laptop stolen → state gone
├─ If engineer leaves → nobody knows state location

Option 2: Remote state (GCS bucket, recommended)
├─ terraform.tfstate in gs://terraform-state-bucket/
├─ Multiple engineers can share
├─ Backed up automatically
└─ Can be replicated cross-region

Protecting Terraform state:

GCS bucket with state file:
├─ Encryption: Google-managed or customer-managed keys
├─ Versioning: enabled (recover deleted state)
├─ Backup: cross-region bucket copy
├─ Access control: only team can read/write
└─ Audit logging: track who accessed state

In Terraform code:
backend "gcs" {
  bucket = "terraform-state-production"
  prefix = "prod"
  encryption_key = "<GCS bucket encryption key>"
}

Secret Management in DR Context

Problem: Secrets Cannot Be In Git

Passwords, API keys must not be committed to Git (security risk).

Solution: Store secrets separately, reference in application.

Pattern 1: Google Secret Manager

Architecture:

Git repository:
├─ Contains: manifests, Terraform code, non-secret config
└─ Does NOT contain: passwords, keys

Google Secret Manager:
├─ Contains: passwords, API keys
├─ Access control: IAM roles
└─ Audit logging: who accessed

Application startup:
1. Pod starts
2. Workload Identity: authenticates to Google Cloud
3. Secret Manager API: fetch secrets
4. Application uses secrets in memory (never in logs)

Disaster recovery:

Cluster recreated from Git + Terraform.
GitOps deploys applications.
Application starts, fetches secrets from Secret Manager.
Recovery complete.

Note: Secret Manager itself is multi-region (Google-managed redundancy).

Advantage: Secrets not in Git, easier to rotate.

Pattern 2: Sealed Secrets (Kubernetes Native)

Architecture:

Git repository:
├─ secrets/database-password.yaml.enc
│  └─ Encrypted with sealing key (public key)
├─ sealing-key.pub (public key, safe to commit)
└─ sealing-key.key (private key, NOT committed)

Private sealing key:
└─ Stored in Secret Manager or HSM
└─ Downloaded during cluster setup
└─ Used to decrypt sealed secrets

Application:
1. GitOps operator deploys sealed secret
2. Sealed secret controller (running in cluster) decrypts
3. Creates native Kubernetes Secret
4. Application uses Secret

Disaster recovery:

Cluster recreated:
├─ Install sealed-secrets controller
├─ Fetch private sealing key from Secret Manager
├─ Mount key to sealed-secrets pod
├─ GitOps deploys encrypted secrets
├─ Controller decrypts and creates native secrets
└─ Application starts

Critical: Sealing key must be in Secret Manager (cross-region protected).

Advantage: Secrets as Git-managed objects (can version control, diff encrypted secrets).

Backup Strategy for Configuration

Strategy 1: Git As Primary Backup

If Git repository properly maintained:

Git repository (GitHub, GitLab):
├─ All infrastructure code
├─ All application manifests
├─ Sealed secret manifests (encrypted)
└─ History: full version control

GitHub backup:
├─ GitHub Enterprise → redundant, managed by GitHub
├─ Automatic retention
└─ Can push to backup organization if needed

Restore procedure:
1. Clone Git repository
2. Run Terraform: infrastructure rebuilt
3. Deploy GitOps operator
4. GitOps pulls manifests, deploys to cluster
5. Secrets decrypted from Secret Manager
6. Recovery complete

Advantage: Simple, already version-controlled, minimal additional work.

Risk: If Git provider hacked or deleted, lose everything.

Strategy 2: Additional Backup Repository

For critical systems:

Primary Git (GitHub):
├─ Active development
├─ CI/CD integration

Backup Git (GitLab, Gitea, self-hosted):
├─ Mirror of primary
├─ Read-only
├─ Updated hourly/daily
└─ Isolated infrastructure (different provider/region)

Backup procedure:
1. GitHub webhook triggers on every push
2. Push to backup repository
3. Or: scheduled sync job (daily)

Advantage: If primary Git provider unavailable, backup available.

Strategy 3: Audit Backup of Actual Resource State

Even with Git-based IaC, actual cluster might drift:

Git says: Deployment has 3 replicas
Actual cluster: Deployment has 5 replicas (manual scale)

Backup should capture actual state:
├─ Export all Kubernetes resources (kubectl get all -A -o yaml)
├─ Export GCP resources (gcloud commands, Terraform import)
└─ Store in GCS or separate Git branch

Procedure (daily):

bash

# Export Kubernetes resources
kubectl get all -A -o yaml > k8s-backup-$(date).yaml
gsutil cp k8s-backup-*.yaml gs://audit-backup-bucket/

# Export GCP resources (if not already in Terraform)
gcloud compute instances list --format=json > gcp-resources-$(date).json
gsutil cp gcp-resources-*.json gs://audit-backup-bucket/

Use case: Detect drift, validate Git representation matches reality.

Terraform State Replication

State File Replication Strategy

Primary state bucket (us-central1):
├─ gs://terraform-state-production
├─ Used by CI/CD (Terraform Cloud, Cloud Build)
├─ Versioning: enabled

Backup state bucket (multi-region):
├─ gs://terraform-state-production-backup
├─ Copy of primary (daily via Transfer Service)
├─ Read-only (prevent accidental overwrites)
└─ In different project (isolation)

Automated copy:

bash

# Daily Cloud Scheduler job
# Copies state file to backup bucket

gsutil -m cp \
  gs://terraform-state-production/* \
  gs://terraform-state-production-backup/

Recovery:

If primary state bucket lost:
1. Copy from backup: gsutil cp gs://terraform-state-production-backup/* gs://terraform-state-production-restored/
2. Update Terraform backend to restored bucket
3. Verify state (terraform state list, terraform state show)
4. Continue operations

Or: import from actual infrastructure

Validation: Configuration Recovery Testing

Test 1: Git Checkout and Terraform Dry-Run

Monthly:

bash

# Clone repository
git clone https://github.com/company/infrastructure.git

# Run terraform plan (no actual changes)
cd infrastructure/terraform
terraform plan -detailed-exitcode

# Verify: plan shows no unexpected changes (drift detection)
# If drift found: reconcile with Git or update Terraform

Test 2: Configuration Export and Comparison

Quarterly:

bash

# Export actual cluster state
kubectl get all -A -o yaml > actual-k8s.yaml

# Generate expected state from GitOps
kubectl apply -f k8s/ --dry-run=client -o yaml > expected-k8s.yaml

# Diff
diff expected-k8s.yaml actual-k8s.yaml

# Review differences:
# - Acceptable (auto-added fields): continue
# - Drift (manual changes): decide to apply or update Git

Test 3: Disaster Recovery Drill (Quarterly)

Full recovery test:

1. Create temporary project
2. Run Terraform: create infrastructure from scratch
3. Deploy GitOps operator
4. Fetch secrets from Secret Manager
5. Verify all applications deployed
6. Run smoke tests
7. Delete temporary infrastructure

Measure: actual recovery time vs RTO target

Common Mistakes

Mistake 1: "All Config in Git, Nothing Else Needed"

Problem:

Git repository lost/deleted
Secret Manager keys lost
Terraform state file lost (not replicated)

Recovery incomplete.

Mitigation: Multiple backups (Git + state file backup + Secret Manager replication).

Mistake 2: "Manual Changes Faster Than Git Workflow"

Problem:

Engineer manually scales deployment (not in Git)
Git and actual state diverge
Recovery from Git restores old state (loses manual scaling)

Mitigation: Enforce GitOps (block manual changes). Use ArgoCD "auto-sync" to force convergence.

Mistake 3: "Sealed Secrets Key Lost in Cluster"

Problem:

Sealing key only on Kubernetes (not backed up elsewhere)
Cluster destroyed
Sealed secrets cannot be decrypted

Recovery impossible.

Mitigation: Sealing key must be in Secret Manager (separate from cluster).

Summary

Configuration backup is critical to DR:

Infrastructure config: Terraform code in Git + state file replicated
Application config: Kubernetes manifests in Git + GitOps operator
Secrets: Secret Manager (cross-region) + sealed keys

GitOps + IaC enables automated recovery (not manual, reproducible).

Configuration Backup: GitOps, Terraform State, Resource Manifests ​

Tại Sao Điều Này Quan Trọng ​

Internal Model: Configuration Management in DR ​

Three Types of Configuration ​

GitOps as Source-of-Truth ​

Terraform State File: Critical But Often Forgotten ​

Secret Management in DR Context ​

Problem: Secrets Cannot Be In Git ​

Pattern 1: Google Secret Manager ​

Pattern 2: Sealed Secrets (Kubernetes Native) ​

Backup Strategy for Configuration ​

Strategy 1: Git As Primary Backup ​

Strategy 2: Additional Backup Repository ​

Strategy 3: Audit Backup of Actual Resource State ​

Terraform State Replication ​

State File Replication Strategy ​

Validation: Configuration Recovery Testing ​

Test 1: Git Checkout and Terraform Dry-Run ​

Test 2: Configuration Export and Comparison ​

Test 3: Disaster Recovery Drill (Quarterly) ​

Common Mistakes ​

Mistake 1: "All Config in Git, Nothing Else Needed" ​

Mistake 2: "Manual Changes Faster Than Git Workflow" ​

Mistake 3: "Sealed Secrets Key Lost in Cluster" ​

Summary ​

References ​

Configuration Backup: GitOps, Terraform State, Resource Manifests

Tại Sao Điều Này Quan Trọng

Internal Model: Configuration Management in DR

Three Types of Configuration

GitOps as Source-of-Truth

Terraform State File: Critical But Often Forgotten

Secret Management in DR Context

Problem: Secrets Cannot Be In Git

Pattern 1: Google Secret Manager

Pattern 2: Sealed Secrets (Kubernetes Native)

Backup Strategy for Configuration

Strategy 1: Git As Primary Backup

Strategy 2: Additional Backup Repository

Strategy 3: Audit Backup of Actual Resource State

Terraform State Replication

State File Replication Strategy

Validation: Configuration Recovery Testing

Test 1: Git Checkout and Terraform Dry-Run

Test 2: Configuration Export and Comparison

Test 3: Disaster Recovery Drill (Quarterly)

Common Mistakes

Mistake 1: "All Config in Git, Nothing Else Needed"

Mistake 2: "Manual Changes Faster Than Git Workflow"

Mistake 3: "Sealed Secrets Key Lost in Cluster"

Summary

References