My Awesome Project

Appearance

Credential Rotation & Zero-Downtime Updates — SSL Certificates, CA Rotation, IP Rotation ​

Why Credential Rotation Matters ​

Credentials (SSL certs, CAs, endpoint IPs) must rotate periodically:

  • Expiration: Certificates expire after N years
  • Compromise: If credentials leaked
  • Compliance: Policy requires rotation
  • Updates: Control plane endpoint might change

SSL Certificate Rotation ​

Lifecycle ​

Certificate issued:
  Expiry: N years later (typically 1 year)
        ↓
  6-9 months before expiry: Start rotation
        ↓
  Issue new certificate with new key
        ↓
  Dual-cert mode: API Server presents both old & new
        ↓
  Clients start accepting new cert gradually
        ↓
  Old cert expires: Remove

Zero-Downtime Approach ​

GKE Certificate Rotation (automatic):

1. Issue new certificate
2. Both certs loaded into API Server
3. Clients can connect using either cert
4. New clients default to new cert
5. Old cert expires, remove

Result: No downtime, clients gradually upgrade

CA Rotation ​

Root CA used to sign API Server certificates

Old CA:
├─ Signs API Server cert v1
├─ Hardcoded in kubelet config
└─ Expiration: N years away

Rotation needed:
├─ Generate new CA
├─ Sign new API Server cert with new CA
├─ Update kubelet config with new CA
└─ Old CA still valid during transition

Control Plane IP Changes ​

Rare but possible:

Old: Cluster endpoint 35.238.123.45
     ├─ Hardcoded in kubeconfig
     └─ Hardcoded in node configs

New: Cluster endpoint 35.238.234.56
     ├─ Requires kubeconfig update
     ├─ Requires node config restart
     └─ During transition: connections might fail

GKE mitigation: DNS name (instead of IP) handles automatic redirects

Auto-Rotation in GKE ​

GKE manages rotation automatically:

bash
# View certificate info
gcloud container clusters describe my-cluster \
  --format="value(certificateConfig)"

# Rotation happens automatically every 3 years
# No manual intervention required

Manual Rotation (If Needed) ​

bash
# For custom CA setup, manual rotation

# 1. Generate new cert
openssl genrsa -out new-ca-key.pem 2048
openssl req -x509 -new -nodes \
  -key new-ca-key.pem \
  -days 3650 \
  -out new-ca-cert.pem

# 2. Update API Server
# (typically automatic in GKE)

# 3. Update kubeconfigs
kubectl config set-cluster my-cluster \
  --certificate-authority=new-ca-cert.pem

Reference Documentation ​

Summary ​

  • SSL certificate: Rotates automatically, dual-cert during transition
  • CA rotation: New CA issued, old one valid until expiry
  • IP changes: Rare, handled via DNS indirection
  • GKE automatic: Management responsibility on Google
  • Zero-downtime: Careful phasing prevents service interruption