Private Worker Pools: Network Isolation & VPC Integration

Tại sao lại quan trọng

Default Cloud Build pools chạy trên Google's managed infrastructure — chúng có public internet access. Nhưng điều gì xảy ra nếu bạn cần:

• Access private database (Cloud SQL instance trong VPC)
• Talk to internal services (tổng đài API ở behind Cloud NAT)
• Compliance requirement: "Builds MUST run trong customer VPC, no public IP"
• Network isolation: Không muốn builds share public IPs với untrusted sources

Đó là lúc private worker pools bắt buộc phải dùng.

Private pools là game changer cho enterprise CI/CD — chúng cho phép builds run trong isolated, customer-controlled networks. Nhưng cái cost là complexity: infrastructure setup, VPC peering, network debugging.

Chương này giải thích cách chúng hoạt động, những trade-offs, và khi nào dùng chúng.

---

Mental Model: Private Worker Pools Architecture

Default Pool Topology (Review)

[Your Cloud Build Request]
        ↓
[Cloud Build Orchestrator Service]
        ↓
[Google-Managed Default Pool]
  ├─ Worker 1 (on Google's network)
  ├─ Worker 2
  └─ Worker N
  
Access:
  ✓ Public internet
  ✓ Google Cloud APIs (via service account)
  ✗ Customer VPC (no direct access)

Private Pool Topology

[Your Cloud Build Request]
        ↓
[Cloud Build Orchestrator Service]
        ↓
[Private Worker Pool]
  ├─ Worker 1 (on customer VPC via peering)
  ├─ Worker 2
  └─ Worker N
  
  ↓ VPC Peering ↓
  
[Customer VPC]
  ├─ GKE cluster
  ├─ Cloud SQL database
  ├─ Internal services
  └─ Private resources
  
Access:
  ✓ Customer VPC (via peering)
  ✓ Public internet (optional NAT)
  ✓ Google Cloud APIs

---

Architecture: Cách Hoạt Động Bên Trong

1. Service Producer Network

Google Cloud BuildER dùng Service Producer Network — một Google-managed network mà Cloud Build controls. Phần cơ bản:

Google's Infrastructure:
  ├─ Service Producer Network (Google manages)
  │  ├─ Private Worker Pool Infrastructure
  │  ├─ Cloud Build Orchestrator
  │  └─ Shared services (logging, artifact storage)
  │
  └─ VPC Peering ← → Your VPC

Your Infrastructure:
  └─ Customer VPC
     ├─ GKE clusters
     ├─ Cloud SQL
     ├─ Firewall rules
     └─ Custom routes

Key distinction: Workers chạy ở service producer network (Google-managed), nhưng chúng connect tới customer VPC qua private peering — không qua public internet.

2. VPC Peering Model

Private pools dùng VPC Service Peering:

Service Producer Network (Google)
        ↕ (peering)
Customer VPC (Your network)

Peering characteristics:

• Private connectivity: All traffic between worker pool and customer VPC goes through private peering (no public IP)
• Unidirectional ingress: Workers initiate connections vào customer VPC (access databases, caches, etc.)
• Bidirectional visibility (optional): If configured, customer VPC có thể reaching back to workers (less common)
• No IP routing required: Google manages routing; you don't see peering link ở VPC peering UI

3. Static IP Ranges

Một feature quan trọng: Private pools có predictable static IP ranges.

gcloud builds worker-pools create my-pool \
  --region=us-central1 \
  --peered-network=projects/my-project/global/networks/my-vpc \
  --network-peering-range=10.144.0.0/20

Benefits:

• Firewall whitelisting: Bạn có thể configure firewall ở backend services (Cloud SQL, Memorystore) để allow only từ10.144.0.0/20
• Audit trail: Static IPs mean bạn có thể trace builds back to specific IP ranges
• No shared IPs: Unlike default pool (dynamic public IPs), private pools have deterministic egress

Example: Cloud SQL firewall rule:

gcloud sql instances patch my-database \
  --allowed-networks=10.144.0.0/20 \
  # Workers at this IP range can connect

---

Setup: Creating Private Worker Pools

Prerequisites

1. VPC đã tồn tại trong project
2. Subnet allocation cho peering (typically /20 CIDR block)

Step 1: Create Private Pool

gcloud builds worker-pools create my-pool \
  --region=us-central1 \
  --peered-network=projects/${PROJECT_ID}/global/networks/my-vpc \
  --network-peering-range=10.144.0.0/20 \
  --machine-type=e2-standard-4

Parameters:

• --region: Gcloud region (workers spin up tại region này)
• --peered-network: Full resource name của customer VPC
• --network-peering-range: CIDR block cho workers (must be /20 hoặc larger)
• --machine-type: Machine type cho workers (default: e2-standard-4)

Step 2: Verify Peering Created

gcloud builds worker-pools describe my-pool --region=us-central1

# Output:
# peeredNetwork: projects/my-project/global/networks/my-vpc
# peeredNetworkIpRange: 10.144.0.0/20
# machineType: e2-standard-4

Step 3: Configure Build to Use Pool

# cloudbuild.yaml
steps:
  - name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', 'gcr.io/$PROJECT_ID/app:$SHORT_SHA', '.']

options:
  # Use private worker pool instead of default
  workerPool: 'projects/${PROJECT_ID}/locations/us-central1/workerPools/my-pool'

images: ['gcr.io/$PROJECT_ID/app:$SHORT_SHA']

hoặc via CLI:

gcloud builds submit --worker-pool=my-pool

Step 4: Firewall Configuration (Customer VPC)

Nếu build cần access resource trong VPC, configure firewall:

gcloud compute firewall-rules create allow-cloud-build \
  --network=my-vpc \
  --allow=tcp,udp \
  --source-ranges=10.144.0.0/20 \
  --target-tags=database-servers

Or more granular: Restrict to specific ports

gcloud compute firewall-rules create allow-cloud-build-mysql \
  --network=my-vpc \
  --allow=tcp:3306 \
  --source-ranges=10.144.0.0/20 \
  --target-tags=mysql-server

---

Constraints & Limitations

Network Isolation Trade-offs

Pro:

• ✓ Private connectivity to VPC resources
• ✓ No public IP exposure
• ✓ Compliance-friendly (network stays inside VPC)
• ✓ Static IP ranges for whitelisting

Con:

• ✗ Increased setup complexity (VPC peering, firewall rules)
• ✗ Cannot access resources protected behind VPC Service Controls (unless specially configured)
• ✗ Higher cost than default pools
• ✗ Limited to specific regions
• ✗ Slower start time (worker provisioning takes longer)

Concurrency & Capacity

Default pool:

• Max 30 concurrent builds per project
• Shared across all projects (Google scales)
• No upfront reservation

Private pools:

• Max concurrent builds = workers provisioned
• You control capacity (scale manually or via autoscaling)
• Cost per worker (even if idle)

# Check current concurrency
gcloud builds worker-pools describe my-pool --region=us-central1 \
  | grep -A 5 "concurrencyLimit"

---

Real-World Scenario: Private Pool with Database Access

Scenario Setup

• Database: Cloud SQL (MySQL) trong VPC
• Build requirement: Tests must run against real database (integration tests)
• Constraint: Database không exposed to public internet

Architecture

Cloud Build Default Pool (public)
  └─ (CANNOT access VPC resources)

Private Worker Pool (10.144.0.0/20, peered to VPC)
  ├─ Connect to Cloud SQL (10.0.1.50:3306)
  ├─ Run integration tests
  └─ Upload test results to GCS

Implementation

1. Create private pool:

gcloud builds worker-pools create db-testing \
  --region=us-central1 \
  --peered-network=projects/my-project/global/networks/vpc-prod \
  --network-peering-range=10.144.0.0/20

2. Cloud SQL firewall:

gcloud sql instances patch my-db \
  --allowed-networks=10.144.0.0/20

3. cloudbuild.yaml:

steps:
  - name: 'gcr.io/cloud-builders/git'
    args: ['clone', 'https://...']
  
  - name: 'gcr.io/cloud-builders/docker'
    args:
      - 'run'
      - '--network=host'  # Share host network with container
      - '-e'
      - 'DB_HOST=10.0.1.50'  # Cloud SQL private IP
      - '-e'
      - 'DB_PORT=3306'
      - 'gcr.io/$PROJECT_ID/test-runner'
      - 'npm test'

options:
  workerPool: 'projects/${PROJECT_ID}/locations/us-central1/workerPools/db-testing'

Benefits:

• Tests run against real database (no mocking)
• Database không exposed publicly
• Build logs contain test results (captured to Cloud Logging)
• Static IP range (10.144.0.0/20) whitelisted ở database firewall

---

Comparison: When to Use Private vs Default Pools

Aspect	Default Pool	Private Pool
Network Access	Public internet only	VPC + public (optional)
Setup Effort	Minimal	VPC peering + firewall config
Cost	Per-build	Per-worker + management overhead
Max Concurrency	30 builds	Custom (you decide)
Public IP	Yes (dynamic)	No (static private range)
VPC Access	No	Yes
Compliance	Not suitable	Suitable for isolated networks
Latency	Low (shared pool)	Slightly higher (provisioning)
Quota	Org-level limit	Per-pool limit

Decision tree:

Does build need VPC access?
├─ No → Use default pool
│       (simpler, cheaper)
└─ Yes → Use private pool
         (more complex, but necessary)

Does compliance require network isolation?
├─ No → Default pool okay
└─ Yes → Private pool required

Are you okay with setup complexity?
├─ No → Default pool
└─ Yes → Private pool (if above conditions met)

---

Troubleshooting Private Pool Issues

"Connection timed out" when accessing VPC resource

Possible causes:

1. Firewall rule not configured — Workers have private IPs, but firewall doesn't allow them
2. Peering not active — VPC peering created but not active
3. Network route missing — VPC doesn't have route to service producer network

Diagnosis:

# 1. Check peering status
gcloud compute networks peerings list --network=my-vpc

# 2. Check firewall rules
gcloud compute firewall-rules list --filter="sourceRanges:10.144.0.0/20"

# 3. Test connectivity from build step
gcloud builds submit \
  --config=debug-build.yaml \
  --worker-pool=my-pool

debug-build.yaml:

steps:
  - name: 'ubuntu:22.04'
    args:
      - sh
      - -c
      - |
        apt-get update && apt-get install -y dnsutils mysql-client
        # Test DNS resolution
        nslookup cloudsql-private-ip
        # Test connection
        mysql -h 10.0.1.50 -u user -p password -e "SELECT 1"

"Worker pool not found" error

Cause: Pool name hoặc region sai

Solution:

# List all pools
gcloud builds worker-pools list

# Verify pool name
gcloud builds worker-pools describe my-pool --region=us-central1

"Peering range conflicts with existing subnet"

Cause: IP range 10.144.0.0/20 overlaps with existing VPC subnet

Solution: Choose different range

gcloud builds worker-pools create my-pool \
  --region=us-central1 \
  --peered-network=projects/${PROJECT_ID}/global/networks/my-vpc \
  --network-peering-range=10.200.0.0/20  # Different range

---

Performance Implications

Build Start Time

Default pool: ~30 seconds (worker already running) Private pool: ~2-3 minutes (provisioning, peering setup)

This is the tradeoff for private network isolation.

Scaling

Default pool: Auto-scales, no user action needed Private pool: Manual scaling

# Scale up
gcloud builds worker-pools update my-pool \
  --region=us-central1 \
  --concurrency-limit=100

# Monitor usage
gcloud monitoring time-series list \
  --filter='metric.type="cloudbuild.googleapis.com/worker_pool_utilization"'

---

References

• Private Pools Overview — Cloud Build Documentation
• Configure Private Pools
• VPC Peering — VPC Documentation
• Cloud Build Quotas — Worker Pools