Private Google Access — Routing to Google APIs Without Internet

Executive Summary

Private Google Access = GCP APIs accessible từ private VMs (không external IP).

Default behavior: Private VM → Google APIs → BLOCKED (no external IP) With PGA: Private VM → 199.36.153.x → Google APIs → ✓ ALLOWED

Key:

✅ No internet egress needed
✅ No egress charges ($0.12/GB saved)
✅ Private IPs 199.36.153.x/30 reserved
✅ Per-region activation

How Private Google Access Works

Without PGA (Default)

Private VM 10.0.1.5 (no external IP):
  gsutil ls gs://my-bucket
  
Routing:
  Destination: storage.googleapis.com → 142.251.x.x (external IP)
  VM routing table: match 0.0.0.0/0
  → Need external IP for SNAT
  
Firewall:
  Egress rule: allow 0.0.0.0/0? 
  If yes: → Attempt internet egress (fails - no external IP)
  If no: → DENIED
  
Result: gsutil FAILS
         Error: Connection timeout (no SNAT possible)

With PGA Enabled

Private VM 10.0.1.5 (no external IP):
  gsutil ls gs://my-bucket
  
Routing:
  Destination: storage.googleapis.com
  
Special route (created by PGA):
  199.36.153.4/30 → Private Google Access
  199.36.153.8/30 → Private Google Access
  199.36.153.12/30 → Private Google Access
  
DNS resolution:
  storage.googleapis.com → 199.36.153.x (one of above)
  
Firewall:
  Ingress/egress: not checked (special Google path)
  
Result: gsutil SUCCEEDS
        No internet egress
        No external IP needed

Architecture: Virtual IP Ranges

Three Reserved Ranges

Range 1: 199.36.153.4/30
  - 199.36.153.4-7
  - Anycast IPs for Google services
  
Range 2: 199.36.153.8/30
  - 199.36.153.8-11
  - Redundancy
  
Range 3: 199.36.153.12/30
  - 199.36.153.12-15
  - Redundancy

All are:
  - Virtual (not real hosts)
  - Anycast (multiple entry points)
  - Reserved (cannot be used for VMs)
  - Global (same ranges in all regions)

Routing Inside GCP

Packet dest: 199.36.153.5 (Private Google Access)

Andromeda SDN:
  1. Intercepts packet at node
  2. Recognizes PGA destination
  3. Routes to Google API backend
  4. Backend response sent back
  5. Packet delivered to VM

Path: Entirely within GCP
      No internet hop
      Low latency (~1-2ms)
      No egress charges

Enabling Private Google Access

Per-Subnet

bash

gcloud compute networks subnets update prod-app \
  --region=us-central1 \
  --enable-private-ip-google-access

# Effect:
#   VMs in this subnet can reach Google APIs via PGA
#   Other subnets in same VPC: Not affected (must enable separately)

Verification

bash

gcloud compute networks subnets describe prod-app \
  --region=us-central1 \
  --format="value(privateIpGoogleAccess)"

Output: True

Services Reachable via PGA

Full List

✅ Cloud Storage (gsutil, SDK)
✅ BigQuery (bq CLI, Python/Java SDK)
✅ Cloud Pub/Sub (publishing, subscribing)
✅ Cloud Datastore (API)
✅ Cloud Tasks (API)
✅ Cloud Logging (agent, SDK)
✅ Cloud Monitoring (agent, SDK)
✅ Cloud Trace (agent, SDK)
✅ Cloud Debugger (agent, SDK)
✅ All Google Cloud APIs (via private.googleapis.com)
✅ Container Registry (pulling images)
✅ Artifact Registry (pulling images)
✅ Google Cloud APIs (compute, storage, etc.)

❌ NOT available:
  - External services (GitHub, npm registry, stripe.com)
  - GCP public endpoints (via internet DNS)
  - Custom on-premises services

Private vs Restricted Endpoints

Private Endpoints (199.36.153.x/30)

DNS: storage.googleapis.com
Resolves to: 199.36.153.5 (PGA IP)

Access:
  - Private VMs: ✓ YES (via PGA)
  - VMs with external IP: ✓ YES (via PGA or internet)
  - On-premises: ✗ NO (199.36.153.x not routable)

Restricted Endpoints (Private Service Connect)

DNS: restricted.googleapis.com
Resolves to: Private Service Connect endpoint (PSC)

Setup:
  - Create PSC endpoint in VPC
  - Configure Private Service Connect for Google APIs
  
Access:
  - Private VMs: ✓ YES (via PSC)
  - VMs with external IP: ✓ YES
  - On-premises: ✓ YES (via Cloud Interconnect + PSC)

Use case: On-premises access to Google APIs without internet

Security Implications

Network Isolation

Before PGA:
  Private VM → Internet Gateway → External internet → Google APIs
  Exposure: Traffic visible on internet (if misconfigured)

After PGA:
  Private VM → GCP backbone → Google APIs
  Isolation: No internet exposure
  Security: Contained within Google network

Firewall Rules with PGA

VM 10.0.1.5 (no external IP) enabled for PGA:

Without firewall rule:
  Egress rule: DENY 0.0.0.0/0
  Does NOT block PGA
  → VM can access Google APIs (PGA bypass firewall)

Implication:
  Firewall rules don't control PGA traffic
  (GCP routes PGA packets directly)
  
To restrict PGA access:
  1. Disable PGA on subnet
  2. Or: Use VPC Service Controls (separate control)

VPC Flow Logs & PGA

Logging PGA Traffic

VPC Flow Logs enabled:
  Direction: EGRESS
  Destination: 199.36.153.x (PGA range)
  
Log entry:
  src_ip: 10.0.1.5
  dst_ip: 199.36.153.5
  protocol: TCP
  dst_port: 443
  bytes_sent: 1024
  
Cost: Standard VPC Flow Logs pricing applies
      (~$0.50/GB)

Private Google Access cost: $0 (no egress charge)

Identifying PGA vs Internet Traffic

bash

gcloud logging read \
  'resource.type="gce_instance" AND 
   destination_ip_address=~"199.36.153\.[0-9]+"' \
  --limit=10

Result: All destination IPs in 199.36.153.0/24 range
        = Private Google Access traffic
        (not internet-bound)

Use Cases

Use Case 1: Private GKE Cluster

GKE Cluster: Private (no external IPs on nodes)
  Nodes: 10.2.0.0/16 subnet
  Pods: 10.100.0.0/16 (via alias IPs)

Requirements:
  - Pull container images from Container Registry
  - Push/pull logs to Cloud Logging
  - Query BigQuery
  - Read configs from Secret Manager

Solution:
  Enable PGA on subnet 10.2.0.0/16
  Nodes access Google APIs privately
  No external IP needed

Cost savings: $2-5/month per node (no internet egress)

Use Case 2: Data Pipeline (no internet access)

Architecture:

Cloud Compute Engine:
  VM runs Airflow DAG
  Private IP only (security policy: no external IPs)

DAG tasks:
  1. Read from BigQuery (SELECT)
  2. Process data locally
  3. Write to Cloud Storage (PUT)
  4. Upload logs to Cloud Logging

Firewall rules:
  EGRESS: DENY 0.0.0.0/0 (block internet)
  
With PGA:
  BigQuery: ✓ Accessible via 199.36.153.x
  Cloud Storage: ✓ Accessible via 199.36.153.x
  Cloud Logging: ✓ Accessible via 199.36.153.x
  
  Result: All data pipeline tasks work
          No internet exposure

Use Case 3: Compliance & Encryption

Compliance requirement:
  - No traffic leaves VPC (data residency)
  - No unencrypted communication

Traditional approach:
  - Cloud NAT for egress (visible on internet)
  - Compliance failed
  
PGA approach:
  - Private VMs access Google APIs
  - Stays within GCP backbone
  - Encrypted by default (HTTPS to APIs)
  
  Result: Compliance met
          Zero internet egress

Troubleshooting PGA

Symptom: PGA Not Working (timeout)

bash

Diagnosis:

1. Check PGA enabled:
   gcloud compute networks subnets describe prod-app \
     --region=us-central1 \
     --format="value(privateIpGoogleAccess)"
   
   If false: Enable it
   
2. Test connectivity from VM:
   gcloud compute ssh vm-1
   $ curl -I https://www.googleapis.com
   
   Timeout? → PGA not working
   
3. Check firewall egress rules:
   gcloud compute firewall-rules list \
     --filter="network:prod-vpc AND direction:EGRESS"
   
   If DENY 0.0.0.0/0: Does it exclude 199.36.153.0/24?
   (Should not, PGA bypasses firewall anyway)

4. Check DNS resolution:
   gcloud compute ssh vm-1
   $ nslookup storage.googleapis.com
   
   If resolves to external IP (not 199.36.x.x):
   → DNS resolution wrong (should be PGA IP)
   
   Solution: Clear DNS cache or check /etc/resolv.conf

5. Verify VM subnet:
   gcloud compute instances describe vm-1 \
     --format="value(networkInterfaces[0].subnetwork)"
   
   VM must be in a subnet with PGA enabled

Symptom: Some Services Work, Others Don't

Symptom:
  gsutil ls gs://bucket/ ✓ Works
  bq query --use_legacy_sql=false "..." ✗ Timeout

Diagnosis:
1. Both should work (both use PGA)

Possible causes:
  a) BigQuery query needs external data (GCS via internet)
  b) BigQuery not in PGA allowlist (rare)
  c) Network path different per service
  
Solution:
  1. Check BigQuery query for external references
  2. Simplify query (test with local data)
  3. Enable Cloud Logging debug (see actual traffic)
  4. Contact Google support if still fails

Best Practices

✅ Do:

Enable PGA for all private workloads (free benefit)
Document PGA usage in network design
Monitor PGA traffic via VPC Flow Logs
Combine with VPC Service Controls (additional control)
Use restricted endpoints for on-premises access

❌ Don't:

Assume firewall rules block PGA (they don't)
Disable PGA to "block" Google API access (use VPC-SC instead)
Use PGA for external service access (won't work)
Confuse PGA with Cloud NAT (different paths)

Comparison with Cloud NAT

yaml

Cloud NAT:
  - Use: Internet access (github.com, npm registry)
  - IP: External VM IP or NAT IP
  - Cost: Egress charges apply ($0.12/GB)
  - Scope: All destinations (internet)
  
Private Google Access:
  - Use: Google API access only
  - IP: 199.36.153.x (virtual)
  - Cost: $0 (free)
  - Scope: Google APIs only
  
Decision matrix:
  Need Google APIs only? → PGA ✓
  Need internet access? → Cloud NAT ✓
  Both? → PGA + Cloud NAT ✓
  (PGA first, Cloud NAT for external)

Conclusion

Private Google Access enables secure, private access to Google Cloud APIs:

Zero-cost: No egress charges
Secure: Stays within GCP backbone
Simple: Enable per-subnet
Performant: Low latency via Andromeda

Essential for: Private GKE, data pipelines, compliance-sensitive workloads.

Enables: Full serverless-like experience for private VMs.

Private Google Access — Routing to Google APIs Without Internet ​

Executive Summary ​

How Private Google Access Works ​

Without PGA (Default) ​

With PGA Enabled ​

Architecture: Virtual IP Ranges ​

Three Reserved Ranges ​

Routing Inside GCP ​

Enabling Private Google Access ​

Per-Subnet ​

Verification ​

Services Reachable via PGA ​

Full List ​

Private vs Restricted Endpoints ​

Private Endpoints (199.36.153.x/30) ​

Restricted Endpoints (Private Service Connect) ​

Security Implications ​

Network Isolation ​

Firewall Rules with PGA ​

VPC Flow Logs & PGA ​

Logging PGA Traffic ​

Identifying PGA vs Internet Traffic ​

Use Cases ​

Use Case 1: Private GKE Cluster ​

Use Case 2: Data Pipeline (no internet access) ​

Use Case 3: Compliance & Encryption ​

Troubleshooting PGA ​

Symptom: PGA Not Working (timeout) ​

Symptom: Some Services Work, Others Don't ​

Best Practices ​

Comparison with Cloud NAT ​

Conclusion ​

Private Google Access — Routing to Google APIs Without Internet

Executive Summary

How Private Google Access Works

Without PGA (Default)

With PGA Enabled

Architecture: Virtual IP Ranges

Three Reserved Ranges

Routing Inside GCP

Enabling Private Google Access

Per-Subnet

Verification

Services Reachable via PGA

Full List

Private vs Restricted Endpoints

Private Endpoints (199.36.153.x/30)

Restricted Endpoints (Private Service Connect)

Security Implications

Network Isolation

Firewall Rules with PGA

VPC Flow Logs & PGA

Logging PGA Traffic

Identifying PGA vs Internet Traffic

Use Cases

Use Case 1: Private GKE Cluster

Use Case 2: Data Pipeline (no internet access)

Use Case 3: Compliance & Encryption

Troubleshooting PGA

Symptom: PGA Not Working (timeout)

Symptom: Some Services Work, Others Don't

Best Practices

Comparison with Cloud NAT

Conclusion