System-generated Routes — Understanding GCP's Built-in Routing

Executive Summary

GCP automatically creates routes cho các special-case destinations:

✅ Subnet routes (0.0.0.0/0 for each subnet) — VPC-local
✅ Default route (0.0.0.0/0) — Internet egress
✅ Special routes (199.36.153.x/30) — Google APIs (Private Google Access)
✅ Reserved paths — GFE, IAP, DNS, serverless — không là routes nhưng affect packets

Hiểu những routes này là key để debug "tại sao traffic đi đây?" hoặc "tại sao bị block?"

Subnet Routes (System-generated)

Automatic Creation

Create subnet: prod-app (10.0.1.0/24) in us-central1

GCP automatically creates:

Route Name:           prod-app
Destination Range:    10.0.1.0/24
Next Hop Type:        Local network (same VPC)
Priority:             0 (highest, immutable)
Scope:                Applies to all regions (global)
Enabled:              Yes (immutable)

Result: Mọi VM trong VPC tự động reach 10.0.1.0/24 locally
        Không cần firewall allow (traffic stays in VPC)
        Không exit network (no egress charges)

Route Lifecycle

Step 1: Create subnet 10.0.1.0/24
  → Immediate route creation
  → Propagates to all regions
  → Instances in ALL regions can reach 10.0.1.0/24

Step 2: Add secondary range 10.10.0.0/16 to same subnet
  → NEW route automatically created: 10.10.0.0/16 → local
  → Old route still exists: 10.0.1.0/24 → local

Step 3: Expand primary range from /24 to /23
  → Route updates: 10.0.1.0/23 (new) and 10.0.0.0/24 (old both valid)
  
  Implication: Packets 10.0.0.0-10.0.1.255 match both routes
               Most specific match wins (10.0.1.0/23 takes 10.0.1.x)
               10.0.0.0/24 takes 10.0.0.x (from original subnet)

Step 4: Delete subnet 10.0.1.0/24
  → Route 10.0.1.0/24 deleted immediately
  → Instances in deleted subnet: packets drop or hit next route
  → If no explicit route, default 0.0.0.0/0 catches traffic
  → Packets exit VPC (unexpected!)

Pitfall: Deleted Subnet Routing

Bad scenario:

Step 1: Create subnet 10.0.1.0/24, deploy instances
        Route: 10.0.1.0/24 → local ✓
        
Step 2: Recreate subnet with different CIDR 10.0.2.0/24
        Instances: still have old primary IPs 10.0.1.x
        But new route only covers 10.0.2.0/24
        Old route deleted
        
        Result: Old VM IPs unreachable!
                Traffic from old 10.0.1.x hits default route 0.0.0.0/0
                Exits VPC to internet (security breach!)

Solution:
  - Always plan CIDR upfront
  - Don't delete/recreate subnets with different CIDRs
  - Or manually add static route: 10.0.1.0/24 → discard
    (blocks unexpected internet egress)

Default Route (0.0.0.0/0)

System-generated Internet Gateway

Every VPC has automatic default route:

Route Name:           default-route
Destination Range:    0.0.0.0/0
Next Hop Type:        Default internet gateway
Priority:             65535 (lowest, system)
Enabled:              Yes (cannot disable)

Effect: 
  - All traffic not matched by other routes → internet
  - Requires external IP on sending VM
  - Traffic exits GCP global backbone
  - Subject to egress charges ($0.12/GB egress to internet)

Traffic Path

VM 10.0.1.5 (external IP: 34.123.45.6) sends to 8.8.8.8:

1. VM routing table check:
   Destination 8.8.8.8
   Match: 0.0.0.0/0 (default route)
   Next hop: default internet gateway
   
2. Source IP preserved: 10.0.1.5 (internal)
   External IP: 34.123.45.6 (from metadata server)
   
3. SNAT at GCP edge:
   Source IP: 34.123.45.6 (visible to 8.8.8.8)
   Destination: 8.8.8.8
   
4. Packet exits Google network
   Return: 8.8.8.8 → 34.123.45.6 → DNAT → 10.0.1.5

Egress without External IP

Private VM 10.0.1.10 (no external IP) sends to 8.8.8.8:

1. VM routing table: 8.8.8.8 → default route
   
2. Default route requires external IP for SNAT
   → Packet dropped silently
   
3. VM sees: Connection timeout (no response)
   Logs: No explicit error
   
Solution 1: Assign external IP
  gcloud compute instances create ... --external-ip
  
Solution 2: Use Cloud NAT
  gcloud compute routers add-nat nat-prod \
    --router=router-prod \
    --nat-all-subnet-ip-ranges \
    --auto-allocate-nat-external-ips
    
Then: Packets → Cloud NAT → internet
      NAT IP used for SNAT
      No external IP needed on VM

Special Routes: Private Google Access

199.36.153.x/30 Routes

GCP reserves three /30 blocks for Google API access:

Range 1: 199.36.153.4/30   (199.36.153.4-7)
Range 2: 199.36.153.8/30   (199.36.153.8-11)
Range 3: 199.36.153.12/30  (199.36.153.12-15)

Purpose: Route packets to Google APIs without internet egress

Virtual routing:
  Traffic to 199.36.153.x → Special GCP handling
  → Routed to Google APIs (Cloud Storage, BigQuery, etc.)
  → No external IP needed
  → No egress charges

Enabling Private Google Access

Default behavior:

VM 10.0.1.5 (no external IP):
  Request: gcloud auth login
  → Needs Google OAuth servers
  → Routing table: default route 0.0.0.0/0
  → Requires external IP (drops silently)
  → FAILS

Enable Private Google Access on subnet:

gcloud compute networks subnets update prod-app \
  --region=us-central1 \
  --enable-private-ip-google-access

Result:
  VM 10.0.1.5 (still no external IP):
  Request: gsutil ls gs://bucket/  (Cloud Storage)
  → Packet to 199.36.153.x:443
  → GCP special routing intercepted
  → Routed to Cloud Storage privately
  → SUCCEEDS

Special routes created automatically:
  199.36.153.4/30 → private Google APIs
  199.36.153.8/30 → private Google APIs
  199.36.153.12/30 → private Google APIs

Use Cases: Private APIs

Services reachable via Private Google Access:

✅ Cloud Storage (gsutil, SDK)
✅ BigQuery (bq CLI, Python API)
✅ Cloud Pub/Sub (publishing, subscribing)
✅ Cloud Datastore (APIs)
✅ Cloud Tasks (API)
✅ Google Cloud APIs (compute, storage, etc.)

❌ NOT available: 
  - External services (stripe.com, github.com, npm registry)
  - GCP public endpoints via internet

Reserved Ranges: Implicit Handling

169.254.x.x/16 (Link-local)

169.254.x.x range = link-local (not routable outside host)

GCP uses for:
  - Metadata server: 169.254.169.254 (GCP-specific)
  - NodeLocal DNS cache: 169.254.20.10 (Kubernetes specific)
  
Traffic to 169.254.x.x:
  - Stays on node
  - Intercepted by node kernel
  - Not forwarded to VPC

Example:
  gcloud compute instances create vm1
  
  Inside VM:
  $ curl http://169.254.169.254/computeMetadata/v1/instance/id
  → Intercepted by node's kernel metadata proxy
  → Responded locally
  → Not sent to VPC

224.0.0.0/4 (Multicast)

Multicast range = group communication

GCP behavior:
  - Multicast traffic DROPPED at VPC boundary
  - Cannot use for pod-to-pod communication
  
Not routable in standard GCP VPC
  (Some limited use in on-premises via Cloud Interconnect)

Special Paths: GFE, IAP, Serverless

Google Front End (GFE) Traffic

When using Global Load Balancer:

Client 203.0.113.5 → HTTPS → GLB (anycast IP 34.110.5.6)

GCP routing:
  - GFE intercepts at Google edge
  - Not subject to VPC routing tables
  - Packet enters GCP network globally
  - Routed to nearest backend

Result:
  VPC routes don't apply to GLB-destined traffic
  Firewall rules STILL apply (backend security)
  
Important: Global LB is not CRUD through VPC routes
           Separate ingress path from compute

Identity-Aware Proxy (IAP)

Scenario: Private workload behind IAP

External client 203.0.113.5:
  → HTTPS → IAP service (managed by Google)
  → IAP validates identity
  → Routes to private backend 10.0.1.10 (no external IP)

VPC routing:
  Backend 10.0.1.10 routes:
  - No external IP needed
  - No route 0.0.0.0/0 needed
  - Firewall rule: allow tcp:8080 from IAP health check IPs
    (35.191.0.0/16, 130.211.0.0/22)

Packet path is NOT through standard VPC routing
  → IAP service directly access backend
  → Separate Google-managed path

Serverless NEGs (Cloud Run, App Engine)

Serverless services have NO VMs, NO VPC routes

When attaching serverless to LB:

Global LB → Serverless NEG (Cloud Run)
           → Cloud Run service deployed
           → No VPC IP
           → Routes to Cloud Run infrastructure
           
VPC routing doesn't apply:
  - Cloud Run IPs NOT in VPC
  - Cloud Run routes NOT in VPC routing tables
  - But firewall rules can restrict access
    (using VPC Service Controls or Cloud Armor)

Reserved Ranges: Cannot Use

Blocked CIDR ranges (cannot use in subnets):

169.254.0.0/16         Link-local (169.254.169.254 metadata server)
224.0.0.0/4            Multicast
255.255.255.255/32     Broadcast
0.0.0.0/8              This network
127.0.0.0/8            Loopback (localhost)

If you try:

bash

gcloud compute networks subnets create invalid-subnet \
  --network=prod-vpc \
  --region=us-central1 \
  --range=127.0.0.0/24

ERROR: (gcloud.compute.networks.subnets.create)
  Could not connect to the endpoint URL for compute
  ← Reserved IP range cannot be used

Routing Evaluation Order: Complete Picture

When packet arrives at VM, routing decision:

Step 1: Check subnet routes (priority 0)
  10.0.1.0/24 → local?
  10.10.0.0/16 → local?
  Match: ROUTE

Step 2: Check static routes (priority 1-64999)
  192.168.0.0/16 → vpn-gateway?
  10.20.0.0/16 → instance-a?
  Match: ROUTE (if multiple, most specific wins)

Step 3: Check dynamic routes (BGP, priority 200)
  192.168.2.0/24 → cloud-router?
  Match: ROUTE

Step 4: Check special routes
  199.36.153.x/30?
  Match: ROUTE to Private Google APIs

Step 5: Check default route (priority 65535)
  0.0.0.0/0 → default-igw?
  Match: ROUTE to internet
  
Step 6: No match
  → Packet DROPPED
  → VM sees ICMP: Destination unreachable
  
Example:
  Packet dst=172.16.0.5
  
  No subnet route
  No static route
  No dynamic route
  Not 199.36.153.x
  Match 0.0.0.0/0 (default)
  → Routes to internet gateway
  → Requires external IP
  → If no external IP: DROPPED

Observability & Debugging

List All System Routes

bash

gcloud compute routes list \
  --filter="network:prod-vpc" \
  --format=table

Output:

NAME                    NETWORK   DEST_RANGE       NEXT_HOP_IP    PRIORITY
default-route           prod-vpc  0.0.0.0/0        default-igw    65535
prod-app                prod-vpc  10.0.1.0/24      local           0
prod-db                 prod-vpc  10.0.2.0/24      local           0
route-to-onprem         prod-vpc  192.168.0.0/16   vpn-gateway     1000

Trace Routing Decision

bash

# Get all routes and sort by priority:
gcloud compute routes list \
  --filter="network:prod-vpc" \
  --format="table(priority, destination_range, next_hop_ip, name)" \
  --sort-by priority
  
# Manually check if CIDR matches:
# E.g., for destination 192.168.1.5:
# - 0.0.0.0/0 matches (priority 65535)
# - 192.168.0.0/16 matches (priority 1000) → wins

Monitor Route Changes

bash

gcloud logging read \
  'resource.type="gce_network" AND 
   protoPayload.methodName="compute.routes.create"' \
  --limit=10 \
  --format=json

Best Practices

✅ Do:

Understand subnet routes are automatic (don't double-configure)
Enable Private Google Access for private workloads
Use Cloud NAT instead of external IPs (cost + security)
Plan CIDR layout upfront (avoid delete/recreate)
Document default route behavior to team

❌ Don't:

Assume all traffic matches explicit routes (default route catches remainder)
Delete subnet and recreate with different CIDR (loses routes)
Forget that default route requires external IP or Cloud NAT
Try to use reserved ranges (169.254.x.x, 224.0.0.0/4)
Assume system routes can be modified (immutable)

Conclusion

System-generated routes provide the safety net for VPC traffic:

Subnet routes: Automatic VPC-local connectivity
Default route: Internet egress (explicit path)
Private Google Access: Secure API access without external IPs
Reserved paths: Implicit handling (metadata, multicast)

Understanding these routes prevents unexpected traffic flows and security leaks.

System-generated Routes — Understanding GCP's Built-in Routing ​

Executive Summary ​

Subnet Routes (System-generated) ​

Automatic Creation ​

Route Lifecycle ​

Pitfall: Deleted Subnet Routing ​

Default Route (0.0.0.0/0) ​

System-generated Internet Gateway ​

Traffic Path ​

Egress without External IP ​

Special Routes: Private Google Access ​

199.36.153.x/30 Routes ​

Enabling Private Google Access ​

Use Cases: Private APIs ​

Reserved Ranges: Implicit Handling ​

169.254.x.x/16 (Link-local) ​

224.0.0.0/4 (Multicast) ​

Special Paths: GFE, IAP, Serverless ​

Google Front End (GFE) Traffic ​

Identity-Aware Proxy (IAP) ​

Serverless NEGs (Cloud Run, App Engine) ​

Reserved Ranges: Cannot Use ​

Routing Evaluation Order: Complete Picture ​

Observability & Debugging ​

List All System Routes ​

Trace Routing Decision ​

Monitor Route Changes ​

Best Practices ​

Conclusion ​

System-generated Routes — Understanding GCP's Built-in Routing

Executive Summary

Subnet Routes (System-generated)

Automatic Creation

Route Lifecycle

Pitfall: Deleted Subnet Routing

Default Route (0.0.0.0/0)

System-generated Internet Gateway

Traffic Path

Egress without External IP

Special Routes: Private Google Access

199.36.153.x/30 Routes

Enabling Private Google Access

Use Cases: Private APIs

Reserved Ranges: Implicit Handling

169.254.x.x/16 (Link-local)

224.0.0.0/4 (Multicast)

Special Paths: GFE, IAP, Serverless

Google Front End (GFE) Traffic

Identity-Aware Proxy (IAP)

Serverless NEGs (Cloud Run, App Engine)

Reserved Ranges: Cannot Use

Routing Evaluation Order: Complete Picture

Observability & Debugging

List All System Routes

Trace Routing Decision

Monitor Route Changes

Best Practices

Conclusion