Networking Issues: Connectivity Test Procedures

Symptoms Recognition

Network issues xuất hiện khi:

Pod không kết nối được tới service (timeout, refused, unreachable)
Traffic giữa Pods bị block (dù không có network policy)
Service endpoints không healthy
DNS resolution fails hoặc slow
External traffic không reach vào cluster
Latency cao giữa các Pods

Why This Matters

Networking issues ở GKE khác từ on-premise. GCP sử dụng Andromeda SDN overlay — traffic flow qua multiple layers (veth, iptables, VPC encapsulation, Dataplane). Một error di layer mana pun → end-to-end connectivity fail. Understanding network path troubleshooting là essential để production SREs.

Information Gathering — Quick Diagnostics

Step 1: Verify Pod Network Configuration

bash

# View pod IP assignment
kubectl get pods -n <namespace> -o wide

# Nonton pod network interfaces
kubectl exec <pod-name> -n <namespace> -- ip addr show

# View pod routing table
kubectl exec <pod-name> -n <namespace> -- ip route show

# View default gateway
kubectl exec <pod-name> -n <namespace> -- ip route show default

Interpretasi:

Pod IP should be trong CIDR mà dikonfigurasi saat cluster creation
Default route should point đến host (169.254.1.1 để GKE)
Nếu không có IP hoặc route → CNI không assign properly

Step 2: Test Pod-to-Pod Connectivity

bash

# Từ source pod, ping destination
kubectl exec <source-pod> -n <ns> -- ping <dest-pod-ip>

# Curl với timeout
kubectl exec <source-pod> -n <ns> -- curl -v --max-time 5 http://<dest-pod-ip>:8080

# Nslookup để DNS
kubectl exec <source-pod> -n <ns> -- nslookup <service-name>.<namespace>.svc.cluster.local

# Check internal DNS
kubectl exec <source-pod> -n <ns> -- cat /etc/resolv.conf

Hasil mà diharapkan:

Ping phải respond với latency < 5ms
Curl phải connect (nếu service running)
nslookup phải resolve đến Service IP
resolv.conf phải point đến kube-dns (10.x.x.x)

Step 3: Test Service Connectivity

bash

# View service
kubectl get svc -n <namespace>
kubectl describe svc <service-name> -n <namespace>

# View endpoints
kubectl get endpoints -n <namespace>
kubectl describe endpoints <service-name> -n <namespace>

# Port-forward để test
kubectl port-forward svc/<service-name> 8080:80 -n <namespace>
curl http://localhost:8080

# Test service DNS
kubectl exec <pod> -n <namespace> -- nslookup <service-name>

Cek để:

Service IP phải valid (10.x.x.x)
Endpoints phải list Pod IPs (không phải empty)
Nếu endpoints empty → Pod không match selector

Step 4: Use Connectivity Tests (GCP Native Tool)

bash

# GCP native connectivity test (network layer)
gcloud compute networks connectivity-tests create <test-name> \
  --source-ip=<pod-ip> \
  --destination-ip=<dest-ip> \
  --protocol=tcp \
  --destination-port=8080

# Check result
gcloud compute networks connectivity-tests run <test-name>
gcloud compute networks connectivity-tests describe <test-name>

Output memberikan:

Route analysis (path từ source đến dest)
Firewall rule evaluation
Network policy impact
Packet drop reasons

Step 5: Check Network Policies

bash

# View network policies
kubectl get networkpolicies -n <namespace>
kubectl describe networkpolicy <policy-name> -n <namespace>

# Check mà mana policies apply đến pod
kubectl get networkpolicies -n <namespace> -o json | \
  jq '.items[] | select(.spec.podSelector.matchLabels != null)'

# View policy evaluations (Cilium specific)
kubectl exec <pod> -n <namespace> -- cilium policy get

Cek để:

Policies với empty ingress/egress (block all)
Policies mà không match pod labels
Dataplane V2 vs Calico differences

Diagnostic Decision Tree

Connectivity Issue
│
├─ Pods have IPs?
│  ├─ NO → CNI issue, see Pod Creation Failures section
│  │
│  └─ YES → continue
│
├─ Pod-to-Pod direct IP works?
│  ├─ YES: continue (service/DNS issue likely)
│  │
│  └─ NO → deeper network diagnosis needed
│     ├─ Ping response từ dest pod?
│     │  ├─ YES: ACK received, but something after pod fails
│     │  │  └─ Check pod-level firewall, SO_RCVBUF
│     │  │
│     │  └─ NO: packet lost somewhere
│     │     ├─ Same node? → local veth issue, check CNI logs
│     │     ├─ Different nodes? → inter-node routing, check VPC
│     │     └─ Different cluster? → Network Connectivity Center, VPC peering
│     │
│     └─ TCPdump để packet analysis?
│        └─ Capture packets di source/dest pod, node, check flow
│
├─ Service DNS resolves?
│  ├─ YES: continue (DNS okay, probably endpoint issue)
│  │
│  └─ NO → CoreDNS issue
│     ├─ CoreDNS pod running?
│     ├─ kube-dns service endpoint?
│     └─ Firewall allow DNS (port 53)?
│
├─ Service endpoints populated?
│  ├─ YES: continue (selector mismatch hoặc pod failure)
│  │
│  └─ NO → Pod selector mismatch
│     ├─ Pod labels match selector?
│     ├─ Pod namespace match?
│     └─ Update selector hoặc pod labels
│
├─ Network Policy allows traffic?
│  ├─ NO: policy blocking
│  │  └─ Check ingress/egress rules, namespaceSelector
│  │
│  └─ YES: continue (no policy block)
│
└─ External traffic?
   ├─ Service Type LoadBalancer healthy?
   │  └─ Check backend health checks, firewall rules
   │
   └─ Dataplane V2 issue?
      └─ Check Cilium pod logs, packet drops

Common Root Causes & Fixes

Root Cause 1: Network Policies Blocking Traffic

Dấu hiệu:

Pod-to-Pod direct IP works, Service không
Event: traffic blocked bởi policy
Dataplane V2: cilium-agent logs show "Policy denied"

Nguyên nhân: NetworkPolicy với default-deny hoặc restrictive ingress/egress rules block connection.

Diagnostic:

bash

# View network policies di namespace
kubectl get networkpolicies -n <namespace> -o wide

# View pod labels (để selector matching)
kubectl get pod <pod-name> -n <namespace> --show-labels

# Check policy ingress/egress rules
kubectl get networkpolicies -n <namespace> -o yaml | \
  grep -A10 "ingress:\|egress:"

# Nếu Dataplane V2 (Cilium):
kubectl exec -it <cilium-pod> -n kube-system -- cilium policy get

# Packet drop reason (Cilium)
kubectl exec <pod> -n <namespace> -- \
  tcpdump -i eth0 -nn 'tcp port 80' -c 10

Immediate Fix:

bash

# Option 1: Temporarily disable network policies (debug)
# Edit policy, set empty ingress/egress
kubectl edit networkpolicy <policy-name> -n <namespace>

# Option 2: Add ingress rule để source pod
kubectl patch networkpolicy <policy-name> -n <namespace> --type='json' \
  -p='[{"op":"add","path":"/spec/ingress/-","value":{"from":[{"podSelector":{"matchLabels":{"app":"source-app"}}}],"ports":[{"protocol":"TCP","port":8080}]}}]'

# Option 3: Check policy selector logic
# Make sure podSelector/namespaceSelector correct

Permanent Fix:

Document network policies để mỗi namespace
Test policies trước apply (dry-run)
Implement network policy testing trong CI/CD
Use NetworkPolicy analyzer tools để visualize policies

Prevention:

Require NetworkPolicy PRs để peer review
Implement deny-default + explicit allow pattern
Alert nếu connectivity issues relate đến policy changes

Root Cause 2: Service Endpoint Not Populated

Dấu hiệu:

Service created nhưng endpoints empty
kubectl get endpoints <svc> show no addresses
Service IP not accessible từ pod

Nguyên nhân: Pod selector không match any pods, hoặc pods không ready.

Diagnostic:

bash

# View service selector
kubectl get svc <svc-name> -n <namespace> -o yaml | grep -A3 "selector:"

# View pods mà phải match
kubectl get pods -n <namespace> --show-labels

# Check manual endpoint creation (nếu có)
kubectl get endpoints <svc-name> -n <namespace> -o yaml

# View pod readiness probe
kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Ready\|Probe"

Common Issues:

Issue	Fix
Pod labels không match service selector	Update pod labels hoặc selector
Pod not ready (readiness probe failing)	Fix pod, probe logic, hoặc remove probe
Pod di namespace berbeda	Service và Pod phải same namespace
Headless service với ClusterIP	Remove clusterIP: None nếu ingin endpoints

Immediate Fix:

bash

# Option 1: Fix pod labels
kubectl label pod <pod-name> -n <namespace> app=myapp --overwrite

# Option 2: Update service selector
kubectl patch svc <svc-name> -n <namespace> -p '{"spec":{"selector":{"app":"myapp"}}}'

# Option 3: Check/disable readiness probe
kubectl set probe deployment <dep> --readiness --initial-delay-seconds=10 --timeout-seconds=2

# Option 4: Manual endpoint creation (temporary)
kubectl patch endpoints <svc-name> -n <namespace> -p \
  '{"subsets":[{"addresses":[{"ip":"<pod-ip>"}],"ports":[{"port":8080}]}]}'

Permanent Fix:

Enforce Pod labeling standards (CI/CD validation)
Service selector documentation
Automated testing để service-pod mapping

Root Cause 3: DNS Resolution Failures

Dấu hiệu:

nslookup <service-name> timeout hoặc "no such host"
CoreDNS pod CrashLoop
High DNS query latency
Service accessible by IP nhưng không by name

Nguyên nhân: CoreDNS không running, không reachable, hoặc overload.

Diagnostic:

bash

# View CoreDNS/kube-dns
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl get pods -n kube-system -l k8s-app=coredns

# Check pod logs
kubectl logs -n kube-system -l k8s-app=coredns -f

# Check service endpoints
kubectl get endpoints -n kube-system kube-dns
kubectl get svc -n kube-system kube-dns

# Test DNS từ pod
kubectl exec <pod> -n <namespace> -- nslookup kubernetes.default

# View resolv.conf
kubectl exec <pod> -n <namespace> -- cat /etc/resolv.conf

Immediate Fix:

bash

# Option 1: Restart CoreDNS
kubectl rollout restart deployment coredns -n kube-system

# Option 2: Check if CoreDNS pods stuck
kubectl describe pod <coredns-pod> -n kube-system

# Option 3: Increase CoreDNS replicas (nếu load too high)
kubectl scale deployment coredns -n kube-system --replicas=3

# Option 4: Nếu completely broken, scale down problematic deployment
# và scale up fresh CoreDNS
kubectl delete deployment coredns -n kube-system
# GKE will auto-redeploy hoặc manual recreate

Permanent Fix:

Monitor CoreDNS:
- Metric: coredns request latency
- Alert: CoreDNS pod CrashLoop
- Alert: CoreDNS query latency > 100ms

NodeLocal DNS Cache (recommended):

bash

gcloud container clusters update <cluster> \
  --enable-node-local-dns

Tune CoreDNS caching:

yaml

# Edit coredns ConfigMap
kubectl edit configmap coredns -n kube-system
# Increase cache size, enable negative caching

Prevention:

PodDisruptionBudget để CoreDNS: minAvailable: 1
Affinity rules để spread CoreDNS across nodes
Monitoring DNS latency per pod (custom metrics)

Root Cause 4: Dataplane V2 / Cilium Issues

Dấu hiệu:

Connectivity fail với Dataplane V2 enabled
Cilium pod CrashLoop hoặc stuck NotReady
eBPF program load errors
High packet drop rate

Nguyên nhân: Dataplane V2 uses eBPF (Cilium) để network. eBPF program không load / incompatible kernel version.

Diagnostic:

bash

# Check Dataplane V2 status
gcloud container clusters describe <cluster> | grep dataplaneV2Enabled

# View Cilium pods
kubectl get pods -n kube-system -l k8s-app=cilium
kubectl logs -n kube-system -l k8s-app=cilium -f

# Check eBPF programs
kubectl exec -it <cilium-pod> -n kube-system -- cilium bpf list

# Kernel version (must support eBPF)
kubectl exec <pod> -n <namespace> -- uname -r
# Should be 5.10+

Common Issues:

Issue	Solution
eBPF prog load failed	Kernel version < 5.10, upgrade node image
High packet drop	Increase eBPF program limits, Dataplane V2 known issue
Cilium pod CrashLoop	Disable Dataplane V2 (gcloud ... --enable-dataplane-v2=false)

Immediate Fix:

bash

# Option 1: Disable Dataplane V2 (if blocking)
gcloud container clusters update <cluster> --enable-dataplane-v2=false

# Option 2: Restart Cilium pods
kubectl rollout restart daemonset/cilium -n kube-system

# Option 3: Check node kernel version (if too old)
# Upgrade nodes via node pool upgrade (see Upgrades chapter)

Permanent Fix:

Ensure all nodes kernel version >= 5.10
Monitor Cilium pod health
Use NodeLocal DNS Cache với Dataplane V2 để optimal performance
Test Dataplane V2 trong staging first

Root Cause 5: External Ingress / Load Balancer Issue

Dấu hiệu:

External client không có thể reach cluster
LoadBalancer Service stuck in Pending
Backend health check failures
Firewall rules không match traffic

Nguyên nhân: External traffic blocked bởi firewall, health checks failing, hoặc backend configuration wrong.

Diagnostic:

bash

# View LoadBalancer service
kubectl get svc -n <namespace> -o wide
kubectl describe svc <svc-name> -n <namespace>

# Check health check configuration
gcloud compute backend-services list
gcloud compute backend-services describe <service-name> --global

# Check firewall rules
gcloud compute firewall-rules list --filter="name~<cluster-name>"

# Check NEG endpoints (để GKE Ingress)
kubectl get networkendpointgroups

Immediate Fix:

bash

# Option 1: Create firewall rule để allow external traffic
gcloud compute firewall-rules create allow-http \
  --allow=tcp:80,tcp:443 \
  --source-ranges=0.0.0.0/0 \
  --target-tags=<node-pool-tag>

# Option 2: Check health check endpoint
# Verify pod listening on correct port
kubectl exec <pod> -n <namespace> -- ss -tlnp | grep 8080

# Option 3: Update service port
kubectl patch svc <svc-name> -n <namespace> \
  -p '{"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":8080}]}}'

Prevention & Monitoring

Monitoring Setup

bash

# Cloud Monitoring dashboards
# Metric 1: Pod Network Receive/Send Errors
# Metric 2: Service endpoint availability
# Metric 3: CoreDNS query latency
# Alert if any spike

Network Policy Testing

yaml

# Test network policy before apply
apiVersion: v1
kind: Namespace
metadata:
  name: test-np
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-policy
  namespace: test-np
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 8080

Escalation Criteria

Escalate if:

Packet loss > 0.1% consistently
Dataplane V2 causing widespread connectivity issues
Network path issues between zones/regions (GCP backbone issue)
DDoS/traffic shaping suspected

Control Plane Issues — Nếu API latency tinggi (có thể affect service watch)
Node Issues — Nếu node networking misconfigured
Pod Creation Failures — CNI issues

Networking Issues: Connectivity Test Procedures ​

Symptoms Recognition ​

Why This Matters ​

Information Gathering — Quick Diagnostics ​

Step 1: Verify Pod Network Configuration ​

Step 2: Test Pod-to-Pod Connectivity ​

Step 3: Test Service Connectivity ​

Step 4: Use Connectivity Tests (GCP Native Tool) ​

Step 5: Check Network Policies ​

Diagnostic Decision Tree ​

Common Root Causes & Fixes ​

Root Cause 1: Network Policies Blocking Traffic ​

Root Cause 2: Service Endpoint Not Populated ​

Root Cause 3: DNS Resolution Failures ​

Root Cause 4: Dataplane V2 / Cilium Issues ​

Root Cause 5: External Ingress / Load Balancer Issue ​

Prevention & Monitoring ​

Monitoring Setup ​

Network Policy Testing ​

Escalation Criteria ​

Related Sections ​

References ​

Networking Issues: Connectivity Test Procedures

Symptoms Recognition

Why This Matters

Information Gathering — Quick Diagnostics

Step 1: Verify Pod Network Configuration

Step 2: Test Pod-to-Pod Connectivity

Step 3: Test Service Connectivity

Step 4: Use Connectivity Tests (GCP Native Tool)

Step 5: Check Network Policies

Diagnostic Decision Tree

Common Root Causes & Fixes

Root Cause 1: Network Policies Blocking Traffic

Root Cause 2: Service Endpoint Not Populated

Root Cause 3: DNS Resolution Failures

Root Cause 4: Dataplane V2 / Cilium Issues

Root Cause 5: External Ingress / Load Balancer Issue

Prevention & Monitoring

Monitoring Setup

Network Policy Testing

Escalation Criteria

Related Sections

References