Service Connectivity Debugging — DNS, NetworkPolicy, Routing

Tại sao quan trọng ở Production

Banyak "Pod crash" sebenarnya bukan pod crash. Pod running perfectly, tapi tidak thể connect ke dependency (database, cache, API).

Ketika terjadi:

Application timeout: "connection refused", "i/o timeout", "no such host"
Traffic routing yang unexpected: request sampai wrong pod, atau dropped

Anda perlu understand layer-by-layer: Pod name resolution → service discovery → load balancing → packet routing. Kalau salah satu break, seluruh service down.

Internal Model: Kubernetes Service Architecture

Service dibuat untuk solve container IP problem

Dalam GKE, setiap Pod punya IP address (10.x.y.z). Tapi container IP ephemeral — Pod restart, IP berubah. Application tidak bisa hardcode Pod IP.

Solution: Service. Service adalah abstraction yang:

Select pods (via labels)
Assign stable virtual IP (ClusterIP)
Load balance traffic ke pods behind-nya

yaml

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080
  type: ClusterIP

Result:

Service mendapat ClusterIP (misal: 10.0.10.5)
DNS entry: web.default.svc.cluster.local → 10.0.10.5
Traffic to ClusterIP → route ke backend pods port 8080

DNS: How Pod Resolves Service Name

Ketika pod melakukan curl http://web/, ini yang terjadi:

1. Pod process call getaddrinfo("web")
   ↓
2. musl libc / glibc forward request ke DNS resolver
   (Default: kubelet specify resolver: /etc/resolv.conf)
   ↓
3. /etc/resolv.conf point ke:
   nameserver 10.0.0.10  (kube-dns service ClusterIP)
   ↓
4. DNS query (UDP port 53) dikirm to kube-dns:
   Q: web
   ↓
5. kube-dns CoreDNS process:
   - Check apakah "web" ada di local cluster
   - Kalau namespace omitted → use pod's namespace
   - Return: A record untuk "web.default.svc.cluster.local" = 10.0.10.5
   ↓
6. Pod process mendapat IP 10.0.10.5
   ↓
7. Pod connect to 10.0.10.5:80

Critical: DNS resolver itu containerized pod (kube-dns service). Kalau kube-dns pods down → semua pods gagal resolve.

Service Load Balancing: iptables / eBPF

ClusterIP adalah virtual IP. Paket ke ClusterIP tidak straight-forward forward. GKE implement via:

Option 1: iptables (classic)

Kubelet on each node install iptables rules:

iptables rule untuk "web" service (10.0.10.5:80):
-A KUBE-SVC-XXXX \
  -m statistic --mode random --probability 0.25 \
  -j KUBE-SEP-YYYY  # endpoint 1 (pod 1, port 8080)
-A KUBE-SVC-XXXX \
  -m statistic --mode random --probability 0.33 \
  -j KUBE-SEP-ZZZZ  # endpoint 2 (pod 2, port 8080)
...

Kalau pod A connect to 10.0.10.5:80:

Kernel intercept packet (iptables rule untuk destination IP 10.0.10.5)
Apply DNAT (destination NAT): rewrite destination IP → pod actual IP
Rewrite source IP (SNAT) sehingga return traffic route kembali
Forward packet ke actual pod

Option 2: eBPF (GKE Dataplane V2)

eBPF program attach ke kernel, handle load balancing lebih efficient:

1. Packet arrive at node NIC
2. eBPF XDP program (jalan very early, kernel level) intercept
3. eBPF do lookup: "10.0.10.5:80 → backend pod IP + port"
4. eBPF rewrite packet headers (DNAT, SNAT)
5. Forward to actual pod

eBPF lebih fast, less CPU overhead dibanding iptables.

NetworkPolicy: Firewall untuk Pods

Default Kubernetes: semua pods bisa talk to semua pods (no network policy).

NetworkPolicy add firewall rules:

yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
spec:
  podSelector: {}  # Apply ke semua pods
  policyTypes:
  - Ingress
  ingress: []  # Empty: nothing allowed

This deny semua inbound traffic.

yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-to-api
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: web
    ports:
    - protocol: TCP
      port: 8080

This allow traffic dari pods dengan label app: web ke app: api pods pada port 8080.

Implementation: GKE pake iptables atau eBPF (depend on Dataplane version) untuk enforce rules. Setiap packet check: "apakah source pod allowed to connect to destination pod on this port?"

Service Discovery Resolution

Saat pod reference service:

api-pod curl http://db:5432
                   ↑
                  "db" DNS name

Kubernetes DNS plugin auto-resolve:

Same namespace: db → db.default.svc.cluster.local
Different namespace: db.other → db.other.svc.cluster.local
Headless service: DNS return multiple A records (untuk stateful sets)

Debugging Connectivity Failures

Symptom 1: "Connection Refused" or "Connection Timeout"

bash

Pod A curl http://web:80
# Error: connection refused, atau timeout setelah 30s

Diagnostic path:

bash

# Step 1: Check apakah service exist
kubectl get service web -n default
# If tidak exist → create it

# Step 2: Check apakah service punya endpoints (backend pods)
kubectl get endpoints web -n default
# Or
kubectl describe service web
# Look untuk "Endpoints:" field

# Endpoints empty? → selector not matching any pods
# Fix: check pod labels vs service selector

# Step 3: DNS resolve correct?
kubectl exec -it <pod-a> -- nslookup web
# Expected output:
# Name: web.default.svc.cluster.local
# Address: 10.0.10.5

# Actual output: "NXDOMAIN" or "i/o timeout"?
# → DNS problem

# Step 4: Can curl ClusterIP directly?
kubectl exec -it <pod-a> -- curl http://10.0.10.5:80
# Jika success → DNS issue
# Jika timeout → networking/routing issue

Case A: DNS Cannot Resolve Service

bash

kubectl exec -it <pod-a> -- nslookup web
# output: "NXDOMAIN" or "server address 10.0.0.10#53: temporary failure"

Root causes:

kube-dns service down:

bash

kubectl get pods -n kube-system | grep dns
# Should show "coredns" pods (healthy)

# Check logs
kubectl logs -n kube-system -l k8s-app=kube-dns

DNS resolver configuration wrong:

bash

kubectl exec -it <pod-a> -- cat /etc/resolv.conf
# Should show: nameserver 10.0.0.10 (kube-dns)

# Jika pointing ke wrong IP → pod spec DNS policy wrong
kubectl get pod <pod-a> -o yaml | grep -A5 "dnsPolicy"

Cloud DNS misconfigured (jika GKE use Cloud DNS):

bash

# Check cluster DNS configuration
gcloud container clusters describe <cluster> --format='value(dnsConfig)'

# Try resolve dari node
ssh <node>
nslookup web.default.svc.cluster.local 10.0.0.10

Fix:

Restart kube-dns pods: kubectl rollout restart deployment -n kube-system coredns
Check DNS policy in pod spec (default: ClusterFirst)

Case B: Service Endpoint Empty

bash

kubectl describe service web
# Endpoints: <none>

Meaning: service exist tapi no pods match selector.

bash

# Check service selector
kubectl get service web -o yaml | grep -A2 "selector"
# selector:
#   app: web

# Check pods with matching label
kubectl get pods --selector=app=web
# Should show pods matching label

# Jika no pods → either:
# 1. Pods not created yet
# 2. Pods crashed/pending
# 3. Label tidak match

Fix:

Create pods with correct label
Or update service selector to match existing pods

Symptom 2: "Connection Works Locally But Not Cross-Node"

bash

Pod A @ node-1: curl web:80 → Success (backend pod juga @ node-1)
Pod B @ node-2: curl web:80 → Timeout (backend pod @ node-1)

Root cause: Cross-node networking broken.

bash

# Step 1: Check Pod IP address ranges
kubectl get nodes -o wide
# Check NODE IP, POD CIDR

# Step 2: Check routes on node-2
ssh <node-2>
ip route
# Should have route untuk pod CIDR dari node-1
# e.g., "10.4.0.0/24 via 10.0.0.5 dev gke-node1"

# If route missing:
sudo ip route add 10.4.0.0/24 via 10.0.0.5 dev eth0

# Step 3: Check network connectivity
ping <pod-a-ip>  # From node-2

# Step 4: Check firewall rules
gcloud compute firewall-rules list --filter="direction:INGRESS"
# GKE should auto-create rules allowing pod-to-pod traffic

Most common cause: GCP firewall rules tidak allow pod traffic. GKE default buat rules tapi bisa conflict dengan custom rules.

Fix via GCP console:

Firewall rule yang allow internal GKE CIDR
Priority low enough tidak conflict dengan explicit deny rules

Symptom 3: NetworkPolicy Blocking Traffic

bash

Pod web → Pod api (port 8080): fail

Check NetworkPolicy exist dan block traffic:

bash

# List all NetworkPolicies
kubectl get networkpolicy -A

# Check specific namespace
kubectl describe networkpolicy -n default

# Check apakah policy match source/destination
kubectl get networkpolicy -n default -o yaml

Determine if policy apply:

yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-ingress
spec:
  podSelector:
    matchLabels:
      app: api  # <- Apply ke pods dengan label "app: api"
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: web  # <- Allow dari pods dengan "app: web"
    ports:
    - protocol: TCP
      port: 8080

Debugging:

bash

# Step 1: Check apakah destination pod punya selector label
kubectl get pods -L app -n default
# Check apakah "api" pods punya app=api label

# Step 2: Check apakah source pod punya selector label
# Jika pod tidak punya label → policy tidak match → traffic blocked

# Step 3: Enable NetworkPolicy logging (GKE Dataplane V2 required)
# Create NetworkPolicy dengan logging
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-deny-logging
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress: []  # Deny all
  # GKE: enable logging
  # Check GCP Cloud Logging untuk "networkpolicy_" logs

Monitor NetworkPolicy logs:

bash

gcloud logging read \
  'resource.type="k8s_cluster" AND jsonPayload.policy_name="api-deny-logging"' \
  --limit 20

Symptom 4: Service IP Unreachable Across Pods

bash

Pod A: curl 10.0.10.5:80 → works
Pod B: curl 10.0.10.5:80 → timeout

Root cause: iptables/eBPF rules tidak consistent across nodes, atau kube-proxy failed update.

bash

# Step 1: Check kube-proxy status
kubectl get pods -n kube-system -l k8s-app=kube-proxy

# Step 2: Check kube-proxy logs
kubectl logs -n kube-system -l k8s-app=kube-proxy | head -50

# Step 3: SSH node, check iptables rules
ssh <node-where-pod-b>
sudo iptables-save | grep "web\|10.0.10.5"
# Should show KUBE-SVC rules untuk service

# Step 4: Check conntrack
sudo conntrack -L | grep 10.0.10.5
# Should show connection tracking entries

Most common: kube-proxy lag updating rules. Solution: restart kube-proxy pod.

bash

kubectl rollout restart daemonset kube-proxy -n kube-system

Port-Forward as Debugging Tool

kubectl port-forward tunnel traffic dari local machine to pod, bypassing service/network policy:

bash

# Forward local:8080 → pod:8080
kubectl port-forward pod/api-123 8080:8080

# Now on local machine:
curl http://localhost:8080/health

Why useful:

Bypass NetworkPolicy: port-forward tunnel through kubelet, not constrained by NetworkPolicy
Test pod directly: check apakah pod respond at all (without service routing)
Debug network latency: measure latency pod → local machine

When to use:

Pod working or not? (port-forward to test)
Service connectivity issue atau pod issue? (port-forward to isolate)
Need access to pod without exposing via service? (port-forward + ssh)

Cross-Cutting Issues: Packet Level Debugging

Kalau DNS OK, service OK, tapi still timeout → packet level issue.

Packet Capture & Analysis

bash

# SSH ke node dimana destination pod running
ssh <node>

# Create toolbox container untuk packet capture
sudo toolbox
# or
docker run -it --rm --privileged --net=host google/cloud-sdk bash

# Install tcpdump
apt-get update && apt-get install -y tcpdump

# Capture traffic
tcpdump -i eth0 -s 100 'port 8080' -w /tmp/capture.pcap

# Analyze (dalam container, atau download to local)
tcpdump -r /tmp/capture.pcap -X

Look untuk:

SYN timeout: Source send SYN packet, destination never respond
- Meaning: packet drop, firewall, routing not working
SYN received, ACK not sent: Destination receive SYN tapi tidak send SYN-ACK
- Meaning: port tidak open, atau application not listening
Fragmentation: Large packets fragmented, some fragment drop
- Meaning: MTU mismatch between network hops

MTU (Maximum Transmission Unit) Issue

GKE default MTU: 1460 bytes (VPC MTU 1500 - GCP headers 40 bytes).

Jika pod set MTU berbeda, bisa cause fragmentation/packet drop:

bash

# Check pod NIC MTU
kubectl exec -it <pod> -- ip link show
# eth0: mtu 1500

# Check node default MTU
ssh <node>
ip link show | grep mtu
# eth0: mtu 1460

Fix: Set pod MTU to match node:

yaml

spec:
  containers:
  - name: app
    # After start, set MTU
    lifecycle:
      postStart:
        exec:
          command: ['/bin/sh', '-c', 'ip link set dev eth0 mtu 1460']

GCP-Specific Connectivity Issues

VPC Firewall Rules

GKE punya default firewall rules untuk pod-to-pod communication. Check:

bash

gcloud compute firewall-rules list --filter="name:gke-cluster"
# Should show rules untuk:
# - pod-to-pod (allow internal traffic)
# - pod-to-external (depends on your config)

Kalau missing → add rules:

bash

gcloud compute firewall-rules create allow-pods \
  --allow tcp,udp,icmp \
  --source-ranges 10.0.0.0/8 \
  --target-tags gke-node

Cloud DNS Configuration

GKE bisa use Cloud DNS (managed DNS) vs kube-dns (containerized).

Check which one used:

bash

gcloud container clusters describe <cluster> \
  --format='value(dnsConfig.clusterDns)'

# or via kubeconfig
kubectl get cm kube-dns-cm -n kube-system
# if exist → using kube-dns
# if not exist → using Cloud DNS

Debugging Cloud DNS:

bash

# From pod:
nslookup web.default.svc.cluster.local 8.8.8.8
# or
curl http://metadata.google.internal/computeMetadata/v1/instance/guest-accelerators

# Cloud DNS logs in Cloud Logging
gcloud logging read \
  'resource.type="dns_query" AND jsonPayload.query_name=~"web.default"' \
  --limit 10

Operational Practices

1. Service Troubleshooting Checklist

bash

# For every connectivity issue:
1. kubectl get service <name>  # Exist?
2. kubectl get endpoints <name>  # Have backends?
3. kubectl logs -l app=<label>  # Pods running?
4. kubectl exec <pod> -- nslookup <service>  # DNS work?
5. kubectl exec <pod> -- curl <service-ip>  # IP reach?
6. kubectl get networkpolicies  # Block traffic?
7. gcloud compute firewall-rules list  # Firewall block?

2. DNS Caching Issues

Pod DNS resolver cache results. Jika IP berubah (pod restart):

bash

# Clear DNS cache (if can't wait TTL)
# Option 1: Kill resolver process
kubectl exec <pod> -- killall -9 systemd-resolved  # if using systemd-resolved

# Option 2: Set short TTL in service
# Default: 10 seconds, usually OK

# Option 3: Use headless service (for stateful workloads)

3. Monitor Service Health

bash

# Prometheus query untuk service endpoint health
# count(up{job="kubernetes-pods", app="web"})
# Should match expected replica count

Service Connectivity Debugging — DNS, NetworkPolicy, Routing ​

Tại sao quan trọng ở Production ​

Internal Model: Kubernetes Service Architecture ​

Service dibuat untuk solve container IP problem ​

DNS: How Pod Resolves Service Name ​

Service Load Balancing: iptables / eBPF ​

NetworkPolicy: Firewall untuk Pods ​

Service Discovery Resolution ​

Debugging Connectivity Failures ​

Symptom 1: "Connection Refused" or "Connection Timeout" ​

Symptom 2: "Connection Works Locally But Not Cross-Node" ​

Symptom 3: NetworkPolicy Blocking Traffic ​

Symptom 4: Service IP Unreachable Across Pods ​

Port-Forward as Debugging Tool ​

Cross-Cutting Issues: Packet Level Debugging ​

Packet Capture & Analysis ​

MTU (Maximum Transmission Unit) Issue ​

GCP-Specific Connectivity Issues ​

VPC Firewall Rules ​

Cloud DNS Configuration ​

Operational Practices ​

1. Service Troubleshooting Checklist ​

2. DNS Caching Issues ​

3. Monitor Service Health ​

References ​

Service Connectivity Debugging — DNS, NetworkPolicy, Routing

Tại sao quan trọng ở Production

Internal Model: Kubernetes Service Architecture

Service dibuat untuk solve container IP problem

DNS: How Pod Resolves Service Name

Service Load Balancing: iptables / eBPF

NetworkPolicy: Firewall untuk Pods

Service Discovery Resolution

Debugging Connectivity Failures

Symptom 1: "Connection Refused" or "Connection Timeout"

Symptom 2: "Connection Works Locally But Not Cross-Node"

Symptom 3: NetworkPolicy Blocking Traffic

Symptom 4: Service IP Unreachable Across Pods

Port-Forward as Debugging Tool

Cross-Cutting Issues: Packet Level Debugging

Packet Capture & Analysis

MTU (Maximum Transmission Unit) Issue

GCP-Specific Connectivity Issues

VPC Firewall Rules

Cloud DNS Configuration

Operational Practices

1. Service Troubleshooting Checklist

2. DNS Caching Issues

3. Monitor Service Health

References