Observability & Distributed Tracing trong Service Mesh

Tại sao Service Mesh Observability khác biệt?

Trong microservices không có service mesh, để có distributed tracing bạn phải:

1. Instrument mỗi service với OpenTelemetry SDK
2. Propagate trace headers trong mỗi service call
3. Setup và maintain Jaeger/Zipkin/Tempo
4. Train mỗi team về tracing best practices

Với service mesh (Istio/CSM):

• Envoy tự động emit telemetry cho mọi request đi qua
• Trace headers tự động propagated bởi Envoy (nhưng có giới hạn — xem phần trace propagation)
• RED metrics sẵn có ngay sau khi install mesh, không cần instrument

Đây là giá trị lớn nhất của service mesh từ observability perspective: baseline visibility cho free.

Istio Metrics: RED Framework

Three Core Signal Types

R — Request Rate: Bao nhiêu requests/giây E — Error Rate: % requests fail D — Duration: Latency distribution (P50, P95, P99)

Istio/Envoy tự động collect và emit các metrics này cho mọi service-to-service call.

Istio Standard Metrics

# Requests total (phía client/caller)
istio_requests_total{
  reporter="source",
  source_workload="order-service",
  source_workload_namespace="production",
  destination_workload="payment-service",
  destination_workload_namespace="production",
  destination_service="payment-service.production.svc.cluster.local",
  request_protocol="http",
  response_code="200",
  grpc_response_status="",
  response_flags="-",
  connection_security_policy="mutual_tls"
}

# Request duration (latency) - histogram
istio_request_duration_milliseconds_bucket{
  le="25",        # P25 latency bucket
  ...labels...
}
istio_request_duration_milliseconds_sum{}
istio_request_duration_milliseconds_count{}

# Request bytes size
istio_request_bytes_bucket{}

# Response bytes size
istio_response_bytes_bucket{}

# TCP connections (cho non-HTTP)
istio_tcp_connections_opened_total{}
istio_tcp_connections_closed_total{}
istio_tcp_sent_bytes_total{}
istio_tcp_received_bytes_total{}

Reporter Dimension: Source vs Destination

Một điểm quan trọng: mỗi request được đo hai lần:

• reporter="source": Đo tại Envoy của caller (outbound)
• reporter="destination": Đo tại Envoy của callee (inbound)

Sự khác biệt giữa hai measurements = network latency + Envoy overhead

source_reporter_duration = network_transit + destination_envoy_processing + app_processing
destination_reporter_duration = app_processing + inbound_envoy_processing

Difference = source - destination ≈ network_transit + outbound_envoy_overhead

Thường dùng reporter="destination" để measure actual service latency.

PromQL Queries cho Service Health

# Request rate (QPS)
sum(rate(istio_requests_total{
  reporter="destination",
  destination_workload="payment-service"
}[5m]))

# Error rate (5xx)
sum(rate(istio_requests_total{
  reporter="destination",
  destination_workload="payment-service",
  response_code=~"5.*"
}[5m]))
/
sum(rate(istio_requests_total{
  reporter="destination",
  destination_workload="payment-service"
}[5m]))

# P99 latency
histogram_quantile(
  0.99,
  sum(rate(istio_request_duration_milliseconds_bucket{
    reporter="destination",
    destination_workload="payment-service"
  }[5m])) by (le)
)

# mTLS percentage
sum(rate(istio_requests_total{
  reporter="destination",
  connection_security_policy="mutual_tls"
}[5m]))
/
sum(rate(istio_requests_total{
  reporter="destination"
}[5m]))
* 100

# Circuit breaker events (Envoy metric)
increase(envoy_cluster_upstream_cx_overflow{
  cluster_name=~"outbound.*payment-service.*"
}[5m])

Response Flags: Decode Envoy Status

response_flags trong Istio metrics / access logs encode lý do tại sao request fail:

Flag	Ý nghĩa
-	Không có flag, normal response
UF	Upstream connection failure
UC	Upstream connection termination
UO	Upstream overflow (circuit breaker tripped)
NR	No route (không match VirtualService route)
RL	Rate limited
UAEX	Unauthorized external service (REGISTRY_ONLY mode)
URX	Upstream retry exhausted
DC	Downstream connection termination
LH	Local service health check failure
IH	Strict header validation failure
SI	Stream idle timeout

# Query theo flag
sum by (response_flags) (
  rate(istio_requests_total{
    destination_workload="payment-service",
    response_flags!~"-"  # Exclude normal
  }[5m])
)

Envoy Access Logs

Format mặc định

Envoy emit access log cho mỗi request theo format được configure trong MeshConfig:

[2026-06-25T10:30:00.123Z] "POST /api/v1/payments HTTP/1.1" 200 - via_upstream - "-" 1234 567 45 44 "-" "go-http-client/1.1" "req-id-abc123" "payment-service" "10.0.0.5:8080" outbound|8080|v1|payment-service.production.svc.cluster.local 10.0.1.2:54321 10.96.100.1:8080 10.0.1.3:12345 - default

Decode format:

[timestamp]
"METHOD PATH PROTOCOL"
HTTP_STATUS
GRPC_STATUS_CODE
RESPONSE_FLAGS
BYTES_RECEIVED
BYTES_SENT
DURATION_MS
UPSTREAM_SERVICE_TIME_MS
"X-FORWARDED-FOR"
"USER-AGENT"
"X-REQUEST-ID"
"AUTHORITY"
UPSTREAM_HOST
UPSTREAM_CLUSTER
UPSTREAM_LOCAL_ADDRESS
DOWNSTREAM_LOCAL_ADDRESS
DOWNSTREAM_REMOTE_ADDRESS
ROUTE_NAME

Custom Access Log Format

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: custom-access-log
  namespace: production
spec:
  accessLogging:
  - providers:
    - name: envoy
    match:
      mode: CLIENT_AND_SERVER  # Hoặc CLIENT, SERVER
    filter:
      # Chỉ log errors (giảm volume)
      expression: "response.code >= 500"
    disabled: false

Structured JSON Logging

# Configure JSON format trong MeshConfig
meshConfig:
  accessLogFile: /dev/stdout
  accessLogFormat: |
    {
      "timestamp": "%START_TIME%",
      "method": "%REQ(:METHOD)%",
      "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
      "protocol": "%PROTOCOL%",
      "response_code": "%RESPONSE_CODE%",
      "response_flags": "%RESPONSE_FLAGS%",
      "bytes_received": "%BYTES_RECEIVED%",
      "bytes_sent": "%BYTES_SENT%",
      "duration_ms": "%DURATION%",
      "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
      "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%",
      "user_agent": "%REQ(USER-AGENT)%",
      "request_id": "%REQ(X-REQUEST-ID)%",
      "upstream_host": "%UPSTREAM_HOST%",
      "upstream_cluster": "%UPSTREAM_CLUSTER%",
      "source_namespace": "%ENVIRONMENT(ISTIO_META_NAMESPACE)%",
      "trace_id": "%REQ(X-B3-TRACEID)%",
      "span_id": "%REQ(X-B3-SPANID)%"
    }

Distributed Tracing: Trace Propagation

Istio và Distributed Tracing

Đây là điểm rất quan trọng và thường bị hiểu nhầm:

Envoy TỰ ĐỘNG tạo spans cho mỗi request hop. Nhưng để tạo ra một trace liên tục từ Client → Service A → Service B → Service C, application code phải propagate trace headers giữa các service calls.

Tại sao? Vì Envoy A xử lý request từ Client đến Service A, và Envoy B xử lý request từ Service A đến Service B — nhưng chúng là hai separate spans. Để link chúng thành một trace, Service A phải forward trace headers từ incoming request sang outgoing request đến Service B.

Client → [Envoy A] → Service A app → [Envoy A] → [Envoy B] → Service B app
                                         ↑
              Service A PHẢI propagate trace headers ở đây!
              Nếu không, Envoy A trace và Envoy B trace sẽ KHÔNG connected

Trace Headers: B3 vs W3C TraceContext

B3 Multi-Header (Zipkin format, Istio default):

x-b3-traceid: a1b2c3d4e5f6a1b2
x-b3-spanid: f1e2d3c4
x-b3-parentspanid: a1b2c3d4
x-b3-sampled: 1
x-b3-flags: 1

W3C TraceContext (OpenTelemetry standard, mới hơn):

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
tracestate: rojo=00f067aa0ba902b7

Baggage (application-level propagation):

baggage: user-id=12345,request-type=premium

Application Code: Propagate Headers

// Go example với net/http
func callPaymentService(ctx context.Context, r *http.Request) {
    req, _ := http.NewRequestWithContext(ctx, "POST", "http://payment-service/pay", body)
    
    // PHẢI forward trace headers!
    for _, header := range []string{
        "x-request-id",
        "x-b3-traceid",
        "x-b3-spanid",
        "x-b3-parentspanid",
        "x-b3-sampled",
        "x-b3-flags",
        "x-ot-span-context",
        "traceparent",
        "tracestate",
        "baggage",
    } {
        if val := r.Header.Get(header); val != "" {
            req.Header.Set(header, val)
        }
    }
    
    resp, _ := http.DefaultClient.Do(req)
    // ...
}

# Python example với requests
def call_service(incoming_request):
    trace_headers = {
        "x-request-id": incoming_request.headers.get("x-request-id"),
        "x-b3-traceid": incoming_request.headers.get("x-b3-traceid"),
        "x-b3-spanid": incoming_request.headers.get("x-b3-spanid"),
        "x-b3-sampled": incoming_request.headers.get("x-b3-sampled"),
        "traceparent": incoming_request.headers.get("traceparent"),
    }
    # Filter None values
    trace_headers = {k: v for k, v in trace_headers.items() if v}
    
    response = requests.post(
        "http://payment-service/pay",
        headers=trace_headers,
        json=payload
    )

Best practice: Dùng OpenTelemetry SDK — nó tự động propagate trace context qua gRPC/HTTP clients khi được configure đúng, không cần manual header forwarding.

Sampling Strategies

Không sample 100% requests trong production — chi phí storage quá cao. Istio hỗ trợ nhiều sampling strategies:

1. Random Sampling (Default)

# 1% sampling rate
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: default
  namespace: istio-system
spec:
  tracing:
  - providers:
    - name: stackdriver
    randomSamplingPercentage: 1.0  # 1% of requests

2. Head-based Sampling (tại ingress)

# 100% sampling tại ingress, propagate sampling decision downstream
randomSamplingPercentage: 100.0  # Tại ingress gateway
# Downstream services sẽ follow "x-b3-sampled: 0/1" header từ upstream

3. Tail-based Sampling (OpenTelemetry Collector) Không support native trong Istio. Cần OpenTelemetry Collector với tail sampling processor:

# Chỉ giữ traces có errors hoặc slow (P99)
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
    - name: error-policy
      type: status_code
      status_code: {status_codes: [ERROR]}
    - name: slow-policy
      type: latency
      latency: {threshold_ms: 1000}
    - name: base-policy
      type: probabilistic
      probabilistic: {sampling_percentage: 1}

4. Per-service Sampling Override

# Tăng sampling cho payment service (critical path)
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: payment-tracing
  namespace: production
spec:
  selector:
    matchLabels:
      app: payment-service
  tracing:
  - providers:
    - name: stackdriver
    randomSamplingPercentage: 10.0  # 10% cho payment

Cloud Service Mesh Dashboard

Tính năng Dashboard

CSM Dashboard trong Google Cloud Console (Anthos → Service Mesh):

1. Service Topology

Visual service graph showing:
- Service nodes (sized by traffic volume)
- Edges (request paths, with traffic metrics)
- Health indicators (green/yellow/red)
- mTLS status per edge

2. SLO Tracking

Mỗi service có auto-generated SLOs:
- Availability SLO: % successful requests (non-5xx)
- Latency SLO: % requests dưới threshold (P99 < 500ms)

Error budget display:
- Remaining budget (%)
- Burn rate (hours of budget per hour)
- Alerting khi burn rate exceeds threshold

3. Service Details

Mỗi service:
├── Traffic (RPS, errors, latency P50/P95/P99)
├── Connected services (upstream và downstream)
├── Security (mTLS %, AuthorizationPolicy status)
└── Resources (CPU, memory của Envoy sidecar)

Telemetry Providers Configuration

# Configure CSM telemetry providers
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: default
  namespace: istio-system
spec:
  # Distributed tracing → Cloud Trace
  tracing:
  - providers:
    - name: stackdriver
    randomSamplingPercentage: 1.0
    customTags:
      # Add custom tags đến spans
      service_version:
        literal:
          value: "1.2.0"
      environment:
        environment:
          name: ENVIRONMENT

  # Metrics → Cloud Monitoring (via Managed Prometheus)
  metrics:
  - providers:
    - name: prometheus
    overrides:
    # Add custom metric tags
    - match:
        mode: CLIENT_AND_SERVER
        metric: ALL_METRICS
      tagOverrides:
        source_version:
          value: "source.labels['version'] | 'unknown'"
  
  # Access logs → Cloud Logging
  accessLogging:
  - providers:
    - name: stackdriver
    filter:
      expression: "response.code >= 400"  # Chỉ log errors và 4xx

Integration với Cloud Trace

# Xem traces trong Cloud Trace
gcloud trace list --project=PROJECT_ID --start-time="2026-06-25T10:00:00Z"

# Filter theo service
gcloud trace list \
  --project=PROJECT_ID \
  --filter="labels.component=payment-service AND hasLabel(\"g.co/r/k8s_container/namespace_name\")"

Cloud Trace integration cần:

1. CSM configured với Stackdriver trace provider
2. Workload Identity cho Envoy để write traces
3. Cloud Trace API enabled

Observability Anti-patterns trong Service Mesh

Anti-pattern 1: Chỉ nhìn vào service-level metrics

SAI: "Payment service error rate là 0%" → Assume tất cả OK

VẤN ĐỀ: Phải check cả:
- Source-side metrics (caller perspective)
- Destination-side metrics (callee perspective)
- Infra metrics (Envoy memory, CPU)

Anti-pattern 2: Không configure proper trace propagation

SAI: Chỉ rely vào Envoy automatic tracing

VẤN ĐỀ: Traces bị broken nếu app không propagate headers
→ Không thể trace full request path
→ Không thể identify root cause khi latency spike

ĐÚNG: Validate trace propagation bằng cách check
Cloud Trace cho sample traces với multiple spans

Anti-pattern 3: 100% sampling trong production

SAI: randomSamplingPercentage: 100.0 cho tất cả services

VẤN ĐỀ:
- Chi phí Cloud Trace cao (per-span billing)
- Envoy overhead tăng lên (sidecar phải report mỗi span)
- Storage explosion trong high-traffic services

ĐÚNG:
- 1% cho low-risk services
- 5-10% cho critical path
- 100% chỉ khi debugging specific issue

Anti-pattern 4: Ignore response_flags trong metrics

SAI: Chỉ nhìn vào HTTP status code

VẤN ĐỀ: HTTP 503 có thể do nhiều nguyên nhân:
- UO (circuit breaker) → upstream service cần fix
- NR (no route) → VirtualService config issue
- URX (retry exhausted) → timeout quá ngắn hoặc upstream slow

ĐÚNG: Alert và dashboard dựa trên response_flags để triage nhanh

Debugging Flow: Từ Triệu Chứng đến Root Cause

Workflow khi có latency spike

1. CSM Dashboard → payment-service P99 latency spike
   │
   ▼
2. Drill down: caller perspective vs callee perspective
   - reporter="source" latency cao → network issue hoặc upstream slow
   - reporter="destination" latency cao → app issue
   │
   ▼
3. Check response_flags
   - URX → retry exhausted, upstream timeout
   - UO → circuit breaker open
   - UF → upstream connection failure
   │
   ▼
4. Cloud Trace: Tìm slow traces
   - Identify span nào chiếm thời gian nhiều nhất
   - Database query slow? External API slow? App processing?
   │
   ▼
5. Envoy access logs (Cloud Logging)
   - Filter: response_flags != "-"
   - upstream_host field → identify specific pod bị vấn đề
   │
   ▼
6. Kubernetes metrics
   - Pod resource utilization (CPU throttling? Memory pressure?)
   - Node metrics (disk I/O, network bandwidth?)

Useful Logging Queries

# Cloud Logging query: Istio access logs với errors
resource.type="k8s_container"
labels."k8s-pod/app"="istio-proxy"
jsonPayload.response_code>=500
jsonPayload.upstream_cluster=~"payment-service"
timestamp>="2026-06-25T10:00:00Z"

# Filter theo response_flags
jsonPayload.response_flags="UO" OR jsonPayload.response_flags="URX"

# High latency requests
jsonPayload.duration > 5000  # > 5 seconds

Kết luận

Service mesh observability là "automatic level" — baseline RED metrics và trace spans cho free. Nhưng để get full value:

1. Configure trace header propagation trong application code
2. Use appropriate sampling rate (1-10% cho production)
3. Decode response_flags để triage issues nhanh
4. Layer telemetry: Envoy metrics → traces → logs → application metrics
5. CSM Dashboard cho topology và SLO visualization

Observability không thay thế application instrumentation — nó complement. Envoy biết latency và error rates, nhưng không biết tại sao application logic fail. Application metrics (business KPIs) + Envoy infrastructure metrics = full picture.

References

• Istio Observability
• Telemetry API
• Distributed Tracing
• Envoy access log format
• Cloud Service Mesh observability
• Cloud Trace integration