Trace Propagation & W3C Standards - Context Di chuyển trong Microservices

Tại sao trace context propagation là vấn đề phức tạp

Giả sử frontend service gọi user service:

Frontend                    User Service
+--------+                  +---------+
|GET /   |                  | GET /api|
|        | -----HTTP------> |         |
|        |  trace_id=ABC    |         |
|        |  span_id=DEF     |         |
+--------+                  +---------+

Câu hỏi: Làm sao user service biết rằng request này thuộc trace ABC, parent span là DEF?

Câu trả lời: Frontend phải encode trace context vào HTTP headers, user service phải extract chúng.

Nhưng vấn đề:

1. Header format không standard: Mỗi public cloud (AWS, GCP, Azure) dùng format khác
2. Cross-origin requests: Khi request đi qua multiple systems, header format phải compatible
3. Multiple propagation formats: HTTP headers, gRPC metadata, message queue headers, baggage - mỗi cái khác format
4. Backward compatibility: Legacy systems vẫn dùng old header format

W3C Trace Context - Global Standard

Năm 2020, World Wide Web Consortium standardize W3C Trace Context để unified format.

W3C Trace Context Components

W3C định nghĩa hai headers:

1. traceparent Header

Format:

traceparent: version-trace_id-parent_span_id-trace_flags

Chi tiết:

traceparent: 00-a0f3c87f9a1b2c3d4e5f6a7b8c9d0e1f-1a2b3c4d5e6f7a8b-01
             │  │                                     │                 │
             │  │                                     │                 └─ trace_flags (01=sampled, 00=not sampled)
             │  │                                     └─ parent_span_id
             │  └─ trace_id (32 hex chars, 128 bits)
             └─ version (00 = W3C v1)

trace_id (32 hex characters):

• 128-bit value
• Globally unique identifier cho entire trace
• Ví dụ: a0f3c87f9a1b2c3d4e5f6a7b8c9d0e1f

parent_span_id (16 hex characters):

• 64-bit value
• ID của span hiện tại (service B sẽ xem span này là parent của span của nó)
• Ví dụ: 1a2b3c4d5e6f7a8b

trace_flags (2 hex characters):

• Chỉ 1 bit thực sự dùng (sampled flag)
• 01 = trace is sampled (important, keep it)
• 00 = trace is not sampled (can be discarded)
• Remaining bits = reserved

2. tracestate Header

Optional, carry vendor-specific tracing information:

tracestate: congo=t61rcWpm1,rojo=00f067aa0ba902b7

Format:

tracestate: vendor1=value1,vendor2=value2

Ví dụ:

tracestate: gcp=xxx1234xxx,xray=1-5e6722a7-cc2c02ffc6221fa17d7f0000

Mỗi vendor có thể carry custom metadata:

• GCP: custom attributes
• AWS X-Ray: additional trace info
• DataDog: rum-specific data

Important: tracestate có maximum length 512 characters. Nếu vượt quá, must drop entries from left.

W3C Trace Context Complete Example

Frontend service receives HTTP request:

GET / HTTP/1.1
Host: app.example.com

Frontend generates trace:

import uuid

trace_id = uuid.uuid4().hex  # a0f3c87f9a1b2c3d4e5f6a7b8c9d0e1f
parent_span_id = uuid.uuid4().hex[:16]  # 1a2b3c4d5e6f7a8b
sampled = True  # Based on sampling decision

headers = {
    "traceparent": f"00-{trace_id}-{parent_span_id}-{'01' if sampled else '00'}"
}

Frontend calls user service:

GET /api/user/123 HTTP/1.1
Host: user-service.internal
traceparent: 00-a0f3c87f9a1b2c3d4e5f6a7b8c9d0e1f-1a2b3c4d5e6f7a8b-01

User service receives, extracts:

import re

traceparent = request.headers.get("traceparent", None)
if traceparent:
    match = re.match(r"^(\d{2})-([a-f0-9]{32})-([a-f0-9]{16})-([a-f0-9]{2})$", traceparent)
    if match:
        version, trace_id, parent_span_id, flags = match.groups()
        sampled = (flags == "01")
        
        # Create new span with inherited trace_id
        new_span_id = generate_new_span_id()
        # new_span_id will be parent_span_id for downstream services

User service calls product service:

GET /api/product/456 HTTP/1.1
Host: product-service.internal
traceparent: 00-a0f3c87f9a1b2c3d4e5f6a7b8c9d0e1f-2c3d4e5f6a7b8c9d-01
                                                  ^ NEW span_id (parent for product service)

GCP-Specific Propagation: X-Cloud-Trace-Context

GCP predates W3C Trace Context standard. Older GCP services (App Engine legacy, Cloud Functions gen1) dùng X-Cloud-Trace-Context header format:

X-Cloud-Trace-Context: TRACE_ID/SPAN_ID;o=TRACE_TRUE

Chi tiết:

X-Cloud-Trace-Context: a0f3c87f9a1b2c3d4e5f6a7b8c9d0e1f/1234567890;o=1
                       │                                     │           │
                       │                                     │           └─ o=1 (sampled), o=0 (not sampled)
                       │                                     └─ span_id (decimal, not hex!)
                       └─ trace_id (hex format)

Differences từ W3C:

• span_id là decimal (không hex)
• Format không standardized (slash + semicolon)
• Chỉ có trace_id + span_id, không có version field

GCP Migration: X-Cloud-Trace-Context → W3C traceparent

Cloud Trace ngay nay recommend W3C traceparent, nhưng:

• App Engine standard (legacy) vẫn inject X-Cloud-Trace-Context
• OpenTelemetry exporters support cả hai
• GCP client libraries tự động handle conversion

Best practice: Support cả hai formats để compatibility:

def extract_trace_context(headers):
    # Try W3C first
    traceparent = headers.get("traceparent")
    if traceparent:
        return parse_w3c_traceparent(traceparent)
    
    # Fallback to GCP format
    gcp_header = headers.get("X-Cloud-Trace-Context")
    if gcp_header:
        return parse_gcp_trace_context(gcp_header)
    
    # No trace context found
    return None

Context Propagation Mechanisms

Mechanism 1: HTTP Headers (REST APIs)

Request → Response cycle truyền context qua HTTP headers.

Frontend                              User Service
   |                                      |
   +---(HTTP)--[traceparent header]-----> |
   |                                      | Span created
   |                                      | Parent span_id = received span_id
   |                                      |
   |<-----(HTTP response)--[tracing]------|
   |

OpenTelemetry provides extractors:

from opentelemetry.propagators.jaeger import JaegerPropagator
from opentelemetry.propagators.w3c_trace_context import W3CTraceContextPropagator

# Automatically extract from HTTP headers
context = W3CTraceContextPropagator().extract(carrier=request.headers)

Mechanism 2: gRPC Metadata

gRPC dùng metadata để truyền headers, tương tự HTTP.

service UserService {
  rpc GetUser(GetUserRequest) returns (User) {}
}

Client:

from grpc import secure_channel
from grpc.experimental.aio import secure_channel

stub = UserService_pb2_grpc.UserServiceStub(channel)

# Metadata = gRPC headers
metadata = (
    ("traceparent", "00-a0f3c87f9a1b2c3d4e5f6a7b8c9d0e1f-1a2b3c4d5e6f7a8b-01"),
)

response = stub.GetUser(request, metadata=metadata)

Server:

async def GetUser(self, request, context):
    # Extract metadata (headers)
    metadata = dict(context.invocation_metadata())
    traceparent = metadata.get("traceparent")
    
    # Parse trace context
    ...

Mechanism 3: Message Queue Headers

Khi request di chuyển through message queue (Pub/Sub, Kafka), trace context phải embedded trong message.

Pub/Sub message attributes:

from google.cloud import pubsub_v1

publisher = pubsub_v1.PublisherClient()

# Embed trace context in message attributes
attributes = {
    "traceparent": "00-a0f3c87f9a1b2c3d4e5f6a7b8c9d0e1f-1a2b3c4d5e6f7a8b-01",
}

publisher.publish(
    topic_path,
    b"message content",
    **attributes
)

# Subscriber extracts
def callback(message):
    traceparent = message.attributes.get("traceparent")
    # Parse and create span

Mechanism 4: Baggage - Extra Context

Baggage là mechanism để truyền key-value pairs alongside trace context.

W3C định nghĩa baggage header:

baggage: userId=12345,requestId=req-xyz,env=prod

Use cases:

• User ID (useful trong analysis)
• Request ID (correlation between systems)
• Custom business attributes

OpenTelemetry Baggage:

from opentelemetry.baggage import set_baggage, get_baggage

# Set baggage
set_baggage("user_id", "12345")
set_baggage("request_id", "req-xyz")

# Baggage automatically propagated to downstream services
# Downstream service retrieves
user_id = get_baggage("user_id")  # "12345"

Important caveat: Baggage propagates to all downstream services, even if they don't need it. This can:

• Increase header size (network overhead)
• Leak sensitive information (PII if not careful)

So chỉ đưa vào baggage những gì thực sự cần.

Challenges in Trace Propagation

Challenge 1: Service Boundaries - Some Services Don't Propagate

Giả sử:

Frontend → API Gateway → User Service → Database

Nếu User Service gọi database, database là:

• External service?
• Internal service?
• Database driver (không HTTP, dùng proprietary protocol)?

Database driver không tự động propagate trace context (vì protocol không standard). Solution:

# Manual span creation for database call
with tracer.start_as_current_span("db.query") as span:
    span.set_attribute("db.system", "postgresql")
    result = db.execute("SELECT * FROM users")

Kết quả: Database call có span, nhưng chỉ locally (không propagate tới database server vì database protocol không support).

Challenge 2: Cross-Origin Requests & CORS

Khi frontend JavaScript fetch từ backend:

fetch("/api/user", {
  headers: {
    "traceparent": "00-abc123-..."
  }
})

Browser's CORS preflight:

OPTIONS /api/user
Access-Control-Request-Headers: traceparent

Server CORS response:

Access-Control-Allow-Headers: traceparent

Jika server không whitelist traceparent header → header di-block by browser → trace context lost.

Mitigation:

# Express server
app.use(cors({
    exposedHeaders: ['traceparent', 'tracestate']
}))

Challenge 3: Third-Party APIs & External Services

Khi call external API (Stripe, Twilio, etc.), mereka kontrolnya tidak ada.

Your Service                 Stripe API
   |                              |
   +---(HTTP)--[traceparent]----> |
                                   X (Stripe không support trace context)

Stripe tidak export trace data ke Cloud Trace. Bagaimana untuk debug?

Solution:

1. Create synthetic span untuk external call:

   with tracer.start_as_current_span("external.stripe.charge") as span:
       span.set_attribute("external.service", "stripe")
       response = stripe.Charge.create(...)

2. Log request/response untuk later correlation:

   logger.info("stripe_call", extra={
       "trace_id": trace_id,
       "stripe_request_id": response.id
   })

3. Store correlation di Stripe metadata (nếu support):

   stripe.Charge.create(
       amount=1000,
       metadata={"trace_id": trace_id}
   )

Challenge 4: Asynchronous Workflows

Khi request triggers async job:

HTTP Request                  Background Job
   |                               |
   +---(queue)---[message]-------> |
   |                               | Run later (30 mins, hours)
   | Response returned
   | (job not started yet)

trace_id nào dùng cho background job?

Option 1: Inherit from request

# Request handler
trace_id = get_current_span().get_span_context().trace_id
queue.enqueue(job, trace_id=trace_id)  # Embed dalam job

# Background worker
@background_task
def job(trace_id):
    tracer.start_as_current_span("background_job", attributes={
        "trace_id": trace_id
    })

Lợi ích: Same trace_id, connect request→job Nhược điểm: Job chạy lâu sau, timeline confusing (request done rồi job mới start)

Option 2: New trace, link to original

# Background worker
@background_task
def job():
    # Create NEW trace_id
    new_trace_id = generate_trace_id()
    
    # But link back to original
    span.add_link(Link(
        SpanContext(trace_id=original_trace_id, ...)
    ))

Lợi ích: Clean trace hierarchy Nhược điểm: Phải manually correlate

Challenge 5: Clock Skew & Out-of-Order Spans

Khi multiple services havekhác nhau, spans có thể receive out-of-order:

Service A (clock +5s)        Service B (clock -2s)
Span A: start=10:30:00       
        end=10:30:00.100
            |
            +---> HTTP ---> Span B: start=10:29:53
                                   end=10:29:53.050

Trace timeline reconstruction:

Actual order:        10:29:53 ----> 10:29:53.050 ----> 10:30:00 ----> 10:30:00.100
                     (B start)       (B end)           (A start)       (A end)

But recorded as:     10:30:00 ----> 10:30:00.100 ----> 10:29:53 ----> 10:29:53.050
                     (A start)       (A end)           (B start)       (B end)

Result: Timeline inverted! Span B appears to start after service A already called it.

Mitigation:

• Cloud Trace backend detects clock skew
• Automatic re-ordering based on parent-child relationships
• Metric: clock_skew_millis tracked per trace

Baggage & Correlation IDs

Baggage Propagation Flow

Request                 Baggage propagates
  |                          |
  +--[user_id=123]---------> Service B
  |   [request_id=xyz]           |
  |                              +--[user_id=123]--------> Service C
  |                              |  [request_id=xyz]
  |                              |

Mỗi service inherit và có thể add baggage:

from opentelemetry.baggage import set_baggage, get_baggage

# Service B receives
user_id = get_baggage("user_id")  # "123"

# Service B adds its own baggage
set_baggage("service_b_processed", "true")

# Service C receives both original + new baggage

Baggage Use Cases

Baggage Key	Use Case	Example
user_id	Identify user for analysis	Filter traces by user
request_id	Correlate across systems	Link logs + traces
tenant_id	Multi-tenancy	Isolate tenant data
session_id	Session tracking	Correlate user journey
feature_flag	A/B testing correlation	Analyze impact

Baggage Size Concerns

Baggage header có maximum size limit (HTTP headers typically ~8KB).

Nếu baggage quá lớn:

# BAD
set_baggage("user_data", json.dumps(large_user_object))  # 50KB
# → Exceeds HTTP header limit
# → Header gets dropped
# → Trace context lost

Best practice:

• Keep baggage minimal (only IDs, not large objects)
• <10 keys per baggage
• Value size <100 bytes each

Production Patterns

Pattern 1: Validate Header Propagation

Thêm metric để track header propagation success:

@app.before_request
def track_trace_context():
    traceparent = request.headers.get("traceparent")
    
    if traceparent:
        labels = {"status": "present"}
    else:
        labels = {"status": "missing"}
    
    propagation_counter.inc(labels=labels)

Query: "Is trace context propagated across all requests?"

SELECT
  service,
  COUNT(*) as total_requests,
  COUNT(CASE WHEN traceparent IS NOT NULL THEN 1 END) as with_traceparent,
  ROUND(COUNT(CASE WHEN traceparent IS NOT NULL THEN 1 END) / COUNT(*) * 100, 2) as percentage
FROM request_logs
WHERE timestamp > NOW() - INTERVAL 1 HOUR
GROUP BY service

Pattern 2: Sampled Flag Propagation

Ensure sampled flag propagates correctly:

# Frontend sampling decision
should_sample = random() < 0.1  # 10%
traceparent = f"00-{trace_id}-{span_id}-{'01' if should_sample else '00'}"

# Backend check: does Backend honor sampled flag?
@app.before_request
def honor_sampled_flag():
    traceparent = request.headers.get("traceparent")
    if traceparent:
        _, _, _, flags = parse_traceparent(traceparent)
        should_sample = (flags == "01")
        
        # Backend respects frontend decision
        set_sampling_decision(should_sample)

Pattern 3: Baggage Audit

Log baggage content để detect PII leaks:

@app.after_request
def audit_baggage(response):
    baggage_header = request.headers.get("baggage")
    if baggage_header:
        # Check for PII patterns
        if re.search(r"\b\d{3}-\d{2}-\d{4}\b", baggage_header):  # SSN pattern
            logger.warning("Potential PII in baggage", baggage=baggage_header)
    return response

References

• W3C Trace Context Specification
• Google Cloud Trace Context Propagation
• OpenTelemetry Propagators
• Cloud Trace Context Documentation