OpenTelemetry Integration - Instrumentation & OTLP Export

Tại sao OpenTelemetry thay vì proprietary client libraries

Trước OpenTelemetry, các public cloud vendors mỗi cái tự định nghĩa tracing client libraries:

AWS: X-Ray SDK
Azure: Application Insights SDK
GCP: Cloud Trace client library

Vấn đề: Nếu dùng X-Ray SDK, sau này migrate sang GCP → phải rewrite instrumentation code.

OpenTelemetry (OTEL) là vendor-neutral, open-source standard. Một khi instrumented bằng OTEL:

Có thể export sang GCP Cloud Trace
Hoặc export sang AWS X-Ray
Hoặc export sang Jaeger, Datadog, New Relic, bất cứ backend nào support OTLP
Chỉ cần thay đổi exporter configuration, không thay đổi application code

OpenTelemetry Architecture - High Level

Application Code
    |
    v
[OpenTelemetry SDK]
    |
    +---> [Tracer]        [Meter]        [Logger]
    |       |               |              |
    +---> [Span creation]  [Metric emit]  [Log emit]
    |       |               |              |
    +---> [Sampler]   (decide keep/drop)
    |       |
    +---> [SpanProcessor]
    |       |
    +---> [Exporter]
            |
            v
        [OTLP over gRPC/HTTP]
            |
            v
    [GCP Cloud Trace API]
            |
            v
        [Cloud Trace Backend]

Mỗi component:

1. TracerProvider & Tracer

python

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Global provider
trace.set_tracer_provider(TracerProvider())

# Get tracer (per module)
tracer = trace.get_tracer(__name__)

# Create span
with tracer.start_as_current_span("process_payment") as span:
    span.set_attribute("payment_id", "12345")
    # ... code

2. Sampler

Sampler quyết định: "Span này có nên exported hay không?"

python

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

# 10% của traces được sampled
sampler = TraceIdRatioBased(0.1)

# Set dalam TracerProvider
trace_provider = TracerProvider(sampler=sampler)

Sampler types:

Sampler	Behavior
`AlwaysOnSampler`	Sample 100% (development)
`AlwaysOffSampler`	Sample 0% (production testing)
`TraceIdRatioBased`	Probabilistic sampling (0-1)
`ParentBased`	Inherit parent decision

3. SpanProcessor

SpanProcessor quyết định: "Khi span complete, làm gì với nó?"

python

from opentelemetry.sdk.trace.export import BatchSpanProcessor, SimpleSpanProcessor

# Batch processor (recommended)
# - Buffer spans in memory
# - Export every 5s or 512 spans
processor = BatchSpanProcessor(exporter)

trace_provider.add_span_processor(processor)

# Simple processor (debug only)
# - Export mỗi span ngay lập tức
# - High overhead
processor = SimpleSpanProcessor(exporter)

Batch processor config:

python

from opentelemetry.sdk.trace.export import BatchSpanProcessor

processor = BatchSpanProcessor(
    exporter,
    schedule_delay_millis=5000,  # Wait 5s
    max_queue_size=2048,         # Buffer tối đa 2048 spans
    max_export_batch_size=512,   # Export 512 spans per batch
)

4. Exporter - OTLP

Exporter chịu trách nhiệm gửi spans tới backend.

GCP Cloud Trace exporter:

python

from opentelemetry.exporter.gcp_trace import CloudTraceExporter

exporter = CloudTraceExporter(
    project_id="my-gcp-project"
)

Generic OTLP exporter (compatible với bất cứ OTLP-compliant backend):

python

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="https://cloud-trace-otlp.googleapis.com:443",
)

5. Meter & Logs (Metrics & Logging)

OpenTelemetry hỗ trợ ba pillars (ba loại data):

python

# Tracing
from opentelemetry import trace
tracer = trace.get_tracer(__name__)

# Metrics
from opentelemetry import metrics
meter = metrics.get_meter(__name__)
counter = meter.create_counter("requests_total")
counter.add(1)

# Logging
from opentelemetry import logs
logger = logs.get_logger(__name__)
logger.emit(LogRecord(...))

Bài này focus vào Tracing, nhưng integrated ecosystem này important để understand.

Instrumentation Strategies

Strategy 1: Auto-Instrumentation

OpenTelemetry provides automatic instrumentations cho popular libraries:

python

# pip install opentelemetry-auto-instrumentation-python

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.auto_instrumentation.python import setup

# One-liner: automatically instrument HTTP, DB, message queues
setup()

Auto-instrumentation includes:

Library	What's Instrumented
Requests	HTTP client calls
Django	HTTP server
FastAPI	HTTP server + routing
SQLAlchemy	Database queries
psycopg2	PostgreSQL
redis	Redis cache
Kafka	Message queue

Benefit: Zero code changes. Just add auto-instrumentation package → all libraries auto-instrumented.

Drawback: Less control over what's traced, might capture things you don't want.

Strategy 2: Manual Instrumentation

python

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def process_order(order_id):
    # Parent span
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order_id", order_id)
        
        # Nested span
        with tracer.start_as_current_span("validate_order") as child_span:
            child_span.set_attribute("order_status", "valid")
            validate(order_id)
        
        # Another operation
        with tracer.start_as_current_span("fetch_inventory") as child_span:
            inventory = fetch_inventory(order_id)
        
        # Calculate total
        total = calculate_total(inventory)

Resulting span tree:

process_order (parent)
├── validate_order (child)
└── fetch_inventory (child)

Strategy 3: Hybrid - Auto + Manual

python

# Auto-instrumentation enabled
from opentelemetry.auto_instrumentation.python import setup
setup()  # Automatically instrument HTTP, DB

# Manual instrumentation for custom logic
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@app.post("/orders")
def create_order(request):
    # HTTP instrumentation is automatic (from auto-instrumentation)
    # But business logic is manual
    
    with tracer.start_as_current_span("validate_business_rules") as span:
        validate_business_rules(request.data)
    
    # DB query automatically instrumented
    order = Order.create(**request.data)
    
    with tracer.start_as_current_span("send_confirmation_email") as span:
        send_email(order.id)

OTLP Protocol - How Data Flows

OTLP Format

Spans được serialized thành OTLP format (Protocol Buffers):

protobuf

message ResourceSpans {
  Resource resource = 1;  // service name, version
  repeated InstrumentationLibrarySpans instrumentation_library_spans = 2;
}

message InstrumentationLibrarySpans {
  repeated Span spans = 1;
}

message Span {
  bytes trace_id = 1;
  bytes span_id = 2;
  bytes parent_span_id = 3;
  string name = 4;
  int64 start_time_unix_nano = 5;
  int64 end_time_unix_nano = 6;
  repeated KeyValue attributes = 7;
  // ... more fields
}

Export Channels

OTLP supports hai transport mechanisms:

1. gRPC (Default)

[Application]
    |
    v
[OTLP Batch in Protocol Buffers]
    |
    v
[gRPC over HTTP/2]
    |
    v
[Cloud Trace gRPC Endpoint]
    |
    v
[Decompress & Process]

Benefits:

Binary format (compact, efficient)
HTTP/2 (multiplexing, low latency)
Faster than HTTP/JSON

Drawback:

Requires gRPC support

Configuration:

python

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="https://cloud-trace-otlp.googleapis.com:443",
    insecure=False,  # Use TLS
)

2. HTTP (JSON)

python

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="https://cloud-trace-otlp.googleapis.com/v1/traces",  # Note: /v1/traces endpoint
)

Benefits:

Widely supported (even restricted networks)
Human-readable JSON

Drawback:

Larger payload (JSON > binary)
HTTP/1.1 (can be bottleneck at high volume)

Batch Size & Export Frequency

python

processor = BatchSpanProcessor(
    exporter,
    schedule_delay_millis=5000,  # Export every 5 seconds
    max_export_batch_size=512,   # Or when 512 spans accumulated
)

Trade-offs:

Setting	Value	Impact
`schedule_delay_millis`	1000	More frequent export (lower latency but more overhead)
`schedule_delay_millis`	10000	Less frequent export (higher latency but fewer API calls)
`max_export_batch_size`	256	Smaller batches (more API calls)
`max_export_batch_size`	2048	Larger batches (fewer API calls, more buffering)

Compression & Network Optimization

OTLP exporter automatically gzip compress payload:

python

exporter = OTLPSpanExporter(
    endpoint="...",
    headers=(("grpc-encoding", "gzip"),),  # Enable compression
)

Impact:

Uncompressed: 100KB/s
Compressed: 10-15KB/s (10x reduction)

Recommended để enable compression untuk high-volume tracing.

GCP-Specific Configuration

Google Cloud Trace Exporter

Simplest cho GCP:

python

from opentelemetry.exporter.gcp_trace import CloudTraceExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

exporter = CloudTraceExporter(
    project_id="my-gcp-project"
)

tracer_provider = TracerProvider()
tracer_provider.add_span_processor(BatchSpanProcessor(exporter))

Automatic:

Authentication (uses Application Default Credentials)
Project ID (inferred from environment)
Endpoint (uses official GCP endpoint)

Authentication

Cloud Trace exporter requires cloudtrace.agent role:

bash

gcloud projects add-iam-policy-binding my-gcp-project \
  --member=serviceAccount:my-sa@my-project.iam.gserviceaccount.com \
  --role=roles/cloudtrace.agent

Authentication mechanisms:

Mechanism	How it Works
Application Default Credentials (ADC)	Application reads env var `GOOGLE_APPLICATION_CREDENTIALS`
Service Account Key	Explicitly provide key file
GKE Workload Identity	Pod assume service account identity
Compute Engine Metadata Server	VM retrieves credentials from metadata server

Best practice: Dùng Workload Identity (GKE) atau Service Account (App Engine), không hardcode keys.

Performance Considerations

CPU Overhead

Tracing overhead (per application):

Activity	CPU Cost
Create span (in-memory)	~100 ns
Set attributes (10 attrs)	~1 microsecond
Close span	~50 ns
BatchSpanProcessor	< 0.1% CPU

Dengan 10K spans/second:

Span creation: 1ms
Batch processing: < 1ms
Total: < 0.2% CPU

Chấp nhận được.

Memory Overhead

Buffer size (BatchSpanProcessor):

python

processor = BatchSpanProcessor(
    exporter,
    max_queue_size=2048,  # Max 2048 spans in memory
)

Memory per span: ~500-1000 bytes

2048 spans × 800 bytes = ~1.6 MB

Chấp nhận được với modern applications.

Network Bandwidth

Export volume:

Span size (uncompressed): ~1 KB
10K spans/second: 10 MB/s
With gzip compression: ~1 MB/s

Chấp nhận được cho most cloud networks.

Export Failure Handling

Nếu Cloud Trace API unavailable:

python

processor = BatchSpanProcessor(
    exporter,
    schedule_delay_millis=5000,
    max_queue_size=2048,
)

# If export fails:
# 1. Queue reached max_queue_size → oldest spans dropped
# 2. Periodic retry (configurable)
# 3. Application keeps running (non-blocking)

Best practice: Monitor export errors:

python

from opentelemetry.sdk.trace.export import SpanExportResult

class MonitoredExporter(OTLPSpanExporter):
    def export(self, spans):
        result = super().export(spans)
        if result == SpanExportResult.FAILURE:
            metrics.increment("otlp_export_failures")
        return result

Instrumentation Best Practices

Pattern 1: Custom Attributes - Business Context

python

tracer = trace.get_tracer(__name__)

@app.post("/orders")
def create_order(request):
    with tracer.start_as_current_span("create_order") as span:
        # Add business context
        span.set_attribute("order.amount", request.amount)
        span.set_attribute("order.customer_tier", request.customer.tier)
        span.set_attribute("order.region", request.region)
        
        order = Order.create(**request.data)
        return order

Later, query traces by business context:

sql

SELECT
  trace_id,
  span.attributes['order.amount'] as amount,
  trace.duration_millis
FROM traces.spans
WHERE span.attributes['order.customer_tier'] = 'enterprise'
  AND trace.duration_millis > 1000

Pattern 2: Exception Handling in Spans

python

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def risky_operation():
    with tracer.start_as_current_span("risky_operation") as span:
        try:
            do_something_risky()
        except ValueError as e:
            # Record exception in span
            span.record_exception(e)
            span.set_attribute("error.type", "ValueError")
            span.set_status(trace.Status(trace.StatusCode.ERROR))
            raise

Span được marked as error dalam Cloud Trace.

Pattern 3: Span Events

Record discrete events within span:

python

with tracer.start_as_current_span("process_batch") as span:
    span.add_event("batch_started", {"size": 1000})
    
    for i, item in enumerate(items):
        process(item)
        if i % 100 == 0:
            span.add_event("batch_progress", {"processed": i})
    
    span.add_event("batch_completed", {"status": "success"})

Events helpful để tracking progress trong long-running operations.

Anti-Patterns

Anti-pattern 1: Sampling Decision in Application Code

python

# BAD
if random() < 0.1:
    record_span()

Problem:

Sampling logic scattered everywhere
Inconsistent
Difficult to change globally

Better:

python

# Good
sampler = TraceIdRatioBased(0.1)
tracer_provider = TracerProvider(sampler=sampler)
# All sampling decisions centralized

Anti-pattern 2: High-Cardinality Attribute Values

python

# BAD
span.set_attribute("user_id", user_id)  # 10M possible values

# This will:
# - Blow up memory (caching all user_ids)
# - Crash trace backend (too many dimensions)

Better:

python

# Only set attributes with bounded cardinality
span.set_attribute("user_tier", user.tier)  # Only "free", "premium", "enterprise"
span.set_attribute("region", region)  # Only 10 regions

Anti-pattern 3: Instrument Everything with SimpleSpanProcessor

python

# BAD
processor = SimpleSpanProcessor(exporter)  # Export mỗi span ngay

At high volume:

CPU spike
Network overhead
Latency increase

Better:

python

processor = BatchSpanProcessor(
    exporter,
    max_export_batch_size=512,
    schedule_delay_millis=5000
)

OpenTelemetry Integration - Instrumentation & OTLP Export ​

Tại sao OpenTelemetry thay vì proprietary client libraries ​

OpenTelemetry Architecture - High Level ​

1. TracerProvider & Tracer ​

2. Sampler ​

3. SpanProcessor ​

4. Exporter - OTLP ​

5. Meter & Logs (Metrics & Logging) ​

Instrumentation Strategies ​

Strategy 1: Auto-Instrumentation ​

Strategy 2: Manual Instrumentation ​

Strategy 3: Hybrid - Auto + Manual ​

OTLP Protocol - How Data Flows ​

OTLP Format ​

Export Channels ​

1. gRPC (Default) ​

2. HTTP (JSON) ​

Batch Size & Export Frequency ​

Compression & Network Optimization ​

GCP-Specific Configuration ​

Google Cloud Trace Exporter ​

Authentication ​

Performance Considerations ​

CPU Overhead ​

Memory Overhead ​

Network Bandwidth ​

Export Failure Handling ​

Instrumentation Best Practices ​

Pattern 1: Custom Attributes - Business Context ​

Pattern 2: Exception Handling in Spans ​

Pattern 3: Span Events ​

Anti-Patterns ​

Anti-pattern 1: Sampling Decision in Application Code ​

Anti-pattern 2: High-Cardinality Attribute Values ​

Anti-pattern 3: Instrument Everything with SimpleSpanProcessor ​

References ​

OpenTelemetry Integration - Instrumentation & OTLP Export

Tại sao OpenTelemetry thay vì proprietary client libraries

OpenTelemetry Architecture - High Level

1. TracerProvider & Tracer

2. Sampler

3. SpanProcessor

4. Exporter - OTLP

5. Meter & Logs (Metrics & Logging)

Instrumentation Strategies

Strategy 1: Auto-Instrumentation

Strategy 2: Manual Instrumentation

Strategy 3: Hybrid - Auto + Manual

OTLP Protocol - How Data Flows

OTLP Format

Export Channels

1. gRPC (Default)

2. HTTP (JSON)

Batch Size & Export Frequency

Compression & Network Optimization

GCP-Specific Configuration

Google Cloud Trace Exporter

Authentication

Performance Considerations

CPU Overhead

Memory Overhead

Network Bandwidth

Export Failure Handling

Instrumentation Best Practices

Pattern 1: Custom Attributes - Business Context

Pattern 2: Exception Handling in Spans

Pattern 3: Span Events

Anti-Patterns

Anti-pattern 1: Sampling Decision in Application Code

Anti-pattern 2: High-Cardinality Attribute Values

Anti-pattern 3: Instrument Everything with SimpleSpanProcessor

References