OpenTelemetry Implementation Guide

Overview

OpenTelemetry (OTel) is a vendor-neutral observability framework for instrumenting, generating, collecting, and exporting telemetry data (traces, metrics, logs). This skill provides guidance for implementing OTEL in Kubernetes environments.

Quick Start

Deploy OTEL Collector on Kubernetes

# Add Helm repo
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

# Install with basic config
helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace monitoring --create-namespace \
  --set mode=daemonset

Send Test Data via OTLP

# gRPC endpoint: 4317, HTTP endpoint: 4318
curl -X POST http://otel-collector:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[]}'

Core Concepts

Signals: Three types of telemetry data:

• Traces: Distributed request flows across services
• Metrics: Numerical measurements (counters, gauges, histograms)
• Logs: Event records with structured/unstructured data

Collector Components:

• Receivers: Accept data (OTLP, Prometheus, Jaeger, Zipkin)
• Processors: Transform data (batch, memory_limiter, k8sattributes)
• Exporters: Send data (prometheusremotewrite, loki, otlp)
• Extensions: Add capabilities (health_check, pprof, zpages)

Collector Configuration

Basic Pipeline Structure

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          endpoint: ${env:MY_POD_IP}:4318

  processors:
    batch:
      timeout: 10s
      send_batch_size: 1024
    memory_limiter:
      check_interval: 5s
      limit_percentage: 80
      spike_limit_percentage: 25

  exporters:
    prometheusremotewrite:
      endpoint: "http://prometheus:9090/api/v1/write"
    loki:
      endpoint: "http://loki:3100/loki/api/v1/push"

  service:
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [prometheusremotewrite]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [loki]
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp/tempo]

Kubernetes Attributes Enrichment

processors:
  k8sattributes:
    auth_type: "serviceAccount"
    passthrough: false
    filter:
      node_from_env_var: ${env:K8S_NODE_NAME}
    extract:
      metadata:
        - k8s.pod.name
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.node.name

Deployment Modes

Common Patterns

Development Environment

• Enable debug exporter for visibility
• Lower resource limits (250m CPU, 512Mi memory)
• Include spot instance tolerations for cost savings

Production Environment

• Implement sampling (10-50% for traces)
• Higher batch sizes (2048-4096)
• Enable autoscaling and PodDisruptionBudget
• Use TLS for all endpoints

Detailed References

For in-depth guidance, see:

• Collector Configuration: COLLECTOR.md
• Kubernetes Deployment: KUBERNETES.md
• Troubleshooting: TROUBLESHOOTING.md
• Instrumentation: INSTRUMENTATION.md

Validation Commands

# Check collector pods
kubectl get pods -n monitoring -l app.kubernetes.io/name=otel-collector

# View collector logs
kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector --tail=100

# Test OTLP endpoint
kubectl run test-otlp --image=curlimages/curl:latest --rm -it -- \
  curl -v http://otel-collector.monitoring:4318/v1/traces

# Validate config syntax
otelcol validate --config=config.yaml

Key Helm Chart Values

mode: "daemonset"  # or "deployment"
presets:
  logsCollection:
    enabled: true
  hostMetrics:
    enabled: true
  kubernetesAttributes:
    enabled: true
  kubeletMetrics:
    enabled: true
useGOMEMLIMIT: true
resources:
  limits:
    cpu: 500m
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 256Mi

Gotchas

• Collector tail_sampling at end of pipeline doesn't release inflight buffer — drop decisions made late still hold memory; OOM under load.
• Auto-instrumentation + manual instrumentation can double-count traces — pick one strategy per service or de-dupe explicitly via sampler.
• Context propagation: HTTP B3 vs W3C traceparent headers don't auto-convert; mixed environments silently break trace continuity.
• OTel SDK vs Collector protocol versions: minor version mismatches can drop attributes silently (especially semantic-convention attributes).
• Resource detection adds platform attributes (cloud.account.id, host.id) that bloat traces — disable when not used for routing.
• Batch processor + memory limiter ordering: limiter must precede batch in the pipeline or memory pressure causes batch drops without backpressure.