LLM Observability with OpenTelemetry: Instrumenting AI Workloads the Cloud-Native Way

If you already run OpenTelemetry for your services, you don't need a new telemetry stack for LLMs. You need the same tracer, the same collector, the same Grafana dashboard — pointed at a new model of work. The recent GenAI semantic conventions let spans, metrics, and logs describe what an LLM actually did: which model, which prompt, how many tokens, how long, what it cost, and which tool it called.

Why OpenTelemetry is the right substrate for LLM telemetry¶

LLM workloads share the same cloud-native shape as everything else: a request comes in, it fans out to tools and providers, it returns a streaming response, and you need to know when it broke. The plumbing that already collects traces from your HTTP services, queues, and databases should collect traces from your LLM calls. That gives you three things for free:

• Correlated traces across the stack. An inbound HTTP span can be the parent of a chat gpt-4o span, which can be the parent of a vector_search span. The whole request timeline is one trace.
• Vendor neutrality. Span data is OTLP. Span data you can send to Honeycomb, Grafana Tempo, Datadog, New Relic, SigNoz, or any other backend. You don't lose portability by going deeper.
• Existing evaluation pipelines. Service-level objectives, error budgets, and on-call rotations already work on top of OTel data. An LLM trace is just a span with extra attributes.

The cost of ignoring this is the trap most teams fall into: a separate Langfuse or Phoenix tenant for AI testing, a separate Datadog view for production, and a CSV export that gets reconciled in a notebook every Monday. That breaks the moment a multi-step agent does a tool call that hits your existing microservice.

The GenAI semantic conventions in 2026¶

The GenAI semantic conventions used to live inside open-telemetry/semantic-conventions. They moved to a dedicated repo — open-telemetry/semantic-conventions-genai — and the old registry entries now carry a deprecation banner pointing to the new location.⁵

OpenTelemetry Python 1.44.0 released on 2026-07-16, and the OpenTelemetry Collector v0.157.0 shipped on 2026-07-21. Semantic conventions are at v1.43.0 (2026-07-03).¹²³

The actual vocabulary is small and focused. The OpenAI span spec calls out these required attributes.⁴

Two things to notice:

1. Costs are not attribute-shaped. The spec records token counts; you compute cost in your backend. Stable token counts let you change pricing or vendor without changing instrumentation.
2. PII is opt-in. The full prompt and response go behind gen_ai.input.messages and gen_ai.output.messages, which require explicit enablement. Put privacy policy at the same place as the decision to record structured logs.

The deprecation table on the old gen-ai.md registry page is also worth a look. gen_ai.usage.prompt_tokens and gen_ai.usage.completion_tokens are gone. Modern instrumentations emit input_tokens and output_tokens. If you're staring at a dashboard full of zeros, you're probably still on the old attribute names.

What every LLM span should look like¶

Here's a minimal but spec-compliant span for a non-streaming chat call.

from opentelemetry import trace
from opentelemetry.semconv.trace import SpanAttributes

tracer = trace.get_tracer("genai.chat")

with tracer.start_as_current_span("chat gpt-4o") as span:
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.provider.name", "openai")
    span.set_attribute("gen_ai.request.model", "gpt-4o")
    span.set_attribute("gen_ai.request.temperature", 0.2)
    span.set_attribute("gen_ai.request.max_tokens", 512)

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2,
        max_tokens=512,
    )

    span.set_attribute("gen_ai.response.id", response.id)
    span.set_attribute("gen_ai.response.model", response.model)
    span.set_attribute("gen_ai.response.finish_reasons", [response.choices[0].finish_reason])
    span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)
    span.set_attribute("gen_ai.usage.cache_read.input_tokens", getattr(response.usage, "cached_tokens", 0) or 0)

Two patterns that matter even before instrumentation:

• Span name is {operation} {model}. This is what the spec calls out, and it makes a Tempo or Honeycomb search trivially useful: chat gpt-4o versus chat Qwen3-8B.
• Always set gen_ai.response.model. The model you requested and the model that actually served the request can differ. GPT-4o requests often land on gpt-4o-2024-08-06 or a newer snapshot. Costs and behavior differ with that suffix.

Streaming responses: the time-to-first-chunk metric¶

Streaming is the default for chat UIs. The slowest signal you need is the time between sending the request and the first byte coming back. The spec reserves gen_ai.response.time_to_first_chunk for that.⁴

import time

start = time.perf_counter()
first_chunk_at = None
final = None

with tracer.start_as_current_span("chat gpt-4o-mini") as span:
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.request.model", "gpt-4o-mini")
    span.set_attribute("gen_ai.request.stream", True)

    stream = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        stream=True,
    )

    for chunk in stream:
        if first_chunk_at is None:
            first_chunk_at = time.perf_counter()
            span.set_attribute(
                "gen_ai.response.time_to_first_chunk",
                round(first_chunk_at - start, 3),
            )
        if chunk.choices and chunk.choices[0].delta.content:
            yield chunk.choices[0].delta.content
        final = chunk

    span.set_attribute("gen_ai.usage.output_tokens", getattr(final.usage, "completion_tokens", 0) if final else 0)

If you only instrument one thing for a chat UI, capture TTFT. It tells you whether the user is waiting on your inference stack, the network, or the model. It is also the only number that lets you tell a slow generation from a slow UX.

Auto-instrumentation for the boring 80%¶

Don't write the span above for every provider. Auto-instrumentation packages offload this work to the OpenTelemetry SDK. The opentelemetry-instrumentation contrib repository ships packages for OpenAI, Anthropic, Bedrock, Vertex, and the major vector databases. The community wrapper traceloop-sdk bundles them and adds extras; its 0.62.1 release shipped 2026-06-28.⁶

Two lines gets you most of the way:

from traceloop.sdk import Traceloop
Traceloop.init()

After that, every call through openai, anthropic, boto3 (Bedrock), langchain, llama-index, and the supported vector DB clients is wrapped in spans with the GenAI attribute names. The export target is set with environment variables that OTel collectors already understand.

export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel-collector.observability.svc:4317"
export OTEL_SERVICE_NAME="chat-api"
export TRACELOOP_BASE_URL="https://otel-collector.observability.svc:4317"

These environment variables are the same ones your existing services already use. The Traceloop wrapper does not pull in a separate backend.

The collector pipeline for LLM traffic¶

The OpenTelemetry Collector v0.157.0 has GenAI-aware processors and exporters out of the box.² A practical pipeline for an LLM service looks like this.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Redact prompt/response bodies by default.
  transform/redact:
    trace_statements:
      - context: span
        statements:
          - replace_all_patterns(attributes["gen_ai.input.messages"], "/.*/", "[REDACTED]")
          - replace_all_patterns(attributes["gen_ai.output.messages"], "/.*/", "[REDACTED]")
          - replace_all_patterns(attributes["gen_ai.system_instructions"], "/.*/", "[REDACTED]")

  # Roll up token counters into a metric stream.
  metrics/genai:
    metrics:
      gen_ai.client.token.usage:
        enabled: true
        attributes:
          - gen_ai.request.model
          - gen_ai.provider.name
          - gen_ai.token.type

  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlp/tempo:
    endpoint: tempo-distributor.observability.svc:4317
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform/redact, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [metrics/genai, batch]
      exporters: [prometheus]

The transform/redact block is the part most teams miss. The semantic conventions mark prompt content as opt-in, but most libraries default to recording it. Add a redaction stage at the edge of the collector so production spans never carry user input unless you've made a deliberate policy decision.

Compute cost from telemetry, not from invoices¶

The OTel attributes don't include a dollar amount. That's the right design. Pricing changes. Vendor pricing differs across SKUs. Token counts are stable.

The cheapest way to convert a token stream into a cost stream is a Prometheus recording rule that combines your token counter with a price map.

# prometheus rules
groups:
  - name: genai.cost
    interval: 30s
    rules:
      - record: genai:cost_usd_per_minute
        expr: |
          sum by (gen_ai_request_model, gen_ai_provider_name) (
            rate(gen_ai_client_token_usage_sum{gen_ai_token_type="input"}[5m]) * on(gen_ai_request_model) group_left(input_price_per_million) genai_pricing
            +
            rate(gen_ai_client_token_usage_sum{gen_ai_token_type="output"}[5m]) * on(gen_ai_request_model) group_left(output_price_per_million) genai_pricing
          )

genai_pricing is a small file-provider metrics target that you own. Update it when Anthropic ships a new model. The math stays in one place.

What to alert on¶

LLM observability is not alert fatigue. The signals that matter are the same ones that matter for any service.

• P99 latency per model, per gen_ai.operation.name
• Error rate with error.type breakdown (rate limit, timeout, content filter)
• Token cost per request outliers — these catch runaway prompts and stuck loops
• Cache hit rate for cache_read.input_tokens — drops mean you're paying for redundant context
• Time to first chunk P95 — UX-facing, tied to the SLA your frontend team committed to
• Tool call rate for agentic workloads — spikes usually mean a prompt regression

Span data also gives you a debug surface you couldn't build with metrics alone. A user reports that the chat was slow yesterday at 14:23, you filter service.name = chat-api and gen_ai.request.model = "gpt-4o" and http.status_code = 200, you pull the trace, and you see the agent spent 11 seconds inside a retrieve_documents tool call. That story is hard to tell from a counter.

What to evaluate, not just observe¶

Tracing tells you what happened. Evaluation tells you whether it was good. OpenTelemetry carries both. The gen_ai.evaluation event is a spec-defined event you can attach to a span to record a judge score, a human rating, or a heuristic check.⁷

from opentelemetry import trace

with tracer.start_as_current_span("chat gpt-4o") as span:
    response = call_model(messages)
    span.add_event(
        "gen_ai.evaluation",
        attributes={
            "gen_ai.evaluation.name": "answer_faithfulness",
            "gen_ai.evaluation.score.value": 0.92,
            "gen_ai.evaluation.score.label": "good",
            "gen_ai.evaluation.explanation": "Answer grounded in retrieved context.",
        },
    )

Now your backend can graph faithfulness over time, group by prompt version, and trigger a rollback when a release drops the score.

A practical first pass¶

If you have nothing, this is the order I'd add things.

• Deploy or point at an existing OTel Collector and a trace backend.
• Add traceloop-sdk to one LLM service. Initialize before any imports of the LLM client.
• Wire OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME env vars.
• Add the transform/redact processor to the collector.
• Build one dashboard: latency P50/P95/P99 by model, tokens per minute by model, error rate by error.type.
• Set one alert: latency P95 > your SLA for 5 minutes.
• Add gen_ai.evaluation events to your offline eval pipeline.

Skip the AI-specific observability vendor for now. The OTel stack you already have is enough.

Summary¶

• OpenTelemetry's GenAI semantic conventions now expose model, token, latency, and tool attributes on standard spans.
• Auto-instrumentation covers OpenAI, Anthropic, Bedrock, Vertex, and the major vector DBs out of the box.
• The same collector, the same trace backend, the same dashboards — no separate SaaS.
• Treat prompt content as opt-in and redact at the collector edge.
• Compute cost from token counters, not from vendor invoices.
• Latency, error rate, TTFT, cache hit rate, and tool-call rate are the alert tiers that actually matter.
• Trace into your existing microservices, not away from them.

The size of the LLM doesn't change the rules of telemetry. It changes the attributes.

Questions or discussion? Connect on LinkedIn, X, or email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.