Beyond the stack trace: observability and verification for async AI agents in production¶

Your agent traces are beautiful and they still won't save you at 2 AM. You can see every LLM call, every token, every tool invocation in Langfuse — and you still can't prove what the agent decided, why, or whether it was allowed to. Observability tells you what happened; it doesn't prove it happened the way it was supposed to.

That gap is the new runtime problem, and the industry noticed this week: Diagrid shipped Verifiable Execution in Dapr 1.18 on June 11, and The New Stack ran a piece arguing agent verification is a runtime concern, not a test-time one.¹²

This post is for platform teams who've already shipped Kubernetes and OpenTelemetry and are now being asked to run agents that mutate production. If "we have OTel traces" is your answer to "what did the agent do," this is the gap you haven't closed yet.

First, scope it — because most of this is overkill for most agents. If your agent is read-only or advisory (it summarizes, drafts, retrieves, recommends, and a human acts), stop at Layer 1: OTel traces and Langfuse are enough. You only need the durability and proof layers when the agent takes side-effecting actions on its own — moves money, mutates records, calls tools that change production state. The dividing line is autonomy over irreversible effects, not how "agentic" it feels. Everything below assumes you're on the wrong side of that line.

The failure mode metrics can't see¶

A normal microservice fails loudly. It returns a 500, throws an exception, blows an SLO, and your dashboards light up. An agent fails quietly: it returns a confident, well-formed, completely wrong answer — and every metric stays green.²

Here's the kind of incident that ruins a Tuesday:

An async support agent runs as a queue worker. A customer asks for a refund. The agent retrieves the wrong order from the vector DB (embedding collision), calls the issue_refund MCP tool for $4,200 instead of $42, the tool succeeds, the workflow completes, the span is green, and the trace shows a perfect 200. Six hours later, finance asks why.

Now the on-call engineer opens the trace. They can see the execute_tool span fired. They can see issue_refund returned success. What they cannot see: which document the agent retrieved, what the model actually reasoned over, whether the $4,200 came from the customer or a hallucination, and — critically — whether this agent was even authorized to issue a refund above $100 without a human.

The trace captured the call. It didn't capture the decision, the state the decision was made on, or the authority the agent acted under. Those three are what you need to reconstruct, replay, and defend an agent's behavior. None of them live in a span by default.

Observability vs. verification¶

These are different problems with different tooling. Conflating them is why teams think they're covered when they aren't.

Observability is necessary and not sufficient. You need both layers. The rest of this post is how to build the second one without throwing away the first.

Anatomy of a verifiable agent run¶

Before tooling, get the data model right. A run is "verifiable" when you can take its record, replay it, and arrive at the exact same decisions — and prove the record wasn't altered. That requires capturing six things, not one.

The first three are an observability concern — OTel and Langfuse handle them well. The last three are a verification concern, and they're where most stacks have nothing. Notice the ordering: each layer depends on the one before it. There's no point signing a history you can't deterministically replay.

Determinism is the whole game¶

Replay only works if the run is deterministic — feed the same recorded inputs back in and you must get the same decisions out. Agents are aggressively non-deterministic by default: model sampling, wall-clock timestamps, random retries, now() in a prompt, unordered tool fan-out.

The fix is the same pattern durable-workflow engines have used for a decade: record-and-replay. You don't re-run the side-effecting work on replay — you replay the recorded results of it. The model call, the vector query, the tool invocation: each is recorded once, and on replay the engine returns the stored result instead of calling out again. The agent's control flow re-executes; its effects do not. That's what makes a crashed agent resumable and a past agent auditable with the same machinery.

The practical stack: three layers¶

Don't replace your tracing. Stack verification on top of it. Three layers, each with a job the layer below can't do.

┌─────────────────────────────────────────────────────────┐
│  Layer 3 — PROOF      Dapr 1.18 Verifiable Execution     │
│  signs history, propagates lineage, attests authority    │
├─────────────────────────────────────────────────────────┤
│  Layer 2 — DURABLE    Dapr Workflow                      │
│  event-sourced history, deterministic replay, crash-safe │
├─────────────────────────────────────────────────────────┤
│  Layer 1 — TRACE      OpenTelemetry GenAI + Langfuse     │
│  spans for every agent step, tool call, and LLM call     │
└─────────────────────────────────────────────────────────┘

Layer 1 — Trace with OTel GenAI semantic conventions¶

The OpenTelemetry GenAI semantic conventions are now the lingua franca for agent telemetry. Use them instead of bespoke attribute names so every backend (Langfuse, Datadog, Honeycomb) understands your spans.³ The convention models an agent run as a top-level invoke_agent span with child chat spans (one per LLM call) and execute_tool spans (one per tool).

from opentelemetry import trace

tracer = trace.get_tracer("refund-agent")

with tracer.start_as_current_span("invoke_agent") as agent_span:
    agent_span.set_attribute("gen_ai.operation.name", "invoke_agent")
    agent_span.set_attribute("gen_ai.agent.name", "refund-agent")
    agent_span.set_attribute("gen_ai.conversation.id", conversation_id)

    # The LLM decision — capture the evidence AND the output
    with tracer.start_as_current_span("chat") as chat_span:
        chat_span.set_attribute("gen_ai.operation.name", "chat")
        chat_span.set_attribute("gen_ai.request.model", "claude-opus-4-8")
        chat_span.set_attribute("gen_ai.usage.input_tokens", usage.input_tokens)
        chat_span.set_attribute("gen_ai.usage.output_tokens", usage.output_tokens)
        chat_span.set_attribute("gen_ai.response.finish_reasons", ["tool_calls"])
        # The retrieved evidence the model reasoned over — the part traces usually drop
        chat_span.set_attribute("rag.document_ids", retrieved_doc_ids)
        chat_span.set_attribute("rag.scores", retrieved_scores)

    # The side-effect — capture args AND an idempotency key
    with tracer.start_as_current_span("execute_tool") as tool_span:
        tool_span.set_attribute("gen_ai.operation.name", "execute_tool")
        tool_span.set_attribute("gen_ai.tool.name", "issue_refund")
        tool_span.set_attribute("gen_ai.tool.call.id", tool_call_id)
        tool_span.set_attribute("app.refund.amount_usd", amount)
        tool_span.set_attribute("app.idempotency_key", idem_key)

The two attributes that turn a trace into evidence are rag.document_ids/rag.scores (what the model saw) and app.idempotency_key (so the side-effect is replay-safe and de-duplicated). Standard auto-instrumentation captures neither. Add them by hand.

Pipe these to Langfuse for the human-friendly view — it groups spans into a session timeline and lets you eyeball the prompt, the retrieved context, and the tool args side by side. That's your observability. It is not your proof: Langfuse data is sampled, mutable, and retention-limited. Good for debugging, useless for an audit six months later.

Layer 2 — Make the run durable and replayable with Dapr Workflow¶

Wrap the agent loop in a Dapr Workflow. The workflow function is your orchestration logic; each side-effecting step is an activity. Dapr persists an event-sourced history: after every activity completes, the result is checkpointed. If the pod dies mid-run, the workflow resumes by replaying history up to the last checkpoint and continuing — no double-charged refunds, no lost state.¹

import dapr.ext.workflow as wf

wfr = wf.WorkflowRuntime()

@wfr.workflow(name="refund_agent")
def refund_agent(ctx: wf.DaprWorkflowContext, order_query: str):
    # Each call is recorded; on replay these return stored results, not re-runs
    context = yield ctx.call_activity(retrieve_context, input=order_query)
    decision = yield ctx.call_activity(llm_decide, input=context)

    # Verification gate BEFORE the side-effect
    if decision["amount_usd"] > 100:
        approval = yield ctx.wait_for_external_event("human_approval")
        if not approval["approved"]:
            return {"status": "rejected", "reason": "exceeded auto-approve limit"}

    result = yield ctx.call_activity(issue_refund, input=decision)
    return {"status": "completed", "result": result}

@wfr.activity(name="issue_refund")
def issue_refund(ctx, decision: dict):
    # Side-effecting MCP tool call — runs exactly once, recorded in history
    return mcp_client.call_tool("issue_refund", decision, idempotency_key=decision["idem_key"])

Two things this buys you that a bare agent loop can't:

• The wait_for_external_event gate is a first-class verification checkpoint. The workflow durably suspends — for minutes or days — until a human approves. No polling, no lost state, no cron-job hack. The refund above $100 cannot fire until someone signs off, and that suspension is part of the permanent record.
• Deterministic replay is free. Because activities are recorded, you can replay any past run from its history and watch the exact decision sequence unfold. That's your audit reconstruction and your crash recovery, from one mechanism.

API note: the Dapr Workflow authoring model (call_activity, wait_for_external_event, event-sourced history) is stable and current. Pin your dapr runtime and SDK versions and check the Dapr Workflow docs before copying — the SDK surface still moves between releases.

Layer 3 — Prove it with Dapr 1.18 Verifiable Execution¶

Durable history answers "what happened." It does not prove the history wasn't edited, or that the agent was authorized to act. That's the gap Dapr 1.18 closes with three new capabilities, all built on SPIFFE workload identity.¹

The framing Diagrid uses is worth stealing: SPIFFE answers "who are you?"; Verifiable Execution answers "how did you get here?"¹ For an agent that moves money, "how did you get here" is the compliance question. Attestation is what stops a caller from hitting the issue_refund activity directly and skipping the > $100 approval gate — the activity can demand proof it was invoked from inside the signed workflow path.

This is where the audit trail becomes tamper-evident rather than merely present. A signed, propagated, attested history is something you can hand to an auditor and a court, not just an on-call engineer.

Maturity flag: Verifiable Execution shipped June 11, 2026 and the CNCF announcement is conceptual — no code samples yet.¹ Treat the exact configuration surface as unstable; prototype against the Dapr 1.18 release notes and Diagrid Catalyst docs, and don't put it on the critical path of a launch until you've run the signing/attestation flow end-to-end yourself. Dapr 1.18 also graduated the Jobs API to stable and made component hot-reloading GA — both relevant if you schedule recurring agent runs.

The pattern: audit trail as a revenue asset¶

Most teams treat audit logging as a cost — a compliance tax you pay grudgingly. Flip it: the agent's verifiable history is the single artifact three different orgs will pay for, and you only have to capture it once. Capture with that audience in mind and the same record serves debugging, compliance, security forensics, and cost attribution.

The unlock is that one well-designed record satisfies all four. The fields overlap almost entirely — inputs, identity, decisions, costs, signatures. Design the schema for the strictest consumer (legal) and the others get their answers for free.

Concretely, emit one signed, append-only record per agent step:

{
  "run_id": "wf-9f3a...",
  "step": 3,
  "agent": "refund-agent",
  "identity": "spiffe://corp/ns/payments/sa/refund-agent",
  "inputs": { "query": "...", "retrieved_doc_ids": ["doc_881"], "scores": [0.71] },
  "decision": { "tool": "issue_refund", "amount_usd": 42, "finish_reason": "tool_calls" },
  "side_effect": { "idempotency_key": "ref_2026...", "result": "ok" },
  "policy": { "gate": "auto_approve_under_100", "passed": true },
  "cost": { "input_tokens": 1840, "output_tokens": 96, "usd": 0.0021 },
  "prev_hash": "sha256:...",
  "signature": "..."
}

The prev_hash chains records into a tamper-evident log (each record commits to the previous one); the signature is what Dapr's history signing provides at the workflow level. This is the difference between "we have logs" and "we can prove it." When the $4,200 refund happens, you don't argue from green dashboards — you replay run wf-9f3a and show, signed, exactly which document the agent retrieved and which gate it should have hit.

Kubernetes-side concerns¶

Agents don't run in a notebook. They run as async workers on your cluster, and the verification stack has real deployment shape.

Sidecar the runtime, sidecar the tools¶

Dapr runs as a sidecar already — that's how the workflow engine, state store, and pub/sub reach your agent without SDK sprawl. Put MCP tool servers in the same pod as sidecars too, so tool calls are a localhost hop the runtime can observe and attest, not an opaque egress to some external service.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: refund-agent
spec:
  template:
    metadata:
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "refund-agent"
        dapr.io/enable-api-logging: "true"   # surface Dapr API calls in logs
    spec:
      containers:
        - name: agent
          image: registry/refund-agent:1.4.0
          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://otel-collector:4317"
        # MCP tool server as a sidecar — localhost, observable, co-scheduled
        - name: mcp-tools
          image: registry/mcp-payments:2.1.0
          ports:
            - containerPort: 8808

Co-scheduling the MCP server matters for verification specifically: a tool call over localhost:8808 stays inside the pod's identity boundary, so the SPIFFE identity attached to the workflow is the identity that made the tool call. Egress to an external tool API breaks that chain — now you're trusting a network boundary instead of a cryptographic one.

Expose agent state the way you expose pods¶

On-call shouldn't grep JSON to learn what a running agent is doing. Dapr's workflow management API exposes run state directly — query it like any other cluster resource:

# Inspect a live or completed run by instance ID
curl localhost:3500/v1.0/workflows/dapr/wf-9f3a/...

# Or from inside the cluster, through the sidecar
kubectl exec deploy/refund-agent -c daprd -- \
  curl -s localhost:3500/v1.0/workflows/dapr/<instance-id>

Wire that into whatever pane of glass your team already uses — a Headlamp panel, a Backstage plugin, a Grafana datasource. The goal: the 2 AM engineer answers "what is this agent doing and what has it done" without reading a single raw log line. A run that's durably suspended on a human-approval gate should look suspended in the UI, not look hung.

Dead-letter queues are agent incidents, not infra noise¶

When an async agent poisons a message and it lands in the DLQ, that's not a transient blip to retry blindly — it's a decision that failed. Treat DLQ depth for agent topics as a first-class alert, and make every DLQ entry carry its run_id so you can pull the signed history immediately. An agent that silently retries into a DLQ and gives up is exactly the quiet failure metrics won't show you.

What verification costs you¶

This isn't free, and pretending otherwise is how you end up with a verification stack nobody maintains. Be honest about the bill before you sign up for it.

The rule: match the rigor to the blast radius. A $4,200 refund justifies all three layers. An internal agent that drafts Jira tickets does not — give it Layer 1 and move on. Over-verifying a low-stakes agent burns the same engineering budget you'll wish you had when a high-stakes one ships.

Checklist: is this agent observable enough to ship?¶

Ten yes/no questions. If you can't answer yes to all ten, you can't reconstruct or defend the agent's behavior — don't put it in front of production side-effects yet.

• Replay — Can you take a past run's record and deterministically replay it to the same decisions?
• Evidence — Do you capture the retrieved context (doc IDs + scores), not just the final answer?
• Decisions — Is every LLM output recorded verbatim, with its finish_reason?
• Side-effects — Does every tool call carry an idempotency key and a recorded result?
• Durability — Does the run survive a pod crash mid-execution without double-firing side-effects?
• Gates — Are high-risk actions blocked behind a durable approval checkpoint, not a soft prompt instruction?
• Identity — Is every step tied to a workload identity (SPIFFE or equivalent)?
• Integrity — Is the history signed or hash-chained so tampering is detectable?
• State visibility — Can on-call see a running agent's state without reading raw logs?
• DLQ wiring — Does every dead-lettered message carry its run_id and fire an alert?

The first four are observability you can build today on OTel + Langfuse. The middle four are durability and verification — Dapr Workflow plus 1.18's signing and attestation. The last two are the operational glue that makes the other eight usable at 2 AM.

Summary¶

Observability and verification are two layers, not one. Tracing tells you what your agent did; verification proves it did the right thing, under the right authority, on evidence you can replay. For a read-only agent, Layer 1 is plenty. The moment an agent mutates production, you need all three.

The one-sentence test before you ship: if your agent issued a wrong $4,200 refund at 2 AM, could you replay the exact run, show the signed evidence it reasoned over, and prove which gate it should have hit? If yes, ship it. If no, you have observability — you don't yet have verification, and for an agent that touches prod, that's the half that matters.

Sources¶

• The agent-failure framing ("a confident, well-formed, completely wrong answer") and the runtime-verification argument come from The New Stack.²
• OTel GenAI agent-span conventions (invoke_agent, chat, execute_tool, and gen_ai.* attributes) are experimental as of mid-2026 but already supported by major backends.³

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.