Debugging LLM Apps in 2026: Tracing Agent Runs Like a Pro

THE AI EDIT14 MIN READ

Ask any team shipping an AI agent in 2026 what broke last week and they will usually describe a symptom, not a cause: the refund agent “hesitated,” the research bot “made something up,” the support triage loop “went quietly.” In 2026 reviews of production agent incidents, roughly 72% of root causes turned out to be prompt, retrieval, or tool-wiring failures — not model outages and not provider bugs. A single 2026 agent run averages seven model calls and five tool calls, producing roughly a dozen spans of context you have to reconstruct by hand. Traditional request logs show you the HTTP call. They never show the decision chain. Tracing fixes that, and the tooling finally makes it tractable.

Why This Matters in 2026

The debugging surface changed shape. Here are the trends that make tracing non-optional now:

  • Agents are the unit of debugging, not requests. A 2026 trace corpus shows a median production agent run issuing 7 model calls, 5 tool calls, and 2 retrieval calls. Any workflow where step four depends on what step two returned means a single HTTP error log tells you almost nothing.
  • Incidents are orchestration failures. In 2026 incident write-ups, an estimated 72% of agent failures originated in prompt drift, retrieval misses, or incorrect tool arguments. Model-provider outages accounted for a low single-digit share.
  • Context windows broke the old debugging habits. Production windows now run 200K–1M tokens. A 600K-token prompt can silently truncate, get summarized by an intermediary, or blow past a per-request cost guardrail with no error raised. Failures now look like “the agent ignored me.”
  • Span economics matured. OpenTelemetry’s GenAI semantic conventions now define 85 GenAI attributes, and all of them remain marked “Development.” Since May 2026 the GenAI conventions ship from their own repository (open-telemetry/semantic-conventions-genai) rather than the core semantic-conventions repo, so vendor attribute sprawl is still real. You need a convention policy, not a default.

There is a second-order effect for teams assembling agents on platforms rather than in code. Many 2026 stacks are configured inside best no-code AI agent builders 2026, which means you may not control the runtime. Tracing becomes the only way to see that a connector fired twice or that a retrieved document set was empty.

Key Concepts, Explained Simply

Trace. One end-to-end user request and everything it caused. Concretely: a trace for “Refund request #4417” contains spans for classifying intent, retrieving the refund policy, calling the issue_refund tool, calling the payments API, and composing the reply.

Span. One timed step inside a trace, with a parent-child relationship and attributes. Spans are the unit of cost in most backends, and the unit of evidence when something goes wrong.

Semantic conventions. A shared vocabulary — gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.tool.name — so a trace emitted by a Python service lines up with one emitted by a TypeScript service or a third-party proxy. Without conventions, every dashboard becomes bespoke.

Replay. Re-running a captured trace with identical inputs to see whether the failure reproduces. This is how you separate “the model behaved differently this time” from “step three was always broken.”

Step 1: Instrument With OpenTelemetry Before You Debug Anything

Start with the data layer, not the UI. OpenTelemetry (OTel) is the layer all four tools in this workflow share: Langfuse, Arize Phoenix, and Helicone all ingest or export OTLP spans, so you can change backends without rewriting instrumentation.

Install the SDK plus auto-instrumentation for your model and agent frameworks. The opentelemetry-instrumentation-openai, -anthropic, -langchain, -crewai, and -haystack packages (from the OpenLLMetry project) cover most stacks in an afternoon. Manual spans handle the rest.

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(
    resource=Resource.create({
        "service.name": "support-agent",
        "service.version": "2026.03.4",
        "deployment.environment": "prod",
    })
)
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4318/v1/traces"))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("acme.support_agent", "1.4.0")

Then export through a collector rather than pointing SDKs directly at a vendor. The collector is where you enforce sampling, redaction, and attribute normalization — three things you cannot retrofit later.

receivers:
  otlp:
    protocols:
      http: {}
processors:
  tail_sampling:
    decision_wait: 5s
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-runs
        type: latency
        latency: {threshold_ms: 8000}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 10}
exporters:
  otlphttp:
    endpoint: https://ingest.your-backend/v1/traces
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlphttp]

Tail sampling beats head sampling for agent workloads: you keep 100% of errors and slow runs while retaining 10% of everything else, typically cutting backend span bills by roughly 90%. The collector itself runs in 150–300 MB of RAM with batching enabled, and the decision_wait of five seconds adds negligible latency because exporters are asynchronous. One caveat: tail sampling requires a stateful collector, so use a single replica or a load-balancing exporter until you need to scale out.

Step 2: Design Spans Around the Questions You Will Actually Ask

Span design is debugging design. Before writing code, write down the five questions you expect to ask during an incident, then name spans to answer them.

Typical questions: Which model call produced this text? Which tool ran, with what arguments? What did retrieval return? Which prompt version was live? What did the run cost?

A workable hierarchy for a support agent:

with tracer.start_as_current_span("execute_tool refund_issue") as span:
    span.set_attribute("gen_ai.operation.name", "execute_tool")
    span.set_attribute("gen_ai.tool.name", "refund_issue")
    span.set_attribute("gen_ai.tool.call.id", call_id)
    result = issue_refund(order_id, amount)
    span.set_attribute("gen_ai.tool.result.status", result.status)
    if not result.ok:
        span.set_attribute("error.type", result.error_code)

Use invoke_agent for the root span, chat for model calls, execute_tool for tools, and a retrieve span for RAG with attributes like app.rag.top_k and app.rag.hit_count. Auto-instrumentation handles the chat spans; you almost always need to write tool spans by hand, because framework helpers rarely know which business action a function represents.

Two rules keep traces readable and cheap. First, never dump full document bodies or model responses into span attributes — the OpenTelemetry SDK’s default limit is 128 attributes per span, and backends bill by span size and count. Store payloads in Langfuse, Phoenix, or Helicone (which have proper blob storage) and put only a hash or URI on the span. Second, keep nesting shallow and meaningful. A run with seven model calls and five tools produces around 12–14 spans; at 1M runs per month that is 14M spans, and span count is the line item that decides your bill.

Always propagate W3C traceparent headers across HTTP and MCP boundaries. A trace that breaks at a service hop produces orphaned spans, which is the most common reason a team’s timeline “looks empty” right when it matters.

Step 3: Get Token Accounting and Cost Attribution Right

Token accounting is where tracing becomes a business tool rather than a debugging tool. Record, on every model span: gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and — where your provider exposes it — cached-input counters. Attribute naming for cache-read tokens has shifted between convention revisions, so pin your own field name in a collector transform rather than relying on the upstream attribute to be stable.

Then attribute cost. In 2026 pricing, a representative run with 18,400 input tokens and 2,150 output tokens at $2.50/M input and $10/M output costs roughly $0.07. At 100,000 runs per month that is about $6,750. A single runaway retry loop that triples a run is visible immediately in per-trace cost, and invisible in aggregate request counts.

Prompt caching makes this sharper. Cached prefixes now cut repeat-input spend by 50–90% depending on provider and cache hit rate, which means “tokens per trace” without a cached/uncached split is a misleading metric. Langfuse and Phoenix both surface per-span and per-trace cost; Helicone aggregates cost by model and session at the proxy layer, which is useful when a single logical run is split across many HTTP clients.

Put three numbers on a dashboard before anything else: median cost per trace, p95 cost per trace, and the ratio of cached to uncached input tokens. In most 2026 deployments, the first thing teams discover is that 20–40% of spend is repeat context that should have been cached, or a tool loop that no one bounded.

Step 4: Replay Runs and Isolate Root Cause

A trace tells you what happened. Replay tells you what will happen again. This is the step that separates a fix from a guess.

Capture. Langfuse v3’s @observe decorator gives you input/output capture, prompt versioning, and dataset creation with minimal code:

import os
from langfuse import Langfuse, observe

lf = Langfuse(
    public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
    secret_key=os.environ["LANGFUSE_SECRET_KEY"],
)

@observe(name="support-agent-run", capture_input=True, capture_output=True)
def handle_refund(user_id: str, order_id: str) -> dict:
    policy = retrieve_policy(order_id)
    result = issue_refund(order_id, policy.amount)
    return {"refund": result.amount, "policy": policy.id}

handle_refund("u_8821", "ord_4417")
lf.flush()

Compare. Langfuse datasets and experiments let you re-run a set of captured traces against a new prompt version and compare outputs side by side. Phoenix offers the same capability through OpenInference auto-instrumentation plus its evaluation API, and it is strong for tree/graph visualization of multi-agent flows:

import phoenix as px
from phoenix.evals import llm_classify

result = llm_classify(
    data=spans,
    name="tool_argument_valid",
    prompt="Did the agent call {tool} with valid arguments? {tool_arguments}",
    llm=llm,
)

Sessionize. Helicone works as a drop-in proxy with zero code changes, and its session header groups a multi-step run into one view:

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url="https://oai.helicone.ai/v1",
    default_headers={
        "Helicone-Auth": os.environ["HELICONE_API_KEY"],
        "Helicone-Session-Id": session_id,
    },
)

Classify the cause. In 2026 practice, agent failures cluster into five buckets: prompt drift or version mismatch, retrieval miss (the document was never indexed, or top-k was too low), tool contract error (wrong argument names or types), context overflow or silent truncation, and nondeterminism (unseeded sampling, temperature above zero, live data changes). The first four are visible in a trace. The fifth is only visible in a replay.

A trace shows you that the instruction was ambiguous; it does not tell you how to rewrite it. That is where deliberate prompt structure pays off — treating the model as a crew you direct rather than a slot machine you interrogate. The AI Director Mode solution lays out that framing, and it pairs well with tracing because explicit intent produces traces that are far easier to interpret.

Step 5: Close the Loop With Evals, Alerts, and Change Control

Tracing only pays off if findings become durable. Three mechanisms do this.

Evals tied to traces. Turn every confirmed failure into a labeled example in a dataset (Langfuse datasets, Braintrust, Promptfoo). Then run the dataset on every prompt or model change. A regression suite of 30–50 real failures catches far more than unit tests that assert a string appears in the output.

Alerts on agent-specific signals. HTTP 5xx rate is a poor proxy for agent health. Alert on: p95 end-to-end trace latency, tool-call failure rate, median tool calls per run (a sudden rise means a loop), cost per trace crossing a threshold, and retrieval hit rate dropping below baseline. Tag alerts with gen_ai.request.model and app.prompt_version so you know whether the regression is a model, a prompt, or your data.

Version everything. Record prompt version, model ID, tool schema hash, and retrieval index build ID on the root span. Model aliases like gpt-5-mini or claude-sonnet-4-5 can change behavior under a fixed name; without the version tags you cannot prove a regression came from the provider. Correlate the trace to your business identifier (a Stripe charge, a Zendesk ticket) so support escalations can be answered in one click.

Common Mistakes & How to Avoid Them

Logging everything into span attributes. Pasting full prompts, retrieved documents, and model responses into attributes violates the 128-attribute default, inflates bills, and quietly ships customer data into your tracing backend. Fix: capture payloads in Langfuse, Phoenix, or Helicone; keep only summaries, hashes, and counts on the span; redact before export, not after.

Breaking the parent-child chain. Creating spans in background tasks or worker threads without carrying the context produces orphaned spans and a flat, unreadable timeline. Fix: use start_as_current_span and pass contextvars explicitly into thread pools; propagate traceparent across every HTTP and MCP hop.

Blaming the model first. “The model hallucinated” is a conclusion, not a diagnosis. Before changing anything, replay the exact trace against the same prompt version twice. If the failure does not reproduce, the cause is usually nondeterminism or live data drift, not a prompt bug.

Sampling away the evidence. Head sampling at 1% drops exactly the rare traces you need, and tail sampling configured to keep only “fast” traces erases timeouts. Fix: always keep 100% of errors, keep slow traces, keep the first trace per conversation, and raise baseline sampling to 10–25% during an active incident.

Cost & Effort Reality Check

Realistic early-2026 numbers:

  • Self-hosting is genuinely free. Langfuse’s core is MIT-licensed open source, Arize Phoenix is Elastic License 2.0 (ELv2) per its official license page, and the OpenTelemetry Collector is Apache 2.0. Phoenix runs comfortably on 2–4 GB of RAM with SQLite or Postgres for a single-team deployment.
  • Hosted tiers are cheap at small volume. Helicone’s free tier covers roughly 10,000 requests per month, with paid plans starting at $79/month (Pro). Langfuse Cloud’s paid tier starts at $29/month (Core) with 100k units per month included, where a unit is one trace, one observation, or one score, and additional units run $8 per 100k. LangSmith’s developer tier covers a few thousand traces per month with paid seats around $39/user/month. Arize’s managed cloud tier is Arize AX rather than Phoenix: its free tier allows 25k trace spans per month, with enterprise pricing quoted separately. Treat all of these as list pricing that shifts — check before budgeting.
  • Managed backends meter volume, and the billing unit differs by vendor. Langfuse bills $8 per 100,000 units, graduating to $7, $6.50, then $6 per 100k at higher volumes, and one unit is one trace, one observation, or one score, so a 14-span run lands at about 15 units, or roughly $0.0012. Arize AX meters span volume directly (25k spans per month on the free tier, 50k on the $50/month plan, custom on Enterprise), while Helicone meters requests plus stored GB instead of spans. Tail-sampling 1M monthly runs down to 10% leaves 100k retained runs, or about 1.5M units, which on Langfuse’s graduated scale runs roughly $107/month in usage charges before the $29/month Core plan, not $300.
  • Engineering time is the real cost. One to two days for credible instrumentation; another week for prompt versioning, token dashboards, and one replay workflow; then ongoing maintenance as conventions change. Tool spend is rarely the constraint — span design discipline is.

Getting Started This Week

Day 1–2: One service, end to end. Add the OTel SDK and auto-instrumentation to a single agent service, ship spans through a collector, and verify them in Phoenix or Helicone. Write down exactly three span types you need: model call, tool call, retrieval. Resist instrumenting everything.

Day 3–4: Cost and one dashboard. Attach model, token, and cache attributes to each model span. Build one view with p95 trace latency, tokens per trace, and cost per trace. Add two alerts: error rate and cost-per-trace threshold.

Day 5–7: Replay workflow. Capture 20 real failures into a dataset, re-run them against one prompt or model change, and diff the results. Ship the winner with a version tag on the root span. That loop — instrument, replay, compare, version — is the whole discipline.

FAQ

Do I still need Langfuse, Phoenix, or Helicone if I use OpenTelemetry? No. OpenTelemetry is the instrumentation and data format; those tools are storage, visualization, and evaluation layers on top of it. You can export OTLP to any backend, or run Phoenix locally and keep everything in-house.

Which one should I pick for a multi-agent system? Phoenix tends to be the fastest way to understand agent graphs because of its tree visualization, Langfuse is strongest for prompt management and dataset-based experiments, and Helicone is the least invasive since it works as a proxy with no code change. Running two is common: one for graph analysis, one for evals.

How should I handle PII in traces? Redact at the SDK or collector layer before data leaves your environment, hash user identifiers, restrict backend access with role-based controls, and set short retention windows on production traces. Also sample rather than storing every run, since deleting after the fact is far harder than never collecting it.

Does tracing slow down my app? With asynchronous batch exporting, the instrumentation overhead is typically under 1–2% of request latency. The collector adds a small queueing delay, and tail sampling adds up to its decision_wait (commonly 5 seconds) before a decision, but that happens off the request path. Users never wait on your tracing backend.

Final Thoughts

Debugging LLM applications stopped being about reading model output and started being about reconstructing a decision chain. The teams that ship reliably in 2026 are not the ones with the best models; they are the ones that can answer “which step, which prompt version, which arguments, how many tokens, how many dollars” for any run in under a minute. Instrument with OpenTelemetry, design spans around the questions you will actually ask, make token and cost accounting mandatory, and turn every failure into a replayable case. The tooling — Langfuse, Phoenix, Helicone, plus your own conventions — is only the multiplier. The discipline of tracing every agent run and fixing the orchestration layer, not the prompt guesswork, is what makes the difference.

Disclosure: This article may contain affiliate links. We may earn a commission at no extra cost to you.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top