An AI agent’s bill rarely breaks because of one spectacular model call. It erodes through thousands of ordinary steps: planning, repeated tool calls, bloated context, observability, and evaluations that never appear in the headline token rate. Deloitte estimated that 25% of companies already using generative AI launched agentic AI pilots or proofs of concept in 2025, a share it expects to reach 50% by 2027. That makes cost control an operating requirement, not a finance exercise. This guide turns model tokens, GPU time, retries, tracing, and evals into an auditable monthly budget, with OpenTelemetry, Langfuse, and LiteLLM forming a practical measurement stack.
Why This Matters in 2026
First, agent adoption is multiplying model interactions. A request that looks like one conversation may trigger planning, retrieval, several tool calls, validation, and a final response. A ten-step trajectory can therefore consume ten model invocations before a customer sees an answer. Deloitte’s forecast of rapid agentic AI pilot growth suggests these patterns will move from experiments into production workloads. Gartner also expects more than 40% of agentic AI projects to be canceled by the end of 2027, citing cost, unclear value, and risk controls among the reasons.
Second, model pricing is fragmented. For a 2026 planning baseline, OpenAI lists GPT-5.2 at $1.75 per million input tokens and $14 per million output tokens for short-context workloads. Anthropic’s Sonnet 4.5 is $3 and $15 respectively, while Gemini 2.5 Flash starts at $0.30 per million input tokens and $2.50 per million output tokens. Using one blended rate for all three hides which workflow is actually expensive. Note: Anthropic has deprecated Claude Sonnet 4.5, which retires on November 30, 2026; Claude Sonnet 5.5 ($2 input, $10 output, 1M context) is the recommended replacement, so treat the Sonnet 4.5 figures above as a legacy baseline rather than a planning target.
Third, GPU capacity introduces a different cost shape. An AWS p5.48xlarge Linux instance in us-east-1 now lists at about $55.04 per hour on demand—down from roughly $98.32 earlier in 2026—with eight NVIDIA H100 GPUs. Keeping that instance available for all 720 hours in a 30-day month costs about $39,629 before storage, networking, orchestration, or engineering time.
Finally, measurement has become billable infrastructure. OpenTelemetry and LiteLLM are open-source, but traces still need somewhere to live. Langfuse’s published tiers are Hobby (free, 50,000 units a month), Core at $29 per month, and Pro at $199 per month, with the paid plans including 100,000 units a month before metered overage. The practical question is no longer whether an agent uses AI, but which calls, tools, and infrastructure paths make each successful outcome expensive.
Key Concepts, Explained Simply
LLM token costs are charged for different token classes. Input tokens include the system prompt, conversation history, retrieved documents, and tool output sent to the model. Output tokens cover generated text and tool arguments. Providers may also bill cached input, reasoning, or long-context tokens separately. The advertised input and output rates are therefore starting points, not a complete invoice.
An agent run is the correct unit for budgeting. A run, or session, includes every model call, tool call, retry, and validation attempt required to reach an outcome. Pricing per API request makes short and long workflows look similar; pricing per successful run exposes the difference.
A GPU-hour is compute rented or reserved for model inference. One instance-hour may contain one GPU or eight, so compare the complete instance rate rather than a nominal GPU price. Utilization matters just as much: a fast GPU running at 10% capacity still consumes the same billed hour.
AI FinOps means assigning measurable cost to models, workflows, teams, and customers. That requires a trace to connect model usage with the run that caused it. OpenTelemetry supplies a vendor-neutral data model, LiteLLM can centralize calls and spend rules, and Langfuse can store traces and evaluations.
The following is a quick role map for the main cost-control components:
| Tool | Primary role | 2026 cost behavior | Best fit | Main caveat |
|---|---|---|---|---|
| OpenTelemetry | Standardized traces, metrics, and logs | SDK is free; storage and collectors are not | Custom or multi-vendor stacks | It is instrumentation, not a complete cost dashboard |
| LiteLLM | Model gateway, routing, budgets, virtual keys | Open-source proxy is free; provider fees remain | Multi-model production traffic | Custom prices can become stale |
| Langfuse | LLM tracing, prompt review, and evals | Free and paid hosted tiers | Agent sessions and evaluation workflows | Trace granularity can affect billing |
| LangSmith | Tracing, evaluation, and LangChain tooling | Developer allowance plus metered plans | LangChain-centered teams | Usage can be difficult to forecast |
| OpenLLMetry | Automatic LLM instrumentation using OpenTelemetry | SDK is free; backend costs remain | Rapid OpenTelemetry adoption | Generic metrics still need business labels |
| Arize Phoenix | Open-source tracing and evaluation | Software is free; self-hosting is not | Privacy-sensitive or self-hosted deployments | The team maintains the infrastructure |
Build a Production AI-Agent Cost Model
Step 1: Define the unit economics
Start with the unit of value, normally a completed agent run rather than an API request. Export at least one week of production data, or instrument a staging workload if none exists. For every run, record:
- Number of model and tool calls
- Input and output tokens per call
- Cache-hit rate and model identifier
- Retries, timeouts, and validation failures
- Final status, latency, and business outcome
For this guide, assume 10,000 successful sessions per month, eight model calls per session, 1,600 input tokens per call, and 500 output tokens per call. That produces 80,000 model calls, 128 million input tokens, and 40 million output tokens each month.
Do not rely only on averages. Segment sessions by workflow and outcome. A research agent that makes 30 calls per run should not be averaged with a three-call classification agent. Record p50 and p95 usage because the expensive tail can dominate the invoice.
Context control also reduces waste. Explicit schemas and constrained outputs can prevent malformed tool arguments and validation retries; structured prompting practices such as the AI Director Mode solution are useful when an agent must follow a predictable sequence.
Step 2: Price model calls and retries
Use the actual model for each stage, then calculate the monthly token bill:
input cost = input tokens ÷ 1,000,000 × input price
output cost = output tokens ÷ 1,000,000 × output price
total model cost = input cost + output cost
At GPT-5.2’s standard short-context price, the example workload costs:
128M input tokens × $1.75/M = $224
40M output tokens × $14.00/M = $560
base model cost = $784/month
Add retries as a separate multiplier rather than hiding them in the token average. If 12% of calls are repeated because of timeouts, schema errors, or transient provider failures, the additional model spend is $94.08. A retry can also charge a browser, search, payment, or CRM API a second time, so its fully loaded cost can be higher.
LiteLLM can enforce routing and budget guardrails at the gateway:
model_list:
- model_name: production-standard
litellm_params:
model: openai/gpt-5.2
input_cost_per_token: 0.00000175
output_cost_per_token: 0.000014
litellm_settings:
max_budget: 1000
budget_duration: 30d
success_callback: ["langfuse"]
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
Set max_budget as an emergency ceiling rather than the normal operating budget. Attach request tags such as team, workflow, tenant, and environment; otherwise the dashboard will show what the system spent without showing which business unit caused it.
Step 3: Convert GPU time into a comparable ceiling
Self-hosted inference makes sense when utilization, model control, or predictable high-volume demand justifies the operational overhead. It does not make sense simply because a GPU’s hourly rate appears lower than an API’s per-token rate.
For the AWS p5.48xlarge example:
720 instance-hours × $55.04/hour = $39,629/month
A complete GPU budget must add CPU and RAM, container orchestration, model storage, object storage, network egress, idle capacity, monitoring, and engineer time. A useful formula is:
GPU cost = instance-hours × rate
+ CPU/RAM
+ storage and egress
+ idle-capacity reserve
+ engineering time
Benchmark the workload rather than relying on a vendor’s maximum tokens-per-second figure. Batch size, prompt length, output length, concurrency, and latency targets all affect throughput. Measure the tokens your agent actually generates at the concurrency your customers require.
A hybrid architecture is often more practical: use hosted APIs for normal traffic, reserve GPU capacity for a stable high-volume workflow, and route overflow back to the API. The goal is not maximum self-hosting; it is the lowest reliable cost per successful run.
Step 4: Add the operating costs hidden by token pricing
A complete agent budget should account for several additional categories:
- Tracing and logs: One trace per agent session is usually more useful than one trace per model call. Grouping related calls also prevents Langfuse trace volume from expanding with every internal step.
- Evaluation: Regression tests consume tokens even when they never reach production. OpenAI’s Batch API offers a 50% discount for eligible asynchronous workloads, which is appropriate for scheduled evaluation sets.
- External tools: Browser automation, search, maps, payments, vector databases, and SaaS APIs often charge per action or request.
- Context retrieval: Large retrieved documents increase input tokens even when only a few passages matter.
- Operational infrastructure: Containers, queues, databases, object storage, secrets, security scanning, and network egress all contribute.
- Human review: Escalations and compliance checks consume labor that rarely appears in an AI invoice.
For the example workload, assume a $29 Langfuse Core plan, a $120 monthly tool allowance, and an $80 orchestration and storage baseline. A 2,000-case evaluation set using 1,500 input and 500 output tokens per case requires 3 million input and 1 million output tokens. At GPT-5.2 pricing, that is $19.25 per full evaluation pass, or approximately $9.63 through Batch API.
Step 5: Instrument, attribute, and enforce the budget
OpenTelemetry should define the common trace structure. At minimum, capture these GenAI span attributes:
gen_ai.operation.name
gen_ai.request.model
gen_ai.response.model
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
A collector can standardize the data before it reaches Langfuse or another backend:
exporters:
otlp/langfuse:
endpoint: ${env:LANGFUSE_OTLP_ENDPOINT}
headers:
Authorization: "${env:LANGFUSE_AUTH}"
processors:
memory_limiter:
limit_mib: 512
batch: {}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp/langfuse]
Use the endpoint and credentials shown in the current Langfuse project rather than committing secrets to configuration. LiteLLM should sit in front of model providers and attach the same team and workflow labels. Langfuse then joins those calls to the parent agent session and records evaluation outcomes.
Create alerts at three levels: 50% of the monthly allocation for investigation, 80% for routing changes or approvals, and 100% for hard limits. Review daily spend by workflow, but review monthly spend by customer or business unit. A sudden daily increase may be an incident; a gradual monthly increase is usually a product or pricing change.
Common Mistakes & How to Avoid Them
Using one average token price. An average can hide a small set of expensive models, long prompts, and failed runs. Price each model and workflow separately, then calculate a weighted total from observed traffic.
Treating retries as harmless. A retry can duplicate model, tool, and side-effect charges. Give agents a retry budget, cap repeated attempts, use idempotency keys for write operations, and send exhausted workflows to a dead-letter queue.
Recording unlimited traces and prompts. Full traces improve debugging but can increase storage, create high-cardinality metrics, and expose sensitive data. Redact secrets and personal information, sample successful low-risk runs, and retain verbose traces for errors or sampled sessions.
Optimizing GPU utilization without measuring quality. High throughput is not the objective if routing changes reduce answer accuracy. Benchmark cost per successful task, include evaluation runs in the denominator, and route only traffic whose quality test passes.
Cost & Effort Reality Check
For the 10,000-session example, the preliminary monthly budget is:
- Base model tokens: $784
- 12% model retries: $94.08
- Langfuse tracing: $29
- Evaluation passes: $19.25
- External tool allowance: $120
- GPU overflow: $360 for 120 hours at $3 per GPU-hour
- Orchestration and storage: $80
- Direct-cost subtotal: $1,486.33
- 15% price, traffic, and failure reserve: $222.95
- Recommended monthly ceiling: $1,709
Base model calls represent about 53% of direct spend in this scenario. The other 47% comes from retries, infrastructure, tracing, evaluation, tools, and GPU overflow—exactly the costs that a simple token calculator omits.
At one million sessions per month, linear scaling produces roughly $170,000–$175,000 per month, or about $2.1 million annually. That estimate still requires a repriced observability tier and should not be treated as a vendor quote.
GPU procurement changes the scale dramatically. A continuously available p5.48xlarge costs approximately $39,629 for 720 hours—about 23 times the entire $1,709 monthly agent budget above. For most early-stage systems, a hosted-model or hybrid architecture is more capital-efficient.
Implementation effort is also real. A useful first version—one gateway, tagged requests, session-level traces, a basic evaluation set, and budget alerts—typically takes three to five engineer-days. Maintaining it requires roughly two to four hours per week for cost review, anomaly investigation, and evaluation updates.
Getting Started This Week
-
Establish a seven-day baseline. Export model calls from production or staging. Record tokens, models, tools, retries, latency, and final success status. Calculate cost per completed run, not cost per request.
-
Create one trace hierarchy. Use OpenTelemetry attributes for models and token usage, route requests through LiteLLM, and store one Langfuse trace per agent session. Add
team,workflow,tenant, andenvironmenttags before dashboards are built. -
Set a budget and test it. Enter the measured inputs into the formula above, reserve 15%, and configure alerts at 50%, 80%, and 100%. Run a small evaluation set before changing models or routing rules, then repeat the test against the new configuration.
FAQ
How much does it cost to run AI agents monthly?
A focused pilot can fit within a few hundred dollars, while an early production workflow often costs $1,000–$5,000 per month. Agent-heavy products with extensive tool use, long contexts, and high evaluation volume can reach tens of thousands of dollars; the run-level breakdown is more useful than a generic market average.
Should I pay for tokens or GPU hours?
Pay for tokens when traffic is variable, model quality matters more than hardware control, or usage is below the break-even point for self-hosting. Consider GPU hours when a stable workload sustains high utilization and the engineering, security, and reliability costs are already covered.
How can I reduce costs without reducing quality?
Route simple stages to cheaper models, cache reusable context, cap unnecessary tool loops, remove irrelevant retrieved documents, and use Batch API for asynchronous evaluations. Measure quality after every change so lower token consumption does not conceal a lower success rate.
Are OpenTelemetry, Langfuse, and LiteLLM required?
No. They solve complementary problems: OpenTelemetry standardizes telemetry, LiteLLM centralizes model calls and budget rules, and Langfuse provides an LLM-focused trace and evaluation interface. Teams with established equivalents can keep those systems, provided model usage can still be joined to business outcomes.
The most effective AI FinOps practice is simple: measure the completed run, include every retry and supporting service, and revise the budget as the agent’s behavior changes. Treat the first month as an instrumentation phase rather than an optimization phase; reliable attribution comes before meaningful savings.
Disclosure: This article may contain affiliate links. We may earn a commission at no extra cost to you.
Last Updated: November 1, 2026 | Specs and prices subject to change. Please verify current pricing on Amazon.
Pingback: Prompt Caching and Context Windows in 2026: Cut Costs