How it works

How a request becomes a bill, step by step

Here's one press of the help-desk button. The model reads the whole prompt before the first word comes back and then writes the answer token by token. The bill counts input, output and cached tokens separately.

1. Request sent. The app assembles the prompt: system instructions, tool definitions, the ticket history and any retrieved documents. Caching only works on an identical prefix, so the stable parts go at the start. In Datadog's data, system prompts make up 69% of input tokens. What can go wrong: the prompt exceeds the context window and the call fails; a new model generation counts the same text as about 30-35% more tokens, so a "same price" upgrade can cost more.
2. Queued. The provider admits the request after checking the account's tier, spend cap and rate limits (requests and tokens per minute, per model), then assigns a service tier (standard, priority, flex or batch) and a region. What can go wrong: the request is Throttled, the exception on the diagram: a 429 error that says "retry later". Anthropic returns the same 429 at its monthly spend cap, where retrying won't help until the 1st of next month. Failed requests still count toward OpenAI's per-minute limit, so retries without a random delay keep the bucket empty.
3. First token. In prefill, the model reads every input token in one parallel pass and builds a working memory of the prompt (the KV cache). Prefill is bound by raw compute and sets the time to the first token. A cached prefix is skipped, which is faster and cheaper. What can go wrong: a cache miss after someone edits the system prompt brings back the full price and wait; a reasoning model at its highest effort can think for minutes, so client timeouts set lower kill work you still pay for.
4. Streamed. In decode, the model writes one token per step, re-reading its weights and the cache from memory each time, which is why decode is limited by memory bandwidth and output tokens cost more. A reasoning model writes hidden "thinking" tokens before the visible answer, and they're billed as output. What can go wrong: the answer hits max_tokens and comes back incomplete, possibly after you've paid for thousands of reasoning tokens; the connection drops. The other exception, Failed over, is the gateway switching to another model or host, which it can do cleanly only before the first token arrives.
5. Billed. The provider meters uncached input, cache writes, cache reads, output (reasoning included), service tier and region, prices each line separately and bills prepaid credits, an invoice or a cloud marketplace. What can go wrong: the gateway, the provider's console and the cloud invoice disagree.

That's where the diagram's note comes from. On OpenRouter, the average prompt grew from about 1,500 tokens to more than 6,000 between 2024 and late 2025, and completions from about 150 to 400. Reasoning models and agents use many more tokens per task, which matters more than the price per token.

A few terms carry most of the explanations in this space:

• Why output costs more. Most labs price output at four to six times input (five times at Anthropic, OpenAI's gpt-6.1-sol and Gemini 3.8 Flash). Decode is slower per token than prefill, but the exact multiple is a pricing choice and isn't a measured hardware ratio.
• Reasoning tokens. OpenAI's docs say they're invisible, billed as output and range from a few hundred to tens of thousands per request. By late 2025 reasoning models produced more than half of OpenRouter's tokens.
• Batching and the speed trade. Serving engines such as vLLM, SGLang and Nvidia's TensorRT-LLM pack many users' requests onto the same GPUs. More users per GPU means cheaper tokens and slower answers for each. In SemiAnalysis's InferenceX benchmark, serving one open model about three times faster per user cost about four times as much per token, which is the economics behind "fast" tiers at twice the list price.
• Prompt caching. A repeated prefix costs a tenth of the input price or less to read, and on most Claude models cached reads don't count toward the per-minute input limit either. Yet only 28% of the LLM calls in Datadog's data showed any cache reads.
• Quantization. Storing a model's numbers at lower precision (8 or 4 bits instead of 16) makes it smaller and faster. On Nvidia's newest GPUs, 4-bit DeepSeek-R1 ran 2.2-3.5 times faster than 8-bit. Nvidia reports accuracy losses of about 1% or less on most tasks. It's also why "the same model" differs between hosts.

Cheat sheet: ways to buy inference