~/rena
← blog
ScalabilityCost OptimizationLLMInfrastructure

The Real Cost Curve of LLM Inference at Scale

Per-token prices have collapsed. Production AI bills haven't. Here's where the money actually goes, and the four levers that moved it on systems I've worked on.

RRena Thomas·

Per-token prices have fallen by more than an order of magnitude in two years. Every quarter brings a cheaper, faster, more capable tier. And yet nearly every team I've worked with has an AI bill that grew instead of shrinking.

This isn't a paradox. It's Jevons paradox with extra steps. Cheaper inference means more inference: more agent turns, longer context, more retries, more speculative calls, more features that were uneconomical last year. The unit cost fell and the unit count exploded.

Here's what I've learned about where the money actually goes and what moves it.

Your bill is mostly input tokens

The first thing to measure, and almost nobody does: your input/output token ratio.

On the agent systems I've instrumented, input tokens outnumber output tokens by somewhere between 20:1 and 100:1. A voice agent turn might generate 40 tokens of speech from 8,000 tokens of context — system prompt, tool schemas, retrieved documents, conversation history. Every single turn re-sends nearly all of it.

That ratio is the whole story. Teams optimize output — shorter responses, tighter formatting, smaller models — and move maybe 3% of spend. Meanwhile the system prompt they've never audited has grown to 4,000 tokens of accumulated instructions, half of which contradict each other.

Instrument first:

logger.info('llm.call', {
  model,
  inputTokens: usage.input_tokens,
  cachedTokens: usage.cache_read_input_tokens ?? 0,
  outputTokens: usage.output_tokens,
  latencyMs,
  purpose,        // 'router' | 'agent_turn' | 'summarize' | 'eval'
  sessionId,
})

The purpose field is what makes this actionable. "We spend $40k a month on LLMs" is not a fact you can act on. "62% of spend is agent turns, 19% is a router that could run on a small model, 11% is retries, 8% is summarization" is a roadmap.

On one system that breakdown showed 11% of total spend was failed calls being retried. Nobody had any idea.

Lever 1: Prompt caching

The highest-leverage change available, and it's mostly free.

If your system prompt and tool schemas are stable across turns — and they almost always are — cached input tokens cost a fraction of fresh ones. The requirement is that the cached prefix be byte-identical and at the front of the context.

Which means the single most common mistake is putting something volatile early:

// Cache-hostile: timestamp changes every call, invalidating everything after it
const prompt = `Current time: ${new Date().toISOString()}
${SYSTEM_PROMPT}
${TOOL_DOCS}
${history}`

// Cache-friendly: stable prefix first, volatile content last
const prompt = `${SYSTEM_PROMPT}
${TOOL_DOCS}
${history}
Current time: ${new Date().toISOString()}`

I have found a cache-invalidating timestamp, request ID, or user greeting at the top of the prompt on every production system I've audited. Every one. It's the first thing I look for now, and fixing it has cut input costs by 60–80% more than once.

Structure your context in stability order: static instructions, then static tool definitions, then slowly-changing session facts, then the volatile turn. That ordering is a cost architecture decision, not a formatting preference.

Lever 2: Route by difficulty

Most requests don't need your best model. On a typical agent workload, my rough split is:

  • ~60% trivially classifiable — routing, intent detection, yes/no extraction, formatting
  • ~30% ordinary work a mid-tier model handles fine
  • ~10% genuinely hard — multi-step reasoning, ambiguity, high-stakes output

Sending 100% to the frontier tier is the default and it's expensive. The counter-move is a cheap classifier in front:

const tier = await classifyDifficulty(request)   // small, fast, cheap model
const model = { simple: SMALL, standard: MID, hard: FRONTIER }[tier]

Two caveats I'd insist on. First, the router itself must be cheap — a router that costs as much as the call it's routing is theater. Second, you cannot do this without evals. Downgrading a model without measuring quality is how you save 40% on inference and lose more than that in customer churn. Routing is a cost lever that only exists if you already have a quality measurement.

Lever 3: Attack retries and failures

Retries are invisible spend. A call that times out at 30 seconds and retries twice cost you 3× and delivered nothing.

What worked:

  • Tighter timeouts with a fallback model, rather than long timeouts with retries. If the frontier model hasn't responded in 8 seconds, a mid-tier answer now beats a better answer in 40 seconds — especially in voice, where latency is the product.
  • Validate before you call. A malformed request that the provider rejects is pure waste. Cheap local validation catches most of it.
  • Cap the agent loop. An agent that can call tools indefinitely will occasionally find a cycle and burn thousands of dollars overnight. Hard turn limits with a graceful bail-out. Ask me how I learned this.
  • Alert on cost per session, not total spend. Total spend rises for good reasons. Cost per session rising 4× overnight is a bug, and you want to know at 2am, not on the invoice.

Lever 4: Don't call the model at all

The cheapest inference is the inference you skip.

A surprising fraction of LLM calls in production systems are doing work that isn't language work. Extracting a date. Checking whether a string is a valid ID. Deciding which of four states a record is in. These get implemented as LLM calls because the LLM is right there and it works, and they stay that way.

Audit for it. On one project, replacing three "classification" calls with about 60 lines of ordinary code cut a pipeline's cost by a third and its p99 latency by more than half. It also made that part of the system deterministic and testable, which was worth more than the money.

Adjacent win: semantic caching for repeated questions. In any support or knowledge-base workload, the head of the query distribution is brutally thin. A few hundred questions cover a large share of traffic. Cache those answers with an embedding-similarity lookup and a short TTL, and you serve the common case in tens of milliseconds for approximately nothing.

The number that actually matters

Not cost per token. Not cost per call. Cost per successfully resolved user outcome.

A cheap model that fails half the time and escalates to a human costs far more than an expensive model that resolves the request. A pipeline that saves 30% on inference and adds 2 seconds of latency may lose you more revenue than it saves.

Track cost against the outcome, per session, segmented by workload. Then optimize. Almost every genuinely bad AI cost decision I've seen came from optimizing a metric one level too low.

The through-line with reliability and evals is the same: you can't manage what you don't instrument. Cost is just another production signal, and it's the one most teams are flying blind on.

← all posts