How to · Cost

Track LLM Cost Per Prompt Version, Because Rewrites Change the Bill

By Ghiles Asmani, founder of Weckr · Published August 7, 2026

Short version: tag every call with a prompt version label, aggregate cost by it, and treat prompt changes like the billing events they are. Rewrites routinely move output length 20 to 50 percent, output is the expensive token class, and a quality win that costs 40 percent more is a decision you should make on purpose. If the version is in git but not in your telemetry, cost regressions are real but undatable.

Teams eval prompt quality obsessively and discover prompt cost on the invoice. The fix is one string.

The one string that makes cost experimental

Cost tracking already flows through a feature label. Version it:

const result = await wk.chat(anthropic, {
  model: 'claude-sonnet-4-6',
  max_tokens: 600,
  messages,
  userId: user.id,
  feature: 'summary:v3',     // was 'summary:v2' until today
  plan: user.plan,
});

Every call now lands with tokens, cost, latency, and the version that produced it. Cost per version, cost delta at the switch date, and which users saw which version are all queries instead of guesses. The convention that scales: feature:version, stable prefix so feature rollups survive, suffix bumped on anything that could change behavior, including model and temperature, not just prompt text.

Why rewrites move the bill: output drift

Input cost changes from prompt edits are visible and small: you can count the tokens you added. Output drift is the surprise. Asking for “thorough” answers, adding a reasoning instruction, or restructuring toward lists can move mean output length dramatically, and output is priced five to six times input on current mid tier models. Concretely, on claude-sonnet-4-6 ($3 in, $15 out per million), a summarizer at 2,000 input and 400 output tokens costs $0.012 per call; the rewrite that pushes output to 560 tokens costs $0.0144, a 20 percent feature cost increase that no code review would flag. Estimating this before shipping is the method in estimating AI feature cost; version tags are how you verify it after.

A/B testing prompts on quality per dollar

  1. Split traffic between summary:v2 and summary:v3 however you already split experiments.
  2. Measure quality your usual way: evals, thumbs, retry and regeneration rate (retries are both a quality vote and a cost multiplier).
  3. Read cost per call per version from the same window, including the caching effect: a longer stable prompt that caches at roughly a tenth of the fresh input rate can beat a shorter volatile one on measured cost, which token counting alone gets backwards.
  4. Decide on quality per dollar. Two percent better at forty percent more is usually a no; state the trade explicitly and it decides itself.

The same tagging pattern covers model experiments, summary:v3-haiku against summary:v3, which connects to the wider model routing decision, and to detecting model degradation: a per version baseline is exactly what makes upstream drift visible.

FAQ

How do I track LLM cost per prompt version?

Tag every call with a version label alongside your user id, for example feature: "summary:v3", and aggregate cost by that label. Cost per version then falls out of the same pipeline as cost per feature: same token counts, same price math, one extra dimension. The mistake to avoid is versioning the prompt in git but not in the telemetry, which makes cost regressions undatable.

Why did my new prompt version cost more than the old one?

Usually output length. Prompt edits change how much the model says, and output tokens are priced several times above input. A rewrite that added "be thorough" can add 40 percent output tokens, which is a 30 percent cost increase for the feature at identical traffic. Longer system prompts and added few shot examples also raise input cost, but output drift is the dominant surprise.

Should I A/B test prompts on cost as well as quality?

Yes, as one decision. Run both versions on real traffic with distinct version tags, measure quality however you already do (evals, retry rate, user ratings), and read cost per call per version from the same window. The metric that decides is quality per dollar: a version two percent better at forty percent more cost is usually the wrong trade, and you only see it if cost is in the test.

Does prompt caching change which version is cheaper?

It can flip the answer. A longer but stable system prompt that caches well can cost less in practice than a shorter prompt that changes per call and never caches, because cached input tokens are billed at roughly a tenth of the fresh rate. Compare versions on measured cost from real traffic, not on token counts alone.

What is a good tagging convention for prompt versions?

feature:version, kept boring: summary:v3, extraction:2026-08-07, chat:concise-test. Keep the feature prefix stable so rollups by feature still work, bump the suffix on any change that could alter behavior including model swaps and temperature changes, and keep labels out of user reach since they are your analytics grain.

Keep reading

Estimate AI Feature Cost

Forecast cost per call, user, and month before shipping.

Detect Model Degradation From Telemetry

Settle the vibes debate with your own metadata baseline.

Model Routing for LLMs

Send each request to the cheapest model that fits.

Every prompt change is a pricing change

Once versions are in the telemetry, prompt iteration stops being a cost mystery: each rewrite shows its bill next to its quality. Weckr gives you the per feature, per version cost view from the tag you already pass, with per user margin underneath it. See the feature breakdown on the live demo, or start from the AI cost and margin guide.

See the dashboard with real data, no signup needed.

Try the demo →