Task Cost Benchmark

Most benchmarks compare raw price per million tokens, or they score models on academic tests. Neither answers the question a founder actually has, which is what one feature costs to run. This benchmark measures that. Ten realistic tasks, each written out in full, run against every provider's live API, priced with the same table Weckr uses to recompute cost on every logged call.

The gap between the two questions is wider than it looks. A flat per token rate says nothing about how many tokens a model decides to spend, and models differ enormously on that. Multi turn and agentic features resend their accumulated context on every step, so the same sticker price produces very different bills. Those are the tasks where cost compounds, and they are the tasks most likely to be in the product you are shipping.

The research gap this fills

Two recent papers make the same point from the academic side. CostBench (ACL 2026) argues that current agent evaluations emphasise task completion while overlooking resource efficiency and adaptability, and finds that leading models routinely fail to choose cost optimal plans, with even GPT-5 scoring under 75 percent exact match on the hardest tasks. TerminalWorld (arXiv 2026) documents an efficiency paradox in real terminal work, where success correlates negatively with resource use, reporting r = -0.62 against token usage, because failed attempts burn disproportionately more tokens than successful ones.

Both describe the same blind spot: we measure what a model can do and rarely what it costs to make it do that. This benchmark is the practical, founder facing translation of that concern. It is not a capability ranking and it is not a price list. It asks one question, which is what a feature costs when you actually ship it.

Results

Last run July 23, 2026. Suite version 1.0.0. 40 of 40 task runs completed.

Every figure

What the numbers show

Running the full suite once cost $0.0140 on OpenAI and $0.1484 on Gemini, a spread of 10.6 times for the same ten tasks. The published rates for these 4 models span only 2.0 times on input and 1.9 times on output, so almost none of that spread comes from the sticker price. It comes from how many tokens each model chose to spend.

The clearest example is the classification task. The correct answer is one word, and every model got it right. Gemini spent 198 output tokens producing it, of which 197 were thinking tokens you never see in the response. Thinking tokens are billed at the output rate, so a one word answer is not close to free, and nothing in a per million token price list tells you that.

Compounding is the other effect. The agentic task plans and then executes across four calls, each one resending everything before it, and the multi turn support conversation does the same across three. On Kimi the agentic task costs 4.6 times the single turn support reply on the identical model at the identical rate. A cost model built on price per token will miss that entirely, because the rate never changed.

OpenAI was the cheapest option on 10 of the 10 tasks in this run. The useful reading is not the leaderboard, which a single pricing update can rearrange. It is the size of the gap. Ten identical pieces of work cost 10.6 times more at one end than the other, on models whose published rates differ by far less. That gap is produced by model behaviour rather than by price, and it stays invisible until you measure the task itself.