Why the debate never resolves
Providers really do change serving stacks, and users really do misremember baselines. Anecdotes cannot distinguish the two: the complainer has no logs, the defender has no logs, and the provider’s benchmarks do not cover your prompts. The only person who can answer “did it get worse for my app” is you, and only if you were already recording. The good news is the recording is cheap and needs no stored outputs.
The four metadata signals
- Output tokens per call, per feature. Your fixed summarizer producing 350 token answers for months, then suddenly 210 or 600, is a behavior change you can date to the day. Length shifts are the loudest cheap signal of upstream changes.
- Latency percentiles per model. p50 and p95 per model, not averaged across models. Serving infrastructure changes tend to show here before anywhere else.
- Retry and regenerate rate. Every user tapping retry is casting a quality vote. A rising regeneration rate on one feature is your users benchmarking the model for you, and it also multiplies cost, the pattern from AI agent loop detection in slow motion.
- Feature abandonment. Weekly active users of the AI feature itself. Quality drops users cannot articulate still show up as quiet disuse.
All four come from metadata: token counts, timestamps, user ids, feature tags. If you already track AI cost per user, you are logging most of this today and the baseline is your existing history.
Provider change or your bug? The two checks
- Scope: upstream changes hit every feature on that model simultaneously, across all users. Your regressions follow one feature.
- Deploy correlation: if the shift lines up with your release, it is your prompt or parameters. If it happened on a quiet Tuesday with no deploy, look upstream. Dated model snapshots, where offered, make this cleaner: pin the version, and a drifting alias cannot migrate you silently.
Degradation is also a billing event
Output bloat is the underrated half. If answers get 30 percent longer for identical prompts, your output spend rises about 30 percent, on the priciest token class, for the same product. On Sonnet 4.6 at $15 per million output tokens, a feature doing a million calls a month at 500 output tokens costs $7,500 in output; at 650 tokens it is $9,750. Nothing in your code changed. This is why the response to confirmed drift is the routing playbook: re test siblings and rivals on quality per dollar and move the feature, the method in model routing for LLM apps, priced with current rates from the LLM pricing JSON feed.
FAQ
Is model degradation real or imagined?
Both happen. Providers do adjust serving infrastructure, quantization, and safety layers, and users also misremember baselines and notice variance more after reading complaints. The way out of the argument is measurement: your own per call telemetry of output tokens, latency, refusal markers, and downstream user behavior, compared against your own historical baseline. Vibes argue, baselines settle.
What metrics reveal LLM quality drops without storing outputs?
Four cheap ones: output tokens per call for a fixed feature (sudden shrinkage or bloat means behavior changed), latency percentiles (serving changes often show here first), retry and regeneration rate (users hitting retry is a quality vote), and feature abandonment (users who stop using an AI feature are telling you something). All are computable from metadata you can log without keeping prompt or response text.
How do I separate a provider change from my own regression?
Check scope and check your diffs. A provider change hits every feature using that model at once, across all users, without a deploy on your side. Your regression follows your deploys and usually hits one feature. If output length shifted on one feature the day you shipped, it is your prompt. If it shifted on every claude-sonnet feature on a quiet Tuesday, it is probably not you.
What should I do if I confirm a model degraded for my use case?
Treat it like a price change: re run the routing decision. Test the same feature on the model’s siblings and rivals, compare quality per dollar rather than quality alone, and move the affected feature. Pinned model versions, where the provider offers dated snapshots, buy you time to evaluate instead of being migrated silently.
Can degradation change my costs even if I change nothing?
Yes, through output length. If a model starts producing longer answers for the same prompts, you pay for every extra output token, at the most expensive token class. A 30 percent output bloat on a $15 per million output model is a 20 to 25 percent bill increase for identical traffic. Watching output tokens per call per feature catches this within days.
Keep reading
Keep the baseline before you need it
The teams who settle degradation questions in five minutes are the ones already logging tokens, latency, and cost per feature when the debate starts. Weckr keeps that baseline as a side effect of cost tracking: output tokens, latency, cost, per user, per feature, per model, from two lines of integration. See the history views on the live demo, or start with the AI cost and margin guide.