How to · Building

Rate Limiting AI Endpoints: Count Dollars, Not Just Requests

By Ghiles Asmani, founder of Weckr · Published August 15, 2026

Short version: classic per user rate limiting counts requests, and AI breaks that assumption because one request can cost 100 times another. The design that works is three layers at your proxy: a request rate limit for bursts, input and output bounds for worst case request cost, and the layer most apps skip, a monthly spending limit per user in dollars, enforced before the call, with downgrade to a cheaper model as the default action instead of a wall.

Why request counting fails for AI

A rate limit of 60 requests an hour reads as fair until you price the requests. A user sending short questions to a budget model costs fractions of a cent per call; a user sending 100k token documents to a premium model costs dollars per call. Same request count, three orders of magnitude apart in cost. Request limits still matter as burst protection, but the quantity your business needs bounded is spend, and spend per user is only known if you track it per call, the foundation from tracking AI costs per user.

The three layers

  • Layer 1, burst control: a token bucket per user id on the endpoint, generous enough that humans never see it, tight enough that scripts do. This is stock middleware in most frameworks.
  • Layer 2, shape bounds: cap accepted input length and set max_tokens per feature, which caps the worst possible cost of any single request. Without shape bounds, layer 1 is limiting a number that does not correlate with damage.
  • Layer 3, the spend limit:month to date cost per user checked against their plan’s budget before each call, with two actions: downgrade to a cheaper model, or block. This is the layer that maps limits to money, detailed in per user spending caps for LLM apps.
// layer 3 with Weckr: the check runs before the call, cached per user
const result = await wk.chat(openai, {
  model: 'gpt-5.4',
  messages,
  userId: user.id,
  feature: 'analysis',
  plan: user.plan,          // caps are configured per plan in the dashboard
});
// over cap + action "downgrade": call proceeds on the cheaper model
// over cap + action "block": throws WeckrCapError before any spend

Design details that decide whether it works

  • Enforce before, account after. A limit checked after the provider call is a report, not a limit. The check belongs between auth and the provider request.
  • Degrade before blocking. Downgrade keeps paying customers moving while bounding cost; blocks are for abuse. The commercial difference is churn versus a footnote, per model downgrade on budget.
  • Fail open on your limiter’s outages.If the spend check service is unreachable, allow the call: a limiter that can take your product down has inverted its purpose. Weckr’s check behaves exactly this way by design, per what an AI cost tracker sees and never sees.
  • Add velocity alarms. Limits bound the damage; velocity alerts (tokens per user per minute) tell you a runaway is happening while it happens, per AI agent loop detection.

FAQ

How do I rate limit my AI endpoints per user?

Three layers at your backend proxy, where every AI request passes with a user identity: a request rate limit per user (a token bucket keyed on user id), request shape bounds (input length and max_tokens), and a spending limit per user per month in dollars. The layers catch different failures: bursts, oversized single requests, and slow expensive accumulation respectively.

Why are request rate limits not enough for AI endpoints?

Because AI requests have wildly asymmetric costs: one user’s 60 requests an hour can be short prompts costing a cent total, another’s can carry 100k token contexts costing dollars each. Counting requests treats them identically. Dollar denominated limits, cost so far this month per user, are the limit that actually protects your margin, with request limits as the burst guard in front.

Should hitting a spending cap block the user or degrade the service?

Degrade first, block as the last resort. Swapping the user to a cheaper model at the cap keeps the feature alive while bounding the bleed, and most users never notice. Hard blocks are for free tier abuse and genuine runaway patterns. The order matters commercially: a paying customer hitting a wall mid workflow is a churn event, one quietly moved to a budget model is a footnote.

Where should AI rate limiting live in my architecture?

At the same chokepoint as authentication, your backend endpoint in front of the provider call, because it is the only place with the user identity, before money is spent. Provider side limits protect the provider from you in aggregate, and gateway tools limit by key or team. Only your proxy can limit by your end user, and checks must run before the call, since after is just accounting.

Is there a service that handles per user AI spending limits?

Weckr does the dollar denominated layer as a service: you set monthly caps per plan (block or downgrade), the SDK checks the user’s month to date spend before each call and enforces the action, with results cached so the check adds no meaningful latency. It fails open on network errors so an outage of the service can never take your endpoints down. The request rate layer stays in your stack, most frameworks have it a middleware away.

Keep reading

Per User Spending Caps for LLMs

Cap spend per user, not just per account.

The API Key Proxy Pattern

Never call OpenAI from the frontend. Twenty lines fix it.

Detect and Stop Agent Loops

Catch a runaway agent before it burns your budget.

Layers 1 and 2 are middleware. Layer 3 is Weckr.

The dollar denominated layer needs live per user spend, current prices, and pre call enforcement, which is a service, not a middleware. Weckr ships it with the tracking you need anyway: caps per plan, downgrade or block, velocity alerts, free for 50,000 requests a month. See caps firing on the live demo, or start with the AI cost and margin guide.

See the dashboard with real data, no signup needed.

Try the demo →