How do I handle LLM API rate limits (429s) in production?

How do I handle LLM API rate limits (429s) in production? Do four things. Find out which limit the 429 names. Wait at least as long as the Retry-After header says. Add random jitter to every wait, and set a cap on the number of retries. Then stop your own code from sending more than the API allows, with a queue in front of the client. Most guides stop there. I build agents, and agents add a fifth problem. One user action becomes many model calls, and each call can retry. This guide covers the four basic steps and then the agent problem.

How do you handle a 429? The short answer

Which limit did you hit: RPM, TPM or a daily cap?

A 429 does not say “slow down”. It says one of several limits failed. OpenAI lists RPM, RPD, TPM, TPD and IPM. These mean requests per minute, requests per day, tokens per minute, tokens per day and images per minute. You hit a limit when you reach any one of them first. A fix for one limit does nothing for another.

Anthropic measures the Messages API in three parts for each model class: RPM, input tokens per minute (ITPM) and output tokens per minute (OTPM). The 429 error says which one you exceeded. Read the error body before you write any retry code. As an example, the same page lists 1,000 RPM, 2,000,000 ITPM and 400,000 OTPM for Claude Sonnet 5.5 in the Start tier. The Build tier lists 5,000 RPM for the same model. A job with long prompts hits the token limit long before the request limit.

Two details change how you count. Anthropic says that for most Claude models, “only uncached input tokens count toward your ITPM rate limits”. Prompt caching therefore raises your real throughput. For OpenAI, the page says your limit “is calculated as the maximum of max_tokens and the estimated number of tokens”. A large max_tokens value can use capacity that the response never needs. Anthropic says the opposite for output: OTPM counts only the tokens the model produces, so max_tokens is not a factor there.

Short bursts also count. Anthropic says that a rate of 60 requests per minute “might be enforced as 1 request per second”. Thus 60 requests in the first second of a minute can fail, although the minute total is correct. The API uses a token bucket, so capacity returns continuously and does not reset at fixed times.

Should you read Retry-After?

Yes. It is the cheapest correct signal that you get. Anthropic says the header holds “the number of seconds to wait until you can retry the request. Earlier retries will fail”. The header is not on every 429, so your code needs a fallback. OpenAI gives this rule: “Treat this value as a minimum: wait at least that long.” It also says to add a small random delay, so that clients do not retry at the same time.

OpenAI also warns that “unsuccessful requests contribute to your per-minute limit”. A retry that comes too early does not just fail. It uses part of the capacity that you wait for. This is the reason that a tight retry loop makes the problem last longer.

Both providers also send headers on successful responses. OpenAI sends x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens. Anthropic sends anthropic-ratelimit-requests-remaining and matching input and output token headers. Log them. When the remaining value is near zero, your pacing layer can slow down before the first 429.

One more point about SDKs. OpenAI says its official SDKs retry eligible 429 and 503 responses. If you also write your own retry loop, the two loops multiply. Turn off one of them, or count the SDK retries in your budget.

Why does backoff need jitter?

Say that 60 clients fail at the same moment. A fixed wait makes all 60 retry at the same moment again. The server sees the same spike every time. The AWS Architecture Blog studied this in Exponential Backoff And Jitter. Their result: backoff alone “helps a small amount, but doesn’t solve the problem”. The fix is a “small change to the sleep function”. Each client picks a random delay. With 100 contending clients in their simulation, jitter cut the number of calls by more than half compared with no jitter. They call the no-jitter version “the clear loser”.

The blog compares three jitter types. Full jitter waits a random time between zero and the current backoff limit. Equal jitter keeps some of the backoff and randomizes the rest. Decorrelated jitter bases the next limit on the last random value. The blog finds that equal jitter takes much longer than full jitter, and that decorrelated jitter makes slightly more calls. I use full jitter because it is the simplest of the three.

The figure below shows the effect. It is a toy model with made-up numbers.

Sixty clients send one request in slot 0. The API accepts 10 per slot. Start with Retry now and watch the red parts of the bars. In my run, the clients send 210 requests in total to get 60 answers. Fixed wait sends the same 210, but in spikes of 60, 50, 40, 30, 20 and 10 every fourth slot. Backoff + jitter sends 145 requests. After the first spike, the load stays at 30 or less in every slot and ends in 9 slots. In this toy, Retry now finishes first. A real API counts the rejected requests against your limit, as OpenAI says, so that speed is not free.

What does a correct retry function look like?

This is the function that I use. It reads Retry-After, uses full jitter when the header is missing, stops after a fixed number of attempts, and stops after a total time limit.

import random, time

def call_with_retry(send, max_attempts=5, base=1.0, cap=30.0, deadline=60.0):
    start = time.monotonic()
    for attempt in range(max_attempts):
        resp = send()
        if resp.status_code != 429:
            return resp
        header = resp.headers.get("retry-after")
        if header:
            delay = float(header) + random.uniform(0, 1)  # minimum, plus jitter
        else:
            delay = random.uniform(0, min(cap, base * 2 ** attempt))  # full jitter
        out_of_time = time.monotonic() - start + delay > deadline
        if attempt == max_attempts - 1 or out_of_time:
            return resp  # give the 429 to the caller
        time.sleep(delay)

The function returns the last 429 and lets the caller decide. The caller knows if the user can wait. OpenAI asks for the same two limits: “Limit both the number of attempts and the total time spent retrying.” For async code, replace time.sleep with await asyncio.sleep.

How do you stop sending too much before the 429?

Retries repair a failure. A queue stops the failure. Put one limiter in front of every call to a given model, and set it below the published limit. Use a token bucket with the same shape as the provider limit. Take one request token for each call. Take input tokens by an estimate of the prompt size. Release the call when the bucket has enough. Let the bucket refill at a steady speed.

Make the limiter shared. A limit belongs to an organization or a project, not to one process. If you run ten workers that each allow 100 requests per minute, the API sees 1,000. Keep the counter in one place, for example in a gateway or in Redis. I described a gateway for 10,000 requests per second in how to build LLM guardrails. The same place is a good home for rate limits, because every request already passes it.

Increase traffic slowly. Anthropic says that a sharp increase in usage can cause 429 errors from “acceleration limits”. Its advice is to “increase your traffic gradually and maintain consistent usage patterns”. A batch job that starts at full speed can fail at once, although its average rate is below the limit. Ramp it up over a few minutes.

When should you fall back to another key or provider?

Fall back when the first route stays blocked after your backoff has run. Do not fall back at the first 429. A second route has its own limits, its own prices and its own model behavior. If you switch on every error, you move the load and the retry storm to the next provider.

Two kinds of fallback exist, and they are different. A second key on the same organization does not help. Anthropic says limits are set at the organization level, and OpenAI sends project-scoped headers as well. Check whose limit you hit before you add keys. A second provider, or a second model with its own limit, adds real capacity. Anthropic notes that some model families share one pool, for example the combined Opus 4.x bucket, while Opus 5.5 has a separate limit.

Open a circuit when a route fails often. Send no traffic to it for a short time, then test it with one request. This protects the failing route and your own latency. Test the fallback in staging before you need it. A fallback that nobody ran is not a plan.

What do agents change?

An agent turns one user action into many model calls. A chat message makes one call. An agent task reads files, calls tools and plans again. Each of those steps is a model call, and each call can hit a 429. I describe this in the agent loop. The loop runs until the model stops asking for tools.

Retries multiply with the steps. As my own example, take a task with 12 steps and a retry limit of 5 for each call. In the worst case that is 60 calls for one user action. Now run 200 such tasks at the same time. The queue fills with work that the user may never see. The limit stays full, and every new task starts with a 429. I think this is why agent systems fail in a different way from chat apps. They do not fail on one request. They fail on the sum of all the retries.

So put a budget on the run, as well as on the call. Give each run a maximum number of model calls, a maximum number of retries in total, and a maximum wall time. When the run reaches a limit, stop it and return a clear result. I wrote about the same idea for loops in preventing tool-call cycles. There, a hard step budget turns an endless loop into one that is “expensive but finite”. A retry budget does the same for rate limits.

Add three more rules for agents. Give each task a priority, so that a user-facing task goes before a background task when the queue is full. Make the run resumable, so that a stopped run can continue from its last good step and does not start again. Last, make tool calls idempotent. A retried step must not send the same email twice.

What should you monitor?

Track the 429 rate for each model, the retries for each run, the time spent in waits, and the remaining-capacity headers. Alert on a rising 429 rate, before it gets high. A slow rise shows that traffic is near the limit. Anthropic shows hourly peak usage in the Claude Console, so you can compare it with your own logs.

So how do you handle LLM API rate limits (429s) in production?

Read which limit failed and do not retry a spend cap. Wait at least as long as Retry-After, add full jitter, and cap both the attempts and the total time. Put a shared queue in front of the API so that most 429s never happen. Fall back to a second provider only after the first route stays blocked. For agents, add a budget for each run, because one task makes many calls and every call can retry. If you want help with this layer in your own agent system, I take this kind of work, and you can read more at AI agent developer.