Source: https://aiprimetech.io/blog/rate-limits-and-retries/ Jun 23, 2026 · 7 min · API # Handling Rate Limits and Retries on LLM APIs A By [AI Prime Tech](https://aiprimetech.io/) · AI-assisted At 09:12 on a Tuesday, our summarization pipeline looked healthy: 96 workers online, median model latency around 2.8 seconds, no elevated 5xx rate. Then a customer uploaded a backlog of 38,000 support tickets. Within four minutes, the pipeline was “down” without any server being down. We were hitting two different limits at once: - requests_per_minute: too many API calls - tokens_per_minute: too much prompt + completion volume The naive retry logic made it worse. Every worker saw 429, slept for one second, and retried together. That created a retry wave, which produced more 429s, which created another wave. The system had accidentally built a synchronized denial-of-service attack against its own quota. Handling LLM rate limits is not just “add exponential backoff.” In production, you need to reason about tokens, queues, concurrency, retries, idempotency, fallback providers, and billing behavior as one system. ## The Two Limits That Matter: RPM and TPM Most LLM APIs enforce some combination of: - RPM: requests per minute - TPM: tokens per minute - RPD / TPD: requests or tokens per day - Concurrent request limits: maximum in-flight calls - Model-specific limits: higher limits for smaller models, lower limits for frontier models The important part: RPM and TPM fail differently. If you send many tiny prompts, you hit RPM first. If you send fewer large prompts, you hit TPM first. Example: ``` Limit: - 600 requests/minute - 300,000 tokens/minute Workload A: - 1,000 short classification calls/minute - 200 tokens each - 200,000 tokens/minute total Result: - TPM is fine - RPM fails Workload B: - 100 long document analysis calls/minute - 5,000 tokens each - 500,000 tokens/minute total Result: - RPM is fine - TPM fails ``` A common gotcha: developers throttle requests but not tokens. That works in staging, where prompts are short. Then production users paste PDFs, chat histories, logs, or legal contracts, and the token budget evaporates. In practice, I treat every LLM call as reserving two resources: ``` cost = { requests: 1, tokens: estimated_prompt_tokens + max_output_tokens } ``` Even if the model returns fewer output tokens than max_output_tokens, reserving the upper bound prevents oversubscription. ## What Actually Happens When You Ignore TPM Suppose you have: ``` TPM limit: 120,000 Average request: 4,000 input tokens + 1,000 max output tokens Reserved cost: 5,000 tokens ``` Your real maximum throughput is not “as many workers as possible.” It is: ``` 120,000 / 5,000 = 24 requests per minute ``` That is one request every 2.5 seconds on average. If you run 100 workers, the first batch may all start successfully if the provider allows burstiness. Then the account hits a hard wall. Every subsequent request gets 429, and retries pile up. The fix is not just lowering worker count. You need a scheduler that understands token reservations. ## A Minimal Token-Aware Rate Limiter For a single process, a token bucket is usually enough. You refill capacity over time and require each request to acquire both a request slot and a token slot. Here is a simplified Python example: ``` import asyncio import time from dataclasses import dataclass @dataclass class Bucket: capacity: float refill_per_second: float tokens: float updated_at: float def refill(self): now = time.monotonic() elapsed = now - self.updated_at self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_per_second) self.updated_at = now async def acquire(self, amount: float): while True: self.refill() if self.tokens >= amount: self.tokens -= amount return missing = amount - self.tokens sleep_for = missing / self.refill_per_second await asyncio.sleep(min(sleep_for, 1.0)) class LLMRateLimiter: def __init__(self, rpm: int, tpm: int): now = time.monotonic() self.request_bucket = Bucket( capacity=rpm, refill_per_second=rpm / 60, tokens=rpm, updated_at=now, ) self.token_bucket = Bucket( capacity=tpm, refill_per_second=tpm / 60, tokens=tpm, updated_at=now, ) self.lock = asyncio.Lock() async def acquire(self, estimated_tokens: int): async with self.lock: await self.request_bucket.acquire(1) await self.token_bucket.acquire(estimated_tokens) ``` Usage: ``` estimated_tokens = prompt_tokens + max_output_tokens await limiter.acquire(estimated_tokens) response = await client.chat.completions.create( model="gpt-5.5", messages=messages, max_tokens=max_output_tokens, ) ``` This is deliberately simple. In a distributed system, use Redis, Postgres advisory locks, or a queue with centralized dispatch. The important principle is the same: throttle on both request count and token budget before you call the API. ## Retries: Backoff Is Necessary but Not Sufficient Retries are useful for: - Transient 429 responses - Network timeouts - Temporary 500, 502, 503, or 504 errors - Provider-side overload - Connection resets Retries are dangerous for: - Validation errors - Context length errors - Authentication failures - Safety policy rejections - Deterministic bad prompts - Non-idempotent tool calls The retry policy should classify errors first. | Error Type | Retry? | Typical Action | 429 rate_limit_exceeded | Yes | Backoff, respect retry headers, reduce concurrency | 500/502/503/504 | Yes | Backoff with jitter, limited attempts | Timeout before response body | Maybe | Retry with idempotency key if supported | Context length exceeded | No | Compress, chunk, or route to longer-context model | Invalid API key | No | Disable key, alert operator | Content/safety rejection | No | Return controlled failure or rewrite request | Tool side effect already executed | No/Maybe | Requires idempotency design The basic backoff shape: ``` import random def retry_delay(attempt: int, base=0.5, cap=30.0): exponential = min(cap, base * (2 ** attempt)) return random.uniform(0, exponential) ``` That random.uniform(0, exponential) is jitter. Do not skip it. Without jitter: ``` 100 workers fail 100 workers sleep 1s 100 workers retry 100 workers fail 100 workers sleep 2s 100 workers retry ``` With jitter: ``` 100 workers fail workers retry across a spread of 0-1s then 0-2s then 0-4s ``` The second pattern gives the provider and your own queue room to recover. ### Respect Retry Headers When Present Many APIs include a hint such as: ``` Retry-After: 12 ``` or provider-specific reset headers. If the API gives you a concrete delay, use it as a floor: ``` delay = max(header_retry_after_seconds, retry_delay(attempt)) await asyncio.sleep(delay) ``` In practice, I also cap total retry time. A user-facing chat request should not quietly retry for two minutes while the user stares at a spinner. A background batch job can wait much longer. Example policy: ``` { "interactive_chat": { "max_attempts": 3, "max_total_retry_seconds": 8 }, "batch_summarization": { "max_attempts": 8, "max_total_retry_seconds": 300 } } ``` ## Queueing Beats Worker Stampedes If you have more work than quota, workers should not all directly call the LLM API. Put a queue in front. A practical architecture: ``` App/API -> Job queue -> Rate-aware dispatcher -> Provider client -> Result store ``` The dispatcher owns: - RPM accounting - TPM accounting - Model selection - retry policy - provider fallback - concurrency limits - queue priority This makes overload visible. Instead of thousands of tasks failing with 429, you see queue depth rising and estimated wait time increasing. For Redis-backed systems, I like separating “ready to run” from “delayed retry” jobs: ``` llm:ready llm:delayed llm:dead_letter ``` A failed job does not immediately return to ready. It gets a scheduled retry timestamp. A small scheduler moves due jobs back into the ready queue. ``` { "job_id": "sum_7f13", "model": "claude-sonnet-4.6", "estimated_tokens": 6200, "attempt": 2, "not_before": "2026-02-03T09:14:22Z" } ``` That not_before field prevents retry storms. ## Concurrency Control Is Not the Same as Rate Limiting Concurrency limits cap in-flight work. Rate limits cap work per time window. You need both. Imagine each call takes 20 seconds and your RPM is 60. A pure RPM limiter permits one request per second. After 20 seconds, you have about 20 concurrent calls. If latency jumps to 90 seconds, the same RPM creates 90 concurrent calls. That may exceed provider or local resource limits. Use a semaphore for concurrency: ``` provider_concurrency = asyncio.Semaphore(20) async def call_model(job): async with provider_concurrency: await limiter.acquire(job.estimated_tokens) return await send_request(job) ``` A common gotcha: acquire the semaphore too early and your workers sit idle holding scarce concurrency slots while waiting for rate tokens. Usually, acquire rate capacity first, then acquire concurrency immediately before the network call. If your limiter wait can be long, avoid occupying worker resources during that wait. ## Graceful Degradation: Spend Quality Where It Matters When rate limits bite, you have options besides failing. For example, a customer support assistant might degrade like this: | Pressure Level | Strategy | User Impact | Normal | Use strongest configured model | Best quality | Mild TPM pressure | Reduce max_tokens, trim retrieved context | Slightly shorter answers | Moderate pressure | Route simple tasks to smaller model | Minimal if classification/routing is easy | High pressure | Queue non-urgent jobs | Delayed background work | Severe pressure | Return cached or template response | Lower quality but available Concrete examples: - Route ticket tagging to Haiku 4.5, Qwen, or a small Llama variant instead of using a frontier model. - Use Sonnet 4.6 for normal reasoning and reserve Opus 4.8 or GPT-5.5 for escalation paths. - Move long-context analysis to Fable 5 or Gemini 3 only when the prompt genuinely needs that context window. - For batch jobs, pause low-priority tenants before impacting interactive traffic. The trade-off is complexity. Every fallback path needs tests, observability, and product acceptance. A worse answer can be more damaging than a delayed answer in some domains. ## Multi-Key and Multi-Provider Fallback Multi-key and multi-provider routing can improve resilience, but it is easy to do badly. There are three separate patterns: | Pattern | What It Solves | Risk | Multiple keys, same provider/account | Operational isolation | May violate provider terms if used to evade limits | Multiple accounts/regions | Fault isolation | More billing and governance complexity | Multiple providers/models | Availability and cost flexibility | Output variance, prompt incompatibility Do not use multiple keys to sneak around limits. Use them for legitimate separation: tenants, environments, products, or billing boundaries. A clean routing policy might look like: ``` { "task": "support_ticket_summary", "primary": { "provider": "anthropic", "model": "claude-sonnet-4.6" }, "fallbacks": [ { "provider": "openai", "model": "gpt-5.5", "when": ["provider_5xx", "rate_limit_after_retry_budget"] }, { "provider": "google", "model": "gemini-3", "when": ["context_length_needed", "primary_unavailable"] } ], "degrade_to": { "provider": "open_model_cluster", "model": "qwen", "when": ["batch_low_priority"] } } ``` If you use an aggregator or a platform like AI Prime Tech for multi-model access, the same engineering rules still apply: track idempotency, budgets, fallback conditions, and final billing semantics explicitly. ## Avoiding Double-Billing on Fallback The hard part of fallback is knowing whether the first request completed. Consider this sequence: ``` Client sends request to Provider A Provider A processes it successfully Network connection drops before client receives response Client retries with Provider B Both providers bill Two completions may exist ``` You cannot fully solve this from the client side unless the provider supports idempotency keys or request lookup. But you can reduce damage. ### Use Idempotency Keys Where Available Generate a stable key per logical operation: ``` import hashlib import json def idempotency_key(task_name, tenant_id, input_payload): canonical = json.dumps(input_payload, sort_keys=True, separators=(",", ":")) digest = hashlib.sha256(canonical.encode()).hexdigest() return f"{tenant_id}:{task_name}:{digest}" ``` Store the operation before sending: ``` { "operation_id": "acme:summarize:9d2a...", "status": "in_flight", "provider": "anthropic", "model": "claude-sonnet-4.6", "created_at": "2026-02-03T09:12:01Z" } ``` Then update it after completion: ``` { "operation_id": "acme:summarize:9d2a...", "status": "completed", "provider_request_id": "req_abc123", "usage": { "input_tokens": 4180, "output_tokens": 612 } } ``` Before retrying elsewhere, check the operation table. If another worker already completed it, return the stored result. ### Distinguish Timeout Types Not all timeouts are equal: - Connect timeout: request likely never reached provider - Read timeout before headers: uncertain - Timeout while streaming tokens: provider almost certainly processed it - Client cancellation: depends on whether cancellation propagated For uncertain cases, I prefer a conservative retry policy: ``` interactive: - retry connect timeout - do not cross-provider retry after partial stream - show recoverable error if status unknown batch: - retry after reconciliation delay - prefer same provider with idempotency key - fallback only after retry budget expires ``` ### Stream Carefully Streaming improves perceived latency, but it complicates retries. Once you have sent partial output to the user, retrying with another model can produce a different continuation. For streaming chat, I usually treat “first token received” as the point of no transparent retry. After that, failures become visible failures: ``` "The response was interrupted. Retry?" ``` That is less elegant than silent recovery, but it avoids stitching together incompatible outputs from different models. ## Observability: The Metrics That Actually Help You need more than success rate. Track at least: ``` llm_requests_total{provider,model,status} llm_tokens_reserved_total{provider,model} llm_tokens_used_total{provider,model} llm_retries_total{provider,model,error_type} llm_queue_depth{priority,task_type} llm_queue_wait_seconds{priority,task_type} llm_rate_limit_hits_total{provider,model,limit_type} llm_fallbacks_total{from_provider,to_provider,reason} llm_duplicate_suppressed_total{task_type} ``` The most useful dashboard panel in real incidents is often not latency. It is: ``` queue_wait_seconds p50 / p95 rate_limit_hits by provider and model tokens_reserved vs tokens_used fallback rate by reason ``` If tokens_reserved is consistently much higher than tokens_used, your reservations are too conservative. If it is too close or below actual usage, you are oversubscribing and will hit TPM unexpectedly. ## A Practical Retry Wrapper Here is a compact pattern for retrying only eligible failures: ``` import asyncio import random import time RETRYABLE_STATUS = {429, 500, 502, 503, 504} class LLMError(Exception): def __init__(self, status, message="", retry_after=None): self.status = status self.retry_after = retry_after super().__init__(message) def jittered_backoff(attempt, base=0.5, cap=20): return random.uniform(0, min(cap, base * (2 ** attempt))) async def call_with_retries(fn, max_attempts=4, max_elapsed=20): started = time.monotonic() for attempt in range(max_attempts): try: return await fn() except LLMError as error: elapsed = time.monotonic() - started if error.status not in RETRYABLE_STATUS: raise if attempt == max_attempts - 1 or elapsed >= max_elapsed: raise delay = jittered_backoff(attempt) if error.retry_after is not None: delay = max(delay, error.retry_after) remaining = max_elapsed - elapsed await asyncio.sleep(min(delay, remaining)) ``` The wrapper is intentionally boring. The sophistication should live in classification, queueing, idempotency, and routing—not in a magical retry loop that tries forever. ## Practical Takeaways - Model rate limits as both RPM and TPM. Token-aware throttling prevents surprises from large prompts. - Use jittered exponential backoff, and respect retry headers when the provider gives them. - Put a queue and dispatcher between your app and LLM providers for batch or high-volume workloads. - Control concurrency separately from rate limits; latency spikes can otherwise create too many in-flight calls. - Design graceful degradation intentionally: smaller models, shorter context, queued background work, or cached responses. - Treat multi-provider fallback as a reliability feature, not a license to spray duplicate requests. - Use idempotency keys and operation records to reduce double-billing and suppress duplicate completions. - Be honest with users when a streaming response fails after tokens have already been emitted. - Build dashboards around queue wait, token reservation, rate-limit hits, retries, and fallback reasons. - Keep retry logic simple; the real production work is budgeting, scheduling, and knowing when not to retry. [API](/blog/tag/api/) A [AI Prime Tech](https://aiprimetech.io/) AI Prime Tech publishes this blog. Articles are drafted with AI assistance; check version-specific details such as model IDs, prices and limits against the official documentation before relying on them. Get cheaper Claude API access One API key for Claude Opus 5.5, Sonnet 5, Haiku 4.5 and Fable 5.1, plus GPT-6 models. Pay as you go, no subscription. [Get Your API Key →](/register) AI Prime Tech is an independent third-party API gateway. Claude™ and Anthropic® are trademarks of Anthropic, PBC. No affiliation or endorsement is implied.