Handling Rate Limits and Retries on LLM APIs

At 09:12 on a Tuesday, our summarization pipeline looked healthy: 96 workers online, median model latency around 2.8 seconds, no elevated 5xx rate. Then a customer uploaded a backlog of 38,000 support tickets.

Within four minutes, the pipeline was “down” without any server being down. We were hitting two different limits at once:

• requests_per_minute: too many API calls
• tokens_per_minute: too much prompt + completion volume

The naive retry logic made it worse. Every worker saw 429, slept for one second, and retried together. That created a retry wave, which produced more 429s, which created another wave. The system had accidentally built a synchronized denial-of-service attack against its own quota.

Handling LLM rate limits is not just “add exponential backoff.” In production, you need to reason about tokens, queues, concurrency, retries, idempotency, fallback providers, and billing behavior as one system.

The Two Limits That Matter: RPM and TPM

Most LLM APIs enforce some combination of:

• RPM: requests per minute
• TPM: tokens per minute
• RPD / TPD: requests or tokens per day
• Concurrent request limits: maximum in-flight calls
• Model-specific limits: higher limits for smaller models, lower limits for frontier models

The important part: RPM and TPM fail differently.

If you send many tiny prompts, you hit RPM first. If you send fewer large prompts, you hit TPM first.

Example:

Limit:
- 600 requests/minute
- 300,000 tokens/minute

Workload A:
- 1,000 short classification calls/minute
- 200 tokens each
- 200,000 tokens/minute total

Result:
- TPM is fine
- RPM fails

Workload B:
- 100 long document analysis calls/minute
- 5,000 tokens each
- 500,000 tokens/minute total

Result:
- RPM is fine
- TPM fails

A common gotcha: developers throttle requests but not tokens. That works in staging, where prompts are short. Then production users paste PDFs, chat histories, logs, or legal contracts, and the token budget evaporates.

In practice, I treat every LLM call as reserving two resources:

cost = {
  requests: 1,
  tokens: estimated_prompt_tokens + max_output_tokens
}

Even if the model returns fewer output tokens than max_output_tokens, reserving the upper bound prevents oversubscription.

What Actually Happens When You Ignore TPM

Suppose you have:

TPM limit: 120,000
Average request: 4,000 input tokens + 1,000 max output tokens
Reserved cost: 5,000 tokens

Your real maximum throughput is not “as many workers as possible.” It is:

120,000 / 5,000 = 24 requests per minute

That is one request every 2.5 seconds on average.

If you run 100 workers, the first batch may all start successfully if the provider allows burstiness. Then the account hits a hard wall. Every subsequent request gets 429, and retries pile up.

The fix is not just lowering worker count. You need a scheduler that understands token reservations.

A Minimal Token-Aware Rate Limiter

For a single process, a token bucket is usually enough. You refill capacity over time and require each request to acquire both a request slot and a token slot.

Here is a simplified Python example:

import asyncio
import time
from dataclasses import dataclass

@dataclass
class Bucket:
    capacity: float
    refill_per_second: float
    tokens: float
    updated_at: float

    def refill(self):
        now = time.monotonic()
        elapsed = now - self.updated_at
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_per_second)
        self.updated_at = now

    async def acquire(self, amount: float):
        while True:
            self.refill()
            if self.tokens >= amount:
                self.tokens -= amount
                return

            missing = amount - self.tokens
            sleep_for = missing / self.refill_per_second
            await asyncio.sleep(min(sleep_for, 1.0))


class LLMRateLimiter:
    def __init__(self, rpm: int, tpm: int):
        now = time.monotonic()
        self.request_bucket = Bucket(
            capacity=rpm,
            refill_per_second=rpm / 60,
            tokens=rpm,
            updated_at=now,
        )
        self.token_bucket = Bucket(
            capacity=tpm,
            refill_per_second=tpm / 60,
            tokens=tpm,
            updated_at=now,
        )
        self.lock = asyncio.Lock()

    async def acquire(self, estimated_tokens: int):
        async with self.lock:
            await self.request_bucket.acquire(1)
            await self.token_bucket.acquire(estimated_tokens)

Usage:

estimated_tokens = prompt_tokens + max_output_tokens
await limiter.acquire(estimated_tokens)

response = await client.chat.completions.create(
    model="gpt-5.5",
    messages=messages,
    max_tokens=max_output_tokens,
)

This is deliberately simple. In a distributed system, use Redis, Postgres advisory locks, or a queue with centralized dispatch. The important principle is the same: throttle on both request count and token budget before you call the API.

Retries: Backoff Is Necessary but Not Sufficient

Retries are useful for:

• Transient 429 responses
• Network timeouts
• Temporary 500, 502, 503, or 504 errors
• Provider-side overload
• Connection resets

Retries are dangerous for:

• Validation errors
• Context length errors
• Authentication failures
• Safety policy rejections
• Deterministic bad prompts
• Non-idempotent tool calls

The retry policy should classify errors first.

The basic backoff shape:

import random

def retry_delay(attempt: int, base=0.5, cap=30.0):
    exponential = min(cap, base * (2 ** attempt))
    return random.uniform(0, exponential)

That random.uniform(0, exponential) is jitter. Do not skip it.

Without jitter:

100 workers fail
100 workers sleep 1s
100 workers retry
100 workers fail
100 workers sleep 2s
100 workers retry

With jitter:

100 workers fail
workers retry across a spread of 0-1s
then 0-2s
then 0-4s

The second pattern gives the provider and your own queue room to recover.

Respect Retry Headers When Present

Many APIs include a hint such as:

Retry-After: 12

or provider-specific reset headers. If the API gives you a concrete delay, use it as a floor:

delay = max(header_retry_after_seconds, retry_delay(attempt))
await asyncio.sleep(delay)

In practice, I also cap total retry time. A user-facing chat request should not quietly retry for two minutes while the user stares at a spinner. A background batch job can wait much longer.

Example policy:

{
  "interactive_chat": {
    "max_attempts": 3,
    "max_total_retry_seconds": 8
  },
  "batch_summarization": {
    "max_attempts": 8,
    "max_total_retry_seconds": 300
  }
}

Queueing Beats Worker Stampedes

If you have more work than quota, workers should not all directly call the LLM API. Put a queue in front.

A practical architecture:

App/API
  -> Job queue
      -> Rate-aware dispatcher
          -> Provider client
              -> Result store

The dispatcher owns:

• RPM accounting
• TPM accounting
• Model selection
• retry policy
• provider fallback
• concurrency limits
• queue priority

This makes overload visible. Instead of thousands of tasks failing with 429, you see queue depth rising and estimated wait time increasing.

For Redis-backed systems, I like separating “ready to run” from “delayed retry” jobs:

llm:ready
llm:delayed
llm:dead_letter

A failed job does not immediately return to ready. It gets a scheduled retry timestamp. A small scheduler moves due jobs back into the ready queue.

{
  "job_id": "sum_7f13",
  "model": "claude-sonnet-4.6",
  "estimated_tokens": 6200,
  "attempt": 2,
  "not_before": "2026-02-03T09:14:22Z"
}

That not_before field prevents retry storms.

Concurrency Control Is Not the Same as Rate Limiting

Concurrency limits cap in-flight work. Rate limits cap work per time window.

You need both.

Imagine each call takes 20 seconds and your RPM is 60. A pure RPM limiter permits one request per second. After 20 seconds, you have about 20 concurrent calls. If latency jumps to 90 seconds, the same RPM creates 90 concurrent calls. That may exceed provider or local resource limits.

Use a semaphore for concurrency:

provider_concurrency = asyncio.Semaphore(20)

async def call_model(job):
    async with provider_concurrency:
        await limiter.acquire(job.estimated_tokens)
        return await send_request(job)

A common gotcha: acquire the semaphore too early and your workers sit idle holding scarce concurrency slots while waiting for rate tokens. Usually, acquire rate capacity first, then acquire concurrency immediately before the network call. If your limiter wait can be long, avoid occupying worker resources during that wait.

Graceful Degradation: Spend Quality Where It Matters

When rate limits bite, you have options besides failing.

For example, a customer support assistant might degrade like this:

Concrete examples:

• Route ticket tagging to Haiku 4.5, Qwen, or a small Llama variant instead of using a frontier model.
• Use Sonnet 4.6 for normal reasoning and reserve Opus 4.8 or GPT-5.5 for escalation paths.
• Move long-context analysis to Fable 5 or Gemini 3 only when the prompt genuinely needs that context window.
• For batch jobs, pause low-priority tenants before impacting interactive traffic.

The trade-off is complexity. Every fallback path needs tests, observability, and product acceptance. A worse answer can be more damaging than a delayed answer in some domains.

Multi-Key and Multi-Provider Fallback

Multi-key and multi-provider routing can improve resilience, but it is easy to do badly.

There are three separate patterns:

Do not use multiple keys to sneak around limits. Use them for legitimate separation: tenants, environments, products, or billing boundaries.

A clean routing policy might look like:

{
  "task": "support_ticket_summary",
  "primary": {
    "provider": "anthropic",
    "model": "claude-sonnet-4.6"
  },
  "fallbacks": [
    {
      "provider": "openai",
      "model": "gpt-5.5",
      "when": ["provider_5xx", "rate_limit_after_retry_budget"]
    },
    {
      "provider": "google",
      "model": "gemini-3",
      "when": ["context_length_needed", "primary_unavailable"]
    }
  ],
  "degrade_to": {
    "provider": "open_model_cluster",
    "model": "qwen",
    "when": ["batch_low_priority"]
  }
}

If you use an aggregator or a platform like AI Prime Tech for multi-model access, the same engineering rules still apply: track idempotency, budgets, fallback conditions, and final billing semantics explicitly.

Avoiding Double-Billing on Fallback

The hard part of fallback is knowing whether the first request completed.

Consider this sequence:

Client sends request to Provider A
Provider A processes it successfully
Network connection drops before client receives response
Client retries with Provider B
Both providers bill
Two completions may exist

You cannot fully solve this from the client side unless the provider supports idempotency keys or request lookup. But you can reduce damage.

Use Idempotency Keys Where Available

Generate a stable key per logical operation:

import hashlib
import json

def idempotency_key(task_name, tenant_id, input_payload):
    canonical = json.dumps(input_payload, sort_keys=True, separators=(",", ":"))
    digest = hashlib.sha256(canonical.encode()).hexdigest()
    return f"{tenant_id}:{task_name}:{digest}"

Store the operation before sending:

{
  "operation_id": "acme:summarize:9d2a...",
  "status": "in_flight",
  "provider": "anthropic",
  "model": "claude-sonnet-4.6",
  "created_at": "2026-02-03T09:12:01Z"
}

Then update it after completion:

{
  "operation_id": "acme:summarize:9d2a...",
  "status": "completed",
  "provider_request_id": "req_abc123",
  "usage": {
    "input_tokens": 4180,
    "output_tokens": 612
  }
}

Before retrying elsewhere, check the operation table. If another worker already completed it, return the stored result.

Distinguish Timeout Types

Not all timeouts are equal:

• Connect timeout: request likely never reached provider
• Read timeout before headers: uncertain
• Timeout while streaming tokens: provider almost certainly processed it
• Client cancellation: depends on whether cancellation propagated

For uncertain cases, I prefer a conservative retry policy:

interactive:
- retry connect timeout
- do not cross-provider retry after partial stream
- show recoverable error if status unknown

batch:
- retry after reconciliation delay
- prefer same provider with idempotency key
- fallback only after retry budget expires

Stream Carefully

Streaming improves perceived latency, but it complicates retries. Once you have sent partial output to the user, retrying with another model can produce a different continuation.

For streaming chat, I usually treat “first token received” as the point of no transparent retry. After that, failures become visible failures:

"The response was interrupted. Retry?"

That is less elegant than silent recovery, but it avoids stitching together incompatible outputs from different models.

Observability: The Metrics That Actually Help

You need more than success rate.

Track at least:

llm_requests_total{provider,model,status}
llm_tokens_reserved_total{provider,model}
llm_tokens_used_total{provider,model}
llm_retries_total{provider,model,error_type}
llm_queue_depth{priority,task_type}
llm_queue_wait_seconds{priority,task_type}
llm_rate_limit_hits_total{provider,model,limit_type}
llm_fallbacks_total{from_provider,to_provider,reason}
llm_duplicate_suppressed_total{task_type}

The most useful dashboard panel in real incidents is often not latency. It is:

queue_wait_seconds p50 / p95
rate_limit_hits by provider and model
tokens_reserved vs tokens_used
fallback rate by reason

If tokens_reserved is consistently much higher than tokens_used, your reservations are too conservative. If it is too close or below actual usage, you are oversubscribing and will hit TPM unexpectedly.

A Practical Retry Wrapper

Here is a compact pattern for retrying only eligible failures:

import asyncio
import random
import time

RETRYABLE_STATUS = {429, 500, 502, 503, 504}

class LLMError(Exception):
    def __init__(self, status, message="", retry_after=None):
        self.status = status
        self.retry_after = retry_after
        super().__init__(message)

def jittered_backoff(attempt, base=0.5, cap=20):
    return random.uniform(0, min(cap, base * (2 ** attempt)))

async def call_with_retries(fn, max_attempts=4, max_elapsed=20):
    started = time.monotonic()

    for attempt in range(max_attempts):
        try:
            return await fn()
        except LLMError as error:
            elapsed = time.monotonic() - started
            if error.status not in RETRYABLE_STATUS:
                raise
            if attempt == max_attempts - 1 or elapsed >= max_elapsed:
                raise

            delay = jittered_backoff(attempt)
            if error.retry_after is not None:
                delay = max(delay, error.retry_after)

            remaining = max_elapsed - elapsed
            await asyncio.sleep(min(delay, remaining))

The wrapper is intentionally boring. The sophistication should live in classification, queueing, idempotency, and routing—not in a magical retry loop that tries forever.

Practical Takeaways

• Model rate limits as both RPM and TPM. Token-aware throttling prevents surprises from large prompts.
• Use jittered exponential backoff, and respect retry headers when the provider gives them.
• Put a queue and dispatcher between your app and LLM providers for batch or high-volume workloads.
• Control concurrency separately from rate limits; latency spikes can otherwise create too many in-flight calls.
• Design graceful degradation intentionally: smaller models, shorter context, queued background work, or cached responses.
• Treat multi-provider fallback as a reliability feature, not a license to spray duplicate requests.
• Use idempotency keys and operation records to reduce double-billing and suppress duplicate completions.
• Be honest with users when a streaming response fails after tokens have already been emitted.
• Build dashboards around queue wait, token reservation, rate-limit hits, retries, and fallback reasons.
• Keep retry logic simple; the real production work is budgeting, scheduling, and knowing when not to retry.