Production best practices

What to get right before an integration carries real traffic.

Reliability

• Retry the right codes. 429, 500, 502, 503, 529 with exponential backoff and jitter. Never blind-retry 400, 401, 402 or 403.
• Set timeouts. A long generation can legitimately take minutes; a hung socket should not take your worker with it. 120s is a reasonable ceiling for non-streaming calls.
• Cap concurrency with a semaphore rather than firing everything and retrying rejections.
• Have a fallback model. If your primary is overloaded, degrading to a smaller model beats returning an error.

Cost

• Set max_tokens on every call. It is a spending cap, not a hint.
• Route by task — see Cost control.
• Turn on prompt caching for stable prefixes.
• Log usage per request with a task label so you can attribute spend later.

Security

• Keys live in environment variables or a secret manager — never in the repo, never in a client bundle.
• One key per service, so revocation is surgical.
• Never build prompts by string-concatenating untrusted user input with instructions; keep user content in a user message and instructions in system.
• Treat model output as untrusted. Do not eval it, do not run generated SQL against a writable role.

Observability

• Log model, latency, usage, stop_reason and your own task type for every call.
• Alert on the stop_reason: max_tokens rate — a rise means answers are being truncated.
• Track cost per task type, not just total. Totals hide which workload regressed.