Choosing the Right Model for the Job (and the Budget)

The Useful Mental Model: Task Shape First, Model Second

In practice, I classify LLM workloads along five axes before I even look at model names:

Model choice becomes much easier once you stop asking “which model is smartest?” and start asking “what failure mode am I buying down?”

For example:

• A password-reset summarizer needs consistency and low cost.
• A code migration assistant needs reasoning and tool use.
• A contract diff tool needs long context, but also careful attribution.
• A customer-facing agent needs latency predictability more than peak benchmark quality.
• A batch enrichment pipeline may care more about throughput per dollar than response elegance.

The wrong model is often not technically wrong. It is economically wrong.

A Practical Model Ladder

Here is the ladder I use for most production systems.

Haiku-Class Models: The Default for Cheap, Fast Language Work

Haiku-class models are excellent first-pass workers. I use them for jobs where the answer is short, the input is clean, and the output format is constrained.

Good fits:

• Ticket tagging
• Short summaries
• Sentiment classification
• Email subject normalization
• FAQ matching
• Simple JSON extraction
• Low-risk moderation prefilters

Example prompt for a cheap first-pass classifier:

{
  "model": "claude-haiku-4.5",
  "temperature": 0,
  "max_tokens": 120,
  "response_format": { "type": "json" },
  "messages": [
    {
      "role": "system",
      "content": "Classify the support ticket. Return only valid JSON."
    },
    {
      "role": "user",
      "content": "Subject: Refund not received\nBody: I canceled two weeks ago and still see a charge."
    }
  ]
}

Expected output:

{
  "category": "billing",
  "urgency": "medium",
  "needs_human": true,
  "reason": "Refund dispute involving payment status"
}

A common gotcha: teams ask cheap models to both reason and format perfectly in one shot. If the model sometimes emits invalid JSON, do not immediately jump to Opus. First tighten the task:

• Set temperature: 0
• Use a JSON schema or tool call
• Reduce the instruction surface area
• Split classification from explanation
• Validate output and retry once

For many production tasks, one retry on Haiku-class is still cheaper and faster than defaulting to a frontier reasoning model.

Sonnet-Class Models: The Workhorse for Real Product Logic

Sonnet-class models are where I usually land for interactive product features. They are strong enough for reasoning, stable enough for tool use, and usually fast enough for user-facing flows.

Good fits:

• Customer support drafting
• Agentic workflows with tools
• Code explanation and light refactoring
• Multi-step document Q&A
• Sales or success copilot workflows
• Structured extraction with messy inputs

In practice, Sonnet is often the best “don’t make me think too hard about routing” choice. If you can only integrate one model tier into a new feature, start here.

Example: a support agent that can inspect an order, policy, and prior tickets.

SYSTEM = """
You are a support resolution assistant.
Use tools before making claims about orders, refunds, or account state.
If policy is ambiguous, say what is ambiguous and escalate.
Return:
- summary
- recommended_action
- customer_reply
- escalate: true/false
"""

request = {
    "model": "claude-sonnet-4.6",
    "temperature": 0.2,
    "max_tokens": 900,
    "tools": [
        {"name": "get_order", "description": "Fetch order state"},
        {"name": "get_refund_policy", "description": "Fetch current refund rules"},
        {"name": "search_prior_tickets", "description": "Search customer history"}
    ],
    "messages": [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": "Customer says they returned item A but were charged again."}
    ]
}

What actually happens when you run this kind of workflow in production: most failures are not “the model is dumb.” They are integration failures.

• The tool returns stale state.
• The model gets too much irrelevant context.
• The policy document has contradictory text.
• The agent is allowed to continue after a failed tool call.
• The prompt asks for both internal analysis and customer-ready prose without separating them.

A Sonnet-class model gives you enough intelligence to recover from some messiness, but it does not remove the need for clean tool contracts.

Opus-Class Models: Expensive, But Sometimes Cheaper Than Mistakes

Opus-class models are for tasks where the cost of a bad answer dominates the cost of inference.

I reach for Opus when:

• The input is ambiguous and high-stakes
• There are many interacting constraints
• The user expects deep reasoning
• The task involves code, policy, or architecture trade-offs
• A failed cheap attempt would waste human time

Examples:

• Reviewing a database migration plan
• Analyzing a production incident timeline
• Generating a complex legal-policy comparison for human review
• Debugging a multi-service agent failure
• Evaluating whether an automated action is safe

I do not use Opus for every “important” task. Importance and difficulty are different. A million password reset tickets are important to users, but they are not individually hard. A single subtle data deletion migration may be both important and hard.

A useful pattern is “escalate on uncertainty”:

def choose_model(task):
    if task["context_tokens"] > 700_000:
        return "fable-5"

    if task["risk"] == "high" and task["requires_reasoning"]:
        return "claude-opus-4.8"

    if task["requires_tools"] or task["multi_step"]:
        return "claude-sonnet-4.6"

    return "claude-haiku-4.5"

That is intentionally simple. The router should be boring enough that your team can debug it at 2 a.m.

Fable 5 and 1M Context: Use It When Retrieval Is the Problem

Long context models are tempting because they remove a lot of retrieval engineering. Fable 5’s 1M context window changes what you can attempt directly: whole-document review, giant transcript synthesis, large repository inspection, and multi-file policy analysis.

But long context is not magic.

A 1M-token window helps when the hard part is “the relevant fact might be anywhere.” It does not guarantee the model will reason perfectly across all million tokens, and it does not make irrelevant context free. Long prompts still cost money, add latency, and can dilute attention.

I use Fable-style long context when:

• The corpus is too interconnected for naive chunking
• Missing one detail is worse than paying for extra context
• The user can tolerate slower responses
• The task benefits from seeing raw surrounding material

Example architecture for large document review:

User request
  -> lightweight classifier
  -> retrieve obvious relevant sections
  -> if confidence high: Sonnet answer with citations/quotes from chunks
  -> if confidence low or query is global: Fable 5 over full document set
  -> optional Opus review for high-risk final answer

This hybrid works better than “always stuff everything into context.” In practice, long-context models are most valuable as an escape hatch when retrieval confidence is low.

Open-Weight Models: Control, Cost, and Throughput

Open-weight models like Llama, Qwen, DeepSeek, MiniMax, and Kimi are not just “cheaper alternatives.” They are operational choices.

They make sense when you need:

• Local or VPC deployment
• Predictable high-volume batch cost
• Fine-tuning or distillation
• Custom inference optimizations
• Data residency control
• Extremely low marginal cost after setup

They are especially good for narrow, repeated tasks. For example, a Qwen or Llama variant behind vLLM can handle classification, extraction, and internal enrichment at high throughput if you control prompt length and output format.

Example local serving shape:

vllm serve Qwen/Qwen3-14B \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

Then call it through an OpenAI-compatible client:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="local"
)

result = client.chat.completions.create(
    model="Qwen/Qwen3-14B",
    temperature=0,
    messages=[
        {"role": "system", "content": "Return valid JSON only."},
        {"role": "user", "content": "Classify: user cannot access invoice PDF"}
    ]
)

The trade-off is engineering ownership. With hosted frontier models, someone else handles capacity, patching, quantization choices, and kernel-level performance work. With open models, that becomes your job.

Open-weight models are a great fit when the workload is stable. They are a worse fit when the product is changing weekly and prompt quality matters more than raw serving cost.

Cheap-First Routing With Escalation

A good router is not a leaderboard. It is a risk control system.

Here is a production-friendly pattern:

1. Try the cheapest model that should be capable.
2. Validate the output mechanically.
3. Ask the model for a calibrated self-check only on observable criteria.
4. Escalate if validation fails, confidence is low, or risk is high.
5. Log the route and outcome for later tuning.

Example router:

def run_with_escalation(task, input_text):
    first_model = pick_first_model(task)

    response = call_model(
        model=first_model,
        input=input_text,
        schema=task["schema"],
        temperature=0
    )

    validation = validate(response, task["schema"])

    if not validation.ok:
        return call_model(
            model="claude-sonnet-4.6",
            input=input_text,
            schema=task["schema"],
            temperature=0
        )

    if task["risk"] == "high" and response.get("needs_review"):
        return call_model(
            model="claude-opus-4.8",
            input=input_text,
            schema=task["schema"],
            temperature=0
        )

    return response

The important part is that escalation is based on measurable signals, not vibes.

Useful escalation signals:

• Invalid JSON
• Missing required fields
• Contradictory answer
• Tool call failed
• Retrieved evidence is weak
• User is asking for irreversible action
• Input exceeds normal length
• Model explicitly marks uncertainty
• Classifier detects legal, medical, security, or financial content

Avoid using only “model confidence” as a numeric score. LLM confidence is not a calibrated probability. It is still useful when constrained to specific checks: “Did the input contain enough information to answer?” is better than “How confident are you?”

Latency, Price, and User Experience

The fastest model is not always the best UX. The best UX is the one that gets the user to a correct result with tolerable waiting.

For interactive systems, I think in latency budgets:

Do not hide slow reasoning behind a spinner without feedback. If an Opus or Fable job needs 90 seconds, make it an explicit analysis task:

Analyzing 184 files...
✓ Indexed repository structure
✓ Found 12 migration scripts
✓ Checking rollback safety

That changes user perception because the system is doing visible work.

Also, stream whenever possible. Streaming does not reduce total completion time, but it improves perceived latency dramatically for chat and drafting workflows.

Real Per-Task Decisions

Here are concrete choices I would make today.

Support Ticket Triage

Use Haiku first.

• Input: short ticket text
• Output: category, urgency, routing queue
• Escalate to Sonnet for billing disputes, angry customers, policy ambiguity
• Human review for refunds above threshold

This is a classic cheap-first workflow.

Customer-Facing Support Agent

Use Sonnet.

• Needs tool use
• Needs tone control
• Needs policy handling
• Needs graceful escalation

Use Haiku only for preprocessing. Use Opus for supervisor review on high-risk cases.

Code Review Assistant

Use Sonnet for ordinary PR review. Use Opus for architecture-sensitive changes, concurrency, security, data migrations, and subtle correctness work.

A common mistake is feeding the whole repo by default. Start with the diff, dependency graph, and nearby files. Escalate to Fable or another long-context model only when the change spans many areas.

Long Contract or Policy Comparison

Use retrieval plus Sonnet for narrow questions. Use Fable 5 when the question is global, like “find all contradictions across these documents.”

If the output influences a real legal or compliance decision, use the model to prepare analysis, not to make the decision.

High-Volume Data Enrichment

Use an open-weight model if volume is high and the task is stable.

Example:

• Normalize company names
• Extract industry tags
• Classify inbound leads
• Generate short internal summaries

Run nightly batches, validate samples, and escalate edge cases to a hosted model.

Research Assistant Over Large Archives

Use Gemini 3, GPT-5.5, Opus, or Fable depending on context length and reasoning needs. I would not cheap-first this unless the query is obviously simple. Research failures are often silent: the model gives a plausible answer while missing the key document.

Budgeting Without Lying to Yourself

The easiest way to overspend is to optimize per-call cost while ignoring retries, human correction, and failed workflows.

Track these metrics by task type and model:

• Input tokens
• Output tokens
• Latency p50/p95
• Validation failure rate
• Escalation rate
• Human override rate
• User retry rate
• Cost per successful task, not cost per call

That last one matters. A cheap model that requires three retries and still causes manual review may be more expensive than Sonnet. An Opus call that prevents a bad migration can be the cheapest decision of the week.

If you use a multi-model gateway, including something like AI Prime Tech’s API access across Claude, GPT, and Gemini families, the main benefit is not just vendor optionality. It is making routing and measurement easier across model classes. But the hard part is still your task design.

Practical Takeaways

• Start with the task shape: transformation, extraction, reasoning, long-context synthesis, or batch enrichment.
• Use Haiku-class models for cheap, fast, constrained work with mechanical validation.
• Use Sonnet-class models as the default for product features that need reasoning, tools, or user-facing quality.
• Use Opus-class models when mistakes are expensive, ambiguity is high, or deep reasoning matters.
• Use Fable 5-style long context when retrieval uncertainty is the main problem, not as a default prompt-dumping strategy.
• Use open-weight models when workload stability, privacy, throughput, or deployment control justify owning inference.
• Build cheap-first routing with explicit escalation signals: invalid output, weak evidence, failed tools, high-risk actions, and uncertainty.
• Measure cost per successful task, not cost per token or per call.
• Keep routers simple enough to debug under pressure.
• Treat model choice as an engineering control surface, not a one-time vendor decision.