Context management

Why long sessions get expensive non-linearly, and the patterns that keep history bounded.

The API is stateless. Every turn resends the entire conversation, so input tokens grow with the transcript — and cost grows with them, on every subsequent call.

The shape of the problem

A conversation of N turns, each adding roughly T tokens, costs on the order of N²T/2 input tokens in total rather than NT. Doubling the session length roughly quadruples the input bill. This is why an agent that felt cheap for ten minutes is alarming after an hour.

Four patterns that work

Summarising older turns

Copydef compact(history, keep=6):
    class="s">""class="s">"Replace everything but the last `keep` turns with one summary turn."class="s">""
    if len(history) <= keep:
        return history
    old, recent = history[:-keep], history[-keep:]
    summary = summarise(old)          class=class="s">"c"># one cheap Haiku call
    return [{class="s">"role": class="s">"user", class="s">"content": fclass="s">"Summary of earlier discussion: {summary}"}] + recent

Run the summarisation on a cheap model. It is a compression task, not a reasoning task.

Tool results are the usual culprit

• A tool returning 500 rows puts 500 rows in every later request. Cap the rows, or return a summary plus a handle to fetch detail on demand.
• Full file contents read by an agent stay in context. Read the section you need, not the whole file.
• Screenshots and images are resent like any other content — drop them once described.
• Build logs and stack traces: keep the first and last 50 lines, discard the middle.