Streaming LLM Responses: Why It Matters for UX and Cost

The 900 ms That Made Our Chat UI Feel “Broken”

A few months ago I was debugging a customer support assistant that had perfectly acceptable end-to-end latency on paper: most responses completed in 4–7 seconds. The model was doing tool selection, retrieving account context, and generating a useful answer.

But users hated it.

The reason was simple: the UI sat blank for the first 3.8 seconds.

No spinner copy, no partial response, no indication that the model had understood the question. Just an empty assistant bubble. When we switched the same endpoint to stream tokens over Server-Sent Events, the average total completion time barely changed. But time-to-first-visible-output dropped to around 700–900 ms for many requests, and the product immediately felt faster.

That is the real reason streaming matters.

Streaming LLM responses is not magic. It does not make the model think faster. It does not reduce total tokens. It does not automatically reduce your bill. But it changes the interaction loop in a way users can feel, and it gives your application more control over cancellation, rendering, moderation, usage accounting, and backpressure.

This article walks through how SSE streaming actually works, what to measure, where billing can get subtle, and the client-side patterns I use in production systems.

What Streaming Actually Means

When you make a non-streaming LLM request, the server usually waits until the model has finished generating the response, then returns a single JSON payload:

{
  "id": "resp_123",
  "model": "some-model",
  "output_text": "Sure — here is the answer...",
  "usage": {
    "input_tokens": 812,
    "output_tokens": 436
  }
}

With streaming, the server sends incremental events as output becomes available. The most common transport is Server-Sent Events, usually abbreviated as SSE.

SSE is just an HTTP response with a long-lived connection and a special content type:

Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive

Instead of one JSON object, the server sends a sequence of small events:

event: message.delta
data: {"delta":"Sure"}

event: message.delta
data: {"delta":" — here"}

event: message.delta
data: {"delta":" is"}

event: message.delta
data: {"delta":" the answer..."}

event: message.done
data: {"finish_reason":"stop","usage":{"input_tokens":812,"output_tokens":436}}

Different APIs use different event names and schemas, but the shape is usually the same:

1. Open an HTTP connection.
2. Send request metadata and prompt.
3. Receive incremental chunks.
4. Append chunks to the current assistant message.
5. Receive a final event with stop reason and, often, usage metadata.
6. Close the stream.

In practice, most LLM “streaming” is not one token per event. Providers may batch tokens into chunks for efficiency. You might receive a single character, a word fragment, several words, or a structured JSON delta. Never build a client that assumes each chunk is a full word or a full sentence.

Latency: Total Time vs Time-to-First-Token

The most important distinction is between total latency and time-to-first-token.

For chat UX, TTFT is usually more important than total time. A response that starts in 800 ms and finishes in 8 seconds often feels better than one that appears all at once after 4 seconds.

For machine-to-machine workflows, streaming often matters less. If your next step cannot run until the full JSON object is complete, streaming may only add implementation complexity.

Measuring TTFT in Python

Here is a simple pattern for measuring first chunk latency and total stream time. The API shape is intentionally generic: adapt the URL and payload to your provider.

import json
import time
import requests

url = "https://api.example.com/v1/chat/completions"

payload = {
    "model": "sonnet-4.6",
    "stream": True,
    "messages": [
        {"role": "user", "content": "Explain SSE streaming in 5 bullet points."}
    ]
}

headers = {
    "Authorization": "Bearer $API_KEY",
    "Content-Type": "application/json",
    "Accept": "text/event-stream",
}

start = time.perf_counter()
first_chunk_at = None
chars = 0

with requests.post(url, headers=headers, json=payload, stream=True) as r:
    r.raise_for_status()

    for raw_line in r.iter_lines(decode_unicode=True):
        if not raw_line:
            continue

        if not raw_line.startswith("data: "):
            continue

        data = raw_line[len("data: "):]

        if data == "[DONE]":
            break

        event = json.loads(data)

        # Provider-specific: this may be event["delta"], event["choices"][0]["delta"], etc.
        delta = event.get("delta", "")

        if delta and first_chunk_at is None:
            first_chunk_at = time.perf_counter()

        chars += len(delta)
        print(delta, end="", flush=True)

end = time.perf_counter()

print("\n\n--- metrics ---")
print(f"ttft_ms={((first_chunk_at or end) - start) * 1000:.0f}")
print(f"total_ms={(end - start) * 1000:.0f}")
print(f"chars={chars}")

A common gotcha: if you test streaming with a local proxy, serverless function, or reverse proxy that buffers responses, you may think the model is not streaming. The model may be streaming correctly, but your infrastructure is holding the chunks until the response completes.

I have seen this with:

• Nginx buffering enabled.
• Cloud functions that do not support streaming responses.
• Compression middleware waiting for larger buffers.
• Web frameworks that build the whole response before returning it.
• Corporate proxies that coalesce small chunks.

For Nginx, this setting often matters:

location /api/llm/stream {
    proxy_pass http://app_backend;
    proxy_http_version 1.1;

    proxy_buffering off;
    proxy_cache off;

    chunked_transfer_encoding on;
    add_header X-Accel-Buffering no;
}

SSE vs WebSockets

SSE is not the only way to stream. WebSockets are common too, especially for bidirectional applications. For most LLM chat interfaces, though, SSE is the simpler default.

The EventSource browser API is convenient, but it only supports GET requests. Most LLM calls need POST bodies and auth headers, so I usually stream with fetch() and ReadableStream.

Browser Client Pattern: Fetch + ReadableStream

A minimal browser implementation looks like this:

async function streamChat({ messages, signal, onDelta, onDone }) {
  const res = await fetch("/api/chat/stream", {
    method: "POST",
    headers: {
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "gpt-5.5",
      stream: true,
      messages
    }),
    signal
  });

  if (!res.ok) {
    throw new Error(`HTTP ${res.status}`);
  }

  const reader = res.body.getReader();
  const decoder = new TextDecoder("utf-8");

  let buffer = "";

  while (true) {
    const { value, done } = await reader.read();

    if (done) break;

    buffer += decoder.decode(value, { stream: true });

    const lines = buffer.split("\n");
    buffer = lines.pop() || "";

    for (const line of lines) {
      const trimmed = line.trim();

      if (!trimmed.startsWith("data:")) continue;

      const data = trimmed.slice(5).trim();

      if (data === "[DONE]") {
        onDone?.();
        return;
      }

      const event = JSON.parse(data);

      if (event.delta) {
        onDelta(event.delta);
      }
    }
  }

  onDone?.();
}

And in a UI:

const controller = new AbortController();

let assistantText = "";

streamChat({
  messages,
  signal: controller.signal,
  onDelta(delta) {
    assistantText += delta;
    renderAssistantMessage(assistantText);
  },
  onDone() {
    markMessageComplete();
  }
});

// Cancel button
document.querySelector("#stop").onclick = () => {
  controller.abort();
};

Do Not Render Every Chunk Naively

If you update React state on every tiny chunk, you can accidentally create a client-side performance problem. Some streams produce dozens of chunks per second. Re-rendering a large chat tree every time is wasteful.

In practice, I usually batch UI updates to animation frames:

let pending = "";
let scheduled = false;

function onDelta(delta) {
  pending += delta;

  if (!scheduled) {
    scheduled = true;

    requestAnimationFrame(() => {
      assistantText += pending;
      pending = "";
      scheduled = false;
      setAssistantText(assistantText);
    });
  }
}

This keeps the interface smooth while preserving the perception of live output.

Handling Partial Output Correctly

Streaming means your application receives incomplete text. That sounds obvious, but many bugs come from treating partial output as final output.

Markdown Is Temporarily Invalid

During streaming, Markdown may be malformed:

Here is the query:

```sql
SELECT *
FROM users
WHERE

The code fence might not be closed yet. A table might have only the header row. A link might be halfway emitted. Your renderer needs to tolerate this.

For chat UIs, I prefer:

• Render Markdown incrementally but defensively.
• Disable dangerous HTML.
• Avoid expensive syntax highlighting until the message is complete.
• Show a subtle “generating” state.
• Re-run final formatting after the stream ends.

JSON Streaming Is Harder

If you ask the model for JSON and stream it, the intermediate state is usually invalid JSON. This is a problem for apps that want to update structured UIs in real time.

For example:

{
  "title": "Incident Summary",
  "severity": "high",
  "steps": [
    "Restart

You cannot parse that yet.

There are three common approaches:

1. Wait until completion, then parse once.
2. Use line-delimited JSON, where each complete line is independently parseable.
3. Stream semantic events, not raw JSON.

For structured streaming, I like event-shaped output:

event: section
data: {"title":"Summary"}

event: bullet
data: {"text":"Database failover started at 09:42 UTC."}

event: bullet
data: {"text":"Read latency remained elevated for 6 minutes."}

That pattern requires more orchestration, but it is much easier to render reliably.

What Actually Happens When Users Hit “Stop”

Cancellation is one of the underappreciated benefits of streaming.

With a non-streaming request, the user may click away while the model continues generating on the server. Depending on the provider and your backend, that generation may still consume output tokens.

With streaming, the client can abort the HTTP request:

const controller = new AbortController();

fetch("/api/chat/stream", {
  method: "POST",
  body: JSON.stringify(payload),
  signal: controller.signal
});

// Later
controller.abort();

But cancellation is not always instantaneous end-to-end.

There are several layers:

• Browser aborts the fetch.
• Your backend sees the connection close.
• Your backend cancels the upstream provider request.
• Provider stops generation.
• Usage is calculated for tokens already processed/generated.

A common gotcha: if your backend proxies the stream but does not propagate cancellation upstream, the user’s browser disconnects but your server keeps reading from the model until completion.

In Node-style pseudocode:

app.post("/api/chat/stream", async (req, res) => {
  const upstreamController = new AbortController();

  req.on("close", () => {
    upstreamController.abort();
  });

  const upstream = await fetch(PROVIDER_URL, {
    method: "POST",
    headers: providerHeaders,
    body: JSON.stringify({
      ...req.body,
      stream: true
    }),
    signal: upstreamController.signal
  });

  res.setHeader("Content-Type", "text/event-stream");

  for await (const chunk of upstream.body) {
    res.write(chunk);
  }

  res.end();
});

This is the difference between “the UI stopped” and “the model stopped.”

Streaming and Cost: What Changes, What Does Not

Streaming changes when you observe usage. It usually does not change the basic unit of billing.

Most LLM APIs charge based on something like:

• Input tokens.
• Output tokens.
• Cached input tokens, if prompt caching applies.
• Tool calls or special modalities, depending on the API.
• Sometimes reasoning tokens or internal budgeted compute for certain model families.

Streaming itself is generally not a discount. If the model generates 700 output tokens, you should expect to account for 700 output tokens whether they arrive incrementally or in one final payload.

Where streaming affects cost is operational:

Usage Accounting Gotcha

Many streaming APIs send usage metadata at the end of the stream:

{
  "type": "message.done",
  "usage": {
    "input_tokens": 1240,
    "output_tokens": 389
  },
  "finish_reason": "stop"
}

If the stream is interrupted before the final event, your application may not receive usage. That does not necessarily mean usage is zero.

For internal accounting, I usually store three fields:

{
  "request_id": "chat_01j...",
  "status": "client_aborted",
  "usage": null,
  "estimated_output_chars": 1842,
  "final_usage_received": false
}

Then reconcile later if the provider exposes request logs or usage events. If not, at least you can estimate roughly for product analytics while avoiding false precision in billing.

Do not bill end users from a character estimate unless your terms explicitly support that. Tokenization varies by model and language.

Backend Architecture Pattern

A production streaming path typically looks like this:

Browser
  |
  | POST /api/chat/stream
  v
App Backend
  |
  | provider streaming request
  v
LLM API

I rarely expose provider keys directly to browsers. The backend gives you:

• Authentication and authorization.
• Rate limiting per user or workspace.
• Prompt assembly and retrieval.
• Tool execution.
• Redaction and logging controls.
• Provider failover.
• Cancellation propagation.
• Usage recording.

A minimal streaming response from a Python FastAPI backend:

import json
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import httpx

app = FastAPI()

PROVIDER_URL = "https://api.example.com/v1/chat/completions"

@app.post("/api/chat/stream")
async def chat_stream(request: Request):
    body = await request.json()

    async def event_generator():
        async with httpx.AsyncClient(timeout=None) as client:
            async with client.stream(
                "POST",
                PROVIDER_URL,
                headers={
                    "Authorization": "Bearer YOUR_PROVIDER_KEY",
                    "Content-Type": "application/json",
                    "Accept": "text/event-stream",
                },
                json={
                    **body,
                    "stream": True,
                },
            ) as upstream:
                async for line in upstream.aiter_lines():
                    if await request.is_disconnected():
                        break

                    if not line:
                        continue

                    # Optionally transform provider-specific events here.
                    yield line + "\n\n"

    return StreamingResponse(
        event_generator(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no",
        },
    )

In practice, I usually normalize provider events into one internal schema:

{
  "type": "delta",
  "text": "partial text here"
}

and then:

{
  "type": "done",
  "finish_reason": "stop",
  "usage": {
    "input_tokens": 812,
    "output_tokens": 436
  }
}

This matters if you support multiple models such as Claude, GPT, Gemini, Llama, Qwen, or Kimi behind one interface. Each provider has slightly different stream semantics. Normalizing early keeps your frontend boring, which is exactly what you want.

TTFT Is Not Just the Model

When TTFT is bad, people often blame the model first. Sometimes that is correct: larger models, longer contexts, and heavier reasoning modes can increase first-token latency. But the first token sits behind a lot of machinery.

Contributors include:

• TLS connection setup.
• Cold starts in your backend.
• Prompt construction and retrieval.
• Vector database latency.
• Tool calls before generation.
• Provider queueing.
• Long input context ingestion.
• Safety checks or policy layers.
• Proxy buffering.
• Client rendering delay.

A practical trace should separate these:

{
  "trace_id": "req_abc",
  "timings_ms": {
    "auth": 12,
    "retrieval": 84,
    "prompt_build": 9,
    "provider_connect": 141,
    "ttfb": 612,
    "first_delta": 804,
    "stream_complete": 5820
  }
}

If retrieval takes 600 ms and the model’s first delta arrives 250 ms later, switching models may not fix your UX. If provider connection setup dominates, connection reuse may help. If frontend rendering waits for a full paragraph, your stream is working but your UI policy is hiding it.

When Not to Stream

Streaming is not always worth it.

I avoid streaming when:

• The response is very short, such as classification or routing.
• The caller is another service that needs complete JSON.
• The output must pass full moderation before display.
• The request runs in a job queue or batch pipeline.
• The framework or hosting layer makes streaming unreliable.
• The product needs atomic answers, not progressive text.

There is also a UX downside: streaming can expose the model changing direction mid-answer. Humans tolerate some of this in chat, but it can look unpolished in high-stakes workflows. For legal, medical, financial, or compliance-heavy interfaces, you may choose to stream only after guardrails or stream into an internal buffer before revealing safe chunks.

Practical Client-Side Patterns

Here are the patterns I recommend for most chat products:

1. Show Status Before Text Arrives

Do not leave the assistant bubble empty. Use explicit states:

• “Reading context…”
• “Searching documents…”
• “Drafting answer…”
• “Still working…”

This is especially useful when tool calls happen before text generation.

2. Keep a Real Stop Button

A stop button is not just UX decoration. It can save output tokens if implemented correctly.

Make sure:

• The browser aborts the request.
• Your backend detects disconnects.
• Your backend aborts the provider request.
• You record the request as canceled.

3. Render Incrementally, Format Finally

During streaming:

• Append text.
• Render lightweight Markdown.
• Avoid heavy syntax highlighting.
• Keep scroll behavior predictable.

After completion:

• Re-parse Markdown.
• Highlight code.
• Enable copy buttons.
• Attach citations or references if your system supports them.

4. Store Partial Messages Carefully

If a stream fails halfway through, decide what the transcript means.

Options:

• Keep partial assistant message with “interrupted” marker.
• Remove it entirely.
• Let user retry from the same prompt.
• Ask the model to continue from partial context.

I usually keep the partial message visibly marked. Silent deletion confuses users, and pretending it completed is worse.

5. Normalize Events

Your frontend should not know five provider-specific streaming formats. Normalize in the backend:

{"type":"delta","text":"Hello"}
{"type":"tool_start","name":"search_docs"}
{"type":"tool_result","name":"search_docs","status":"ok"}
{"type":"done","finish_reason":"stop","usage":{"input_tokens":100,"output_tokens":50}}
{"type":"error","message":"upstream timeout"}

This also makes it easier to switch between model families or route requests dynamically. If you use a multi-model API layer, this normalization may already be part of your gateway; otherwise it is worth building yourself.

Practical Takeaways

• Streaming improves perceived latency by reducing time-to-first-visible-output, not by making total generation free or instant.
• Measure TTFT, total completion time, and tokens per second separately.
• SSE is usually the simplest transport for LLM text streaming; use fetch() streams when you need POST bodies.
• Watch for proxy and framework buffering. Many “streaming bugs” are infrastructure buffering bugs.
• Treat partial output as invalid: Markdown may be incomplete, JSON may not parse, and code fences may still be open.
• Cancellation only saves cost if the abort propagates from browser to backend to provider.
• Usage metadata often arrives at the end of the stream; interrupted streams need careful accounting.
• Do not re-render your entire UI on every tiny chunk. Batch updates with animation frames.
• Normalize provider-specific stream events behind your backend.
• Skip streaming for short classifications, batch jobs, strict structured outputs, or workflows requiring full moderation before display.