Streaming chat completions correctly: SSE, usage chunks and retries

TL;DR: Streaming LLM API responses reduces perceived latency by sending tokens as they are generated, but requires correct Server-Sent Events (SSE) parsing and explicit stream_options to receive usage data. Handling connection drops, rate limits, and insufficient balance errors is essential for a robust implementation. The TokShop API supports this OpenAI-compatible streaming standard.

How do you handle errors and retries in streaming?

Streaming connections are fragile and can be interrupted by network issues, server restarts, or rate limits. To handle this, implement a retry strategy that resends the entire request upon failure, as the API is stateless. Specific errors like HTTP 402 (insufficient balance) or HTTP 429 (rate limiting) should be caught and handled appropriately, with the OpenAI SDK providing built-in retry logic for some cases.

Understanding the SSE format for chat completions

Server-Sent Events is a simple text-based protocol where each event is a line starting with data: followed by a JSON payload. For OpenAI-compatible streaming, the stream sends multiple events:

• Content chunks — data: {"choices":[{"delta":{"content":"Hello"}}]}
• Finish reason — data: {"choices":[{"delta":{},"finish_reason":"stop"}]}
• Usage chunk — data: {"usage":{"prompt_tokens":10,"completion_tokens":5}} (sent only if stream_options={"include_usage": true})
• Stream end — data: [DONE]

A common mistake is assuming the usage chunk always appears. By default, OpenAI-compatible APIs omit usage in streaming mode to reduce latency. To get token counts, you must explicitly set stream_options in your request.

Example request with usage tracking

import openai

client = openai.OpenAI(
    api_key="sk-tok-your-key-here",
    base_url="https://tokshop.xyz/v1"
)

response = client.chat.completions.create(
    model="deepseek-v3.2",
    messages=[{"role": "user", "content": "Explain SSE in one sentence."}],
    stream=True,
    stream_options={"include_usage": True}  # <-- critical
)

Without include_usage, the stream ends after the final content chunk, and you have no idea how many tokens were used. With it, the second-to-last event (before [DONE]) contains the usage object.

Parsing the stream correctly

The OpenAI Python SDK handles SSE parsing internally when you iterate over the response. But if you're working with raw HTTP or another language, you need to parse line by line.

Python: using the SDK (recommended)

full_content = ""
usage = None

for chunk in response:
    if chunk.choices and chunk.choices[0].delta.content:
        full_content += chunk.choices[0].delta.content
        print(chunk.choices[0].delta.content, end="")
    if chunk.usage:
        usage = chunk.usage
        print(f"\n\nUsage: {usage.prompt_tokens} prompt + {usage.completion_tokens} completion tokens")

Raw HTTP parsing (for non-Python clients)

import requests
import json

resp = requests.post(
    "https://tokshop.xyz/v1/chat/completions",
    headers={
        "Authorization": "Bearer sk-tok-your-key-here",
        "Content-Type": "application/json"
    },
    json={
        "model": "deepseek-v3.2",
        "messages": [{"role": "user", "content": "Hi"}],
        "stream": True,
        "stream_options": {"include_usage": True}
    },
    stream=True
)

for line in resp.iter_lines():
    if line:
        decoded = line.decode("utf-8")
        if decoded.startswith("data: "):
            payload = decoded[6:]
            if payload == "[DONE]":
                break
            data = json.loads(payload)
            if "choices" in data and data["choices"]:
                delta = data["choices"][0].get("delta", {})
                if "content" in delta:
                    print(delta["content"], end="")
            if "usage" in data:
                print(f"\nUsage: {data['usage']}")

Note: the usage chunk may arrive after the last content chunk but before [DONE]. Always check for the usage key in every event.

Handling errors and retries in streaming

Streaming connections are more fragile than non-streaming ones. Network blips, server restarts, or rate limits can interrupt the stream mid-response. Here's how to handle common scenarios.

HTTP 402: Insufficient balance

TokShop uses prepaid USD credits. If your balance runs out mid-stream, the server will close the connection with HTTP 402 and a JSON body {"error": {"code": "insufficient_balance"}}. Your client should catch this and either prompt the user to top up or switch to a cheaper model.

try:
    response = client.chat.completions.create(...)
    for chunk in response:
        # process chunk
        pass
except openai.APIStatusError as e:
    if e.status_code == 402:
        print("Insufficient balance. Visit https://tokshop.xyz/pricing to add credits.")
    else:
        raise

Connection drops mid-stream

If the TCP connection drops (e.g., WiFi outage), you'll get an APIConnectionError. The safest retry strategy is to resend the entire request with the same message history. This works because the API is stateless — each request is independent.

import time
from openai import APIError

def stream_with_retry(client, model, messages, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=messages,
                stream=True,
                stream_options={"include_usage": True}
            )
            for chunk in response:
                yield chunk
            return  # success, exit retry loop
        except (APIError, ConnectionError) as e:
            if attempt == max_retries - 1:
                raise
            wait = 2 ** attempt  # exponential backoff: 1s, 2s, 4s
            print(f"Stream failed (attempt {attempt+1}), retrying in {wait}s...")
            time.sleep(wait)

Rate limiting (HTTP 429)

If you hit rate limits, the server returns 429. The OpenAI SDK automatically retries on 429 with exponential backoff by default (configurable via max_retries). For streaming, the same applies — but note that retrying resets the entire stream, so you'll lose any tokens already received.

client = openai.OpenAI(
    api_key="sk-tok-...",
    base_url="https://tokshop.xyz/v1",
    max_retries=2  # default is 2
)

Choosing the right model for streaming

Different models have different generation speeds, which affects streaming latency. As of recent reports, smaller models like DeepSeek V3.2 tend to stream faster token-by-token than larger models with longer context windows.

TokShop offers several models at competitive per-token prices. For streaming use cases where cost matters, consider:

• DeepSeek V3.2 — $0.42/M input tokens, $0.63/M output tokens, 128K context. Good balance of speed and capability.
• GLM 4.6 — $0.9/M input, $3.3/M output, 200K context. Slower but handles very long conversations.
• Qwen3 Coder — $2.25/M input, $11.25/M output, 262K context. Optimized for code generation.

If you're building a chat UI that streams responses, start with a faster, cheaper model and only escalate to larger ones when the task requires it. Check the pricing page for the latest model list.

Practical patterns for production streaming

1. Always request usage in streaming

Set stream_options={"include_usage": True} in every streaming request. Without it, you lose visibility into costs. The TokShop dashboard logs every call with token counts, but the streaming usage chunk gives you real-time feedback.

2. Buffer partial content for display

Don't render each character individually — batch small chunks (e.g., 10ms intervals) to reduce DOM updates in a web UI. The SDK already does this at the API level, but if you're building a frontend, implement a simple debounce.

3. Handle the [DONE] marker

The [DONE] event signals the end of the stream. If you miss it (e.g., due to a parsing bug), your loop may hang waiting for more data. In the raw HTTP approach, explicitly break on [DONE].

4. Log stream interruptions

If a stream fails mid-response, log the partial content and the error. This helps debug whether the issue is network-related or a server-side problem. You can also implement a "resume" feature by asking the user to rephrase their query.

Conclusion

Streaming chat completions with SSE is straightforward once you understand the event format and common failure modes. Always request usage data with stream_options, parse events correctly (including the final usage chunk), and implement retry logic with exponential backoff for transient errors.

For production use, test your streaming implementation against different models and network conditions. The TokShop API follows the OpenAI standard, so code written for OpenAI works without modification — just point the base URL to https://tokshop.xyz/v1. For more details on model pricing and rate limits, visit the docs.

FAQ

How do you get token usage data when streaming?

To receive token usage data (prompt_tokens and completion_tokens) in a streaming response, you must explicitly set stream_options={"include_usage": True} in your request. Without this parameter, the usage chunk is omitted by default to reduce latency.

What should you do if the streaming connection drops?

If the TCP connection drops mid-stream, you should implement a retry strategy that resends the entire original request, as the API is stateless. Using exponential backoff (e.g., 1s, 2s, 4s) between retries is a recommended practice to handle transient failures.

Why is the [DONE] marker important in the SSE stream?

The data: [DONE] event signals the definitive end of the Server-Sent Events stream. Correctly parsing and breaking your read loop upon receiving this marker is crucial to prevent your client from hanging while waiting for more data that will never arrive.