Docs

Streaming

SSE, reading chunks, end-of-stream, and how a mid-stream failure is billed.

With stream=true the response arrives as SSE and the first tokens reach the user far sooner. For any live chat UI this is the right default.

resp = client.chat.completions.create(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "یک داستان کوتاه بنویس"}],
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

The raw shape

data: {"id":"chatcmpl-...","choices":[{"delta":{"content":"سلام"},"index":0}]}

data: {"id":"chatcmpl-...","choices":[{"delta":{"content":"!"},"index":0}]}

data: [DONE]

• The stream ends with data: [DONE].
• The id and model fields stay constant for the whole stream.
• Response headers (including x-request-id) are flushed before the first chunk, so they are always available.

If it breaks mid-stream

Once the first byte is out we cannot fail over — swapping models mid-answer produces a response nobody wants. Tokens genuinely produced up to that point are metered and charged, because we paid upstream for them too.

Behind nginx or a load balancer

We set X-Accel-Buffering: no so intermediaries do not buffer the stream. If a response arrives all at once instead of flowing, the buffering is in your own proxy.