bug: `stream` defaults to `true`, so non-streaming OpenAI-compatible clients receive SSE and fail

[penradioGables]

Summary

ChatCompletionRequest.stream defaults to True in routers/gateway.py. The OpenAI API specifies stream as defaulting to false when the field is omitted, so any client that wants a single JSON response — and therefore omits the field — silently receives an event-stream instead, and fails to parse it.

The agent itself answers correctly; only the transport framing is wrong.

Version: cptr 0.9.10, Python 3.12, SQLite backend. Reproduced against a running instance.

Root cause

routers/gateway.py:

class ChatCompletionRequest(BaseModel):
    model: str
    messages: list[dict]
    stream: bool = True        # <-- OpenAI's documented default is false

The dispatch below it is correct, and the non-streaming branch is fully implemented:

    if body.stream:
        return StreamingResponse(
            _stream(output_queue, completion_id, created, body.model),
            media_type="text/event-stream",
            ...
        )
    else:
        # Non-streaming: collect all text, return as single response
        full_text = await _collect(output_queue)
        return {
            "id": completion_id,
            "object": "chat.completion",
            ...
        }

_collect() drains the queue, handles delta / output / done / error events, and returns a well-formed chat.completion object. The else: branch appears correct — it is simply unreachable in practice, because real clients omit stream rather than sending false explicitly.

Reproduction

# Fails: returns an SSE body where a JSON object is expected
curl -X POST http://<cptr>:8000/v1/chat/completions \
  -H "Authorization: Bearer <key>" -H "Content-Type: application/json" \
  -d '{"model":"cptr/cptr","messages":[{"role":"user","content":"hi"}]}'

# Works: HTTP 200
curl -X POST http://<cptr>:8000/v1/chat/completions \
  -H "Authorization: Bearer <key>" -H "Content-Type: application/json" \
  -d '{"model":"cptr/cptr","stream":true,"messages":[{"role":"user","content":"hi"}]}'

Behind a proxy the failure surfaces as a 500. Via LiteLLM:

litellm.InternalServerError: OpenAIException - Empty or invalid response from LLM
endpoint. Received: 'data: {"id": "chatcmpl-...", "object": "chat.completion.chunk",
"model": "cptr/cptr", "choices": [{"index": 0, "delta": {"role": "assistant",
"content": ""}, "finish_reason": null}]}
...

The correct answer is visible inside that unparsed stream.

There is no client-side workaround

Setting "stream": false explicitly does not help when a proxy sits in between. Verified against LiteLLM 1.95.0: an explicit "stream": false in the client request still produces the same failure, because LiteLLM omits the field on the wire for non-streaming calls rather than forwarding false. The caller cannot reach the else: branch at all.

Suggested fix

 class ChatCompletionRequest(BaseModel):
     model: str
     messages: list[dict]
-    stream: bool = True
+    stream: bool = False

This should be safe for existing consumers: Open WebUI sets stream: true explicitly, so its behaviour is unchanged. Only clients that currently omit the field are affected, and those are the ones that are broken today.

Secondary, in the same branch

The non-streaming response hardcodes token counts to zero:

            "usage": {
                "prompt_tokens": 0,
                "completion_tokens": 0,
                "total_tokens": 0,
            },

This is currently invisible because the branch is unreachable. Once the default is corrected it becomes live, and any proxy or client doing token accounting off usage will record zeros for every non-streaming call. The streaming path already surfaces real counts.

Happy to open a PR for the one-line change if that is useful.

[owui-terminator]

🔍 Related Issues Found

I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:

1. 🟣 #96 bug: OpenAI-compatible streaming usage can end responses early
   Related only at the broad component level: it also concerns OpenAI-compatible chat/completions behavior in the same codepath, but the bug is about premature termination on streaming usage events, not the default stream parameter or SSE vs JSON framing.
   by sandrogauci

---

💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead.

This comment was generated automatically. React with 👍 if helpful, 👎 if not.