DigitalOcean Inference, streamed to a Next.js UI

A minimal Next.js App Router app that streams chat completions from DigitalOcean's Inference Engine to the browser, token by token.

The interesting part is not the model call. It is the proxy in the middle, and whether it preserves the streaming or quietly destroys it.

Why streaming, with numbers

Measured against openai-gpt-oss-120b on 17 August 2026, median of 3 runs:

	time to first token	total
stream: false	15,706 ms	15,706 ms
stream: true, direct to DO	1,265 ms	15,435 ms
stream: true, through this app's route handler	1,388 ms	~7,500 ms

Same model, same prompt. Without streaming the user watches a blank screen for nearly 16 seconds. With it they see the first words in about 1.3 seconds.

The proxy costs roughly 120 ms. That is the number worth checking in your own app, because a route handler that awaits the whole upstream body before responding turns the second row back into the first.

Two things that will bite you

Not every model in /v1/models is callable. The endpoint lists 76 models. Several return 403 this model is not available for your subscription tier. Check before you build a model picker from that list.

Reasoning models do not stream the way you expect. alibaba-qwen3-32b took 7,864 ms to emit its first token even with stream: true, because it thinks before it answers. Streaming does not help if the model is quiet for eight seconds. Model choice matters more than the flag.

Run it

cp .env.example .env.local   # add your DO_INFERENCE_KEY
npm install
npm run dev

Create a model access key in the DigitalOcean console under GradientAI Platform → Model access keys.

What to look at

• lib/stream.ts — the proxy. Pipes chunks through without buffering, handles SSE frames split across network reads, and cancels upstream when the client disconnects.
• app/api/chat/route.ts — the route handler. Node runtime rather than edge, and it forwards req.signal so aborting actually stops the generation.
• app/page.tsx — the client. Renders tokens as they arrive and shows the measured time to first token.

Tests

npm test

Covers the SSE frame parser, including the case that matters: a frame split across two network reads, which silently drops a token if you parse naively.

Licence

MIT