Parking the findings from the 2026-07-30 OpenRouter experiment so the data and the idea are on record. No action planned. Written up by Molty on Matt's behalf.
Why we looked
Explain's median latency is ~7s, dominated by output-token decode speed. Fast inference hosts (Cerebras, Groq) serve open-weight models at 10-20x our decode rate. Question: can any of them clear our accuracy bar?
Bake-off setup
All 21 prompt-test cases, production prompt, via OpenRouter (test key, since revoked-able; no production code touched). Judged by Opus 4.8 with adaptive thinking. Caveat: our prod baseline (19/21) was judged by Opus 4.7 in an earlier eval; the same-judge in-bake-off reference for Sonnet 5 is 16/21.
| Model |
Correct |
p50 |
p90 |
$/1k reqs |
| Sonnet 5 direct (prod, earlier eval, 4.7 judge) |
19/21 |
7.2s |
9.5s |
$15.10 |
| Sonnet 5 via OpenRouter (routed to Bedrock) |
16/21 |
8.6s |
13.2s |
$9.82 (intro) |
gpt-oss-120b (Cerebras/Groq, :nitro) |
13/21 |
0.7s |
1.5s |
$0.19 |
| gemini-3.5-flash |
17/21 |
8.4s |
10.2s |
$13.76 |
| deepseek-v4-flash (reasoning off) |
16/21 |
9.4s |
17.3s |
$0.33 |
qwen3.7-flash was disqualified: its single provider (Alibaba) 429'd even sequential polite traffic.
Findings
- gpt-oss-120b on Cerebras/Groq is the only real speed win: 10x faster, 80x cheaper. But it makes exactly the class of error that got Haiku 4.5 evicted in
d188a44 (invented tail-recursion-elimination narratives, register misattribution). Not acceptable as the default explainer.
- gemini-3.5-flash matches Sonnet 5 on accuracy but is no faster and barely cheaper. No reason to switch.
- The OpenRouter hop itself costs ~1.5s of latency (Sonnet 5 via OR routed to Bedrock vs direct). If a provider seam is ever built: keep Claude direct, proxy only the others.
- History check before proposing any down-tier: Haiku 4.5 was the original model and was replaced for accuracy in
d188a44. In-family down-tiering is a dead end for the default path.
The parked idea: fast-draft tier
A sub-second "quick take" from gpt-oss-120b, explicitly labelled as a draft, either as an opt-in request flag or a progressive UI where the fast answer renders immediately and the Sonnet answer replaces it when ready. The accuracy gap is arguably tolerable for a clearly-badged draft that gets corrected seconds later. Plumbing-wise this is small: a second provider path selected per-request (the build_api_payload seam makes this an afternoon, not a project).
Privacy implications (the real blocker, read this first if reviving)
Today, user code goes to exactly one AI vendor: Anthropic. Any OpenRouter-based tier changes that materially:
- A middleman is added. OpenRouter sees every request. Their logging/retention is configurable per key, but it is a new party.
- The upstream host varies per request.
:nitro routing sent our traffic to Cerebras and Groq; deepseek went to eleven different hosts including Alibaba and DeepInfra. Each has its own retention and training policy; some providers train on inputs unless explicitly excluded.
- CE's public privacy story must be updated first, not after: the site currently tells users their code goes to Anthropic/Claude. A fast tier needs in-UI disclosure, an updated privacy policy, and OpenRouter's data-policy filters pinned to no-training/ZDR providers only (which shrinks the provider pool and may cost some of the speed).
- Alternative that sidesteps 1-3: gpt-oss-120b is open-weight. Self-hosting it (even on modest GPU) keeps code in-house entirely, at the cost of not getting Cerebras-class decode speed. Worth costing if the idea is ever revived.
Revisit triggers
- A fast host serves a model that matches Sonnet-tier accuracy on our eval (rerun: the bake-off harness is a standalone script; ask Molty or rebuild from this issue).
- Product appetite for a clearly-labelled draft tier plus willingness to do the privacy-policy work.
🤖 Written by Molty (Claude) on behalf of @mattgodbolt
Parking the findings from the 2026-07-30 OpenRouter experiment so the data and the idea are on record. No action planned. Written up by Molty on Matt's behalf.
Why we looked
Explain's median latency is ~7s, dominated by output-token decode speed. Fast inference hosts (Cerebras, Groq) serve open-weight models at 10-20x our decode rate. Question: can any of them clear our accuracy bar?
Bake-off setup
All 21 prompt-test cases, production prompt, via OpenRouter (test key, since revoked-able; no production code touched). Judged by Opus 4.8 with adaptive thinking. Caveat: our prod baseline (19/21) was judged by Opus 4.7 in an earlier eval; the same-judge in-bake-off reference for Sonnet 5 is 16/21.
:nitro)qwen3.7-flash was disqualified: its single provider (Alibaba) 429'd even sequential polite traffic.
Findings
d188a44(invented tail-recursion-elimination narratives, register misattribution). Not acceptable as the default explainer.d188a44. In-family down-tiering is a dead end for the default path.The parked idea: fast-draft tier
A sub-second "quick take" from gpt-oss-120b, explicitly labelled as a draft, either as an opt-in request flag or a progressive UI where the fast answer renders immediately and the Sonnet answer replaces it when ready. The accuracy gap is arguably tolerable for a clearly-badged draft that gets corrected seconds later. Plumbing-wise this is small: a second provider path selected per-request (the
build_api_payloadseam makes this an afternoon, not a project).Privacy implications (the real blocker, read this first if reviving)
Today, user code goes to exactly one AI vendor: Anthropic. Any OpenRouter-based tier changes that materially:
:nitrorouting sent our traffic to Cerebras and Groq; deepseek went to eleven different hosts including Alibaba and DeepInfra. Each has its own retention and training policy; some providers train on inputs unless explicitly excluded.Revisit triggers
🤖 Written by Molty (Claude) on behalf of @mattgodbolt