translator: preserve reasoning in both directions for Anthropic-to-OpenAI with vLLM by anencore94 · Pull Request #2956 · theagentrouter/agent-router

anencore94 · 2026-11-01T07:35:07Z

Description

This commit fixes reasoning loss in both directions when translating between the Anthropic Messages API and OpenAI Chat Completions with vLLM (>= v0.30.0) backends:

Response path (OpenAI -> Anthropic): vLLM renamed reasoning_content to reasoning (choices[].message.reasoning in non-streaming and choices[].delta.reasoning in streaming) and no longer emits reasoning_content. Add Reasoning (json:"reasoning,omitempty") to ChatCompletionResponseChoiceMessage and ChatCompletionResponseChunkChoiceDelta, redact it in redactResponseMessage, and fall back to reasoning in both non-streaming and streaming response translation when reasoning_content is unset or empty (ThinkingBlocks > ReasoningContent > Reasoning).
Request path (Anthropic -> OpenAI): appendAnthropicAssistantMessage previously encoded assistant thinking blocks as structured content parts ({"type":"thinking","text":...}), which vLLM ignores when rendering assistant reasoning into the chat template (<think>...</think>). Place assistant thinking text on the message as reasoning_content and preserve structured thinking (with signatures) and redacted_thinking blocks in thinking_blocks, matching the convention established in translator: stream reasoning_content as plain string, signatures via thinking_blocks #2319 and translator: serialize non-streaming Bedrock reasoning_content as plain string #2524.

Related Issues/PRs (if applicable)

Fixes #2937

…enAI with vLLM Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>

netlify · 2026-11-01T07:35:14Z

✅ Deploy Preview for theagentrouter canceled.

Name	Link
🔨 Latest commit	`9501f60`
🔍 Latest deploy log	https://app.netlify.com/projects/theagentrouter/deploys/6aedc6aa88cbc8622a1339cd

anencore94 · 2026-11-02T18:36:24Z

Hi @aabchoo @nacx @yuzisun — could you please approve the CI workflow run and take a look at this PR when you have a moment? It addresses both the response (choices[].message.reasoning / choices[].delta.reasoning) and multi-turn request (reasoning_content + thinking_blocks on assistant messages) sides of #2937. Thanks!

kevinkswSiloAI · 2026-11-05T15:21:44Z

We're hitting the same problem described in #2937 and would like to see this land.

Our setup: Envoy AI Gateway runs in front of vLLM, which serves reasoning models. Clients call the Anthropic Messages API (/v1/messages), Claude Code among them, and the gateway translates to vLLM's Chat Completions API.

Symptom:

Without streaming: the Anthropic response has only the text block, with no thinking block.
With streaming: no thinking content block is ever started and no thinking_delta events are sent. The client gets the text deltas only.

What we found while debugging:

vLLM's raw response does contain the reasoning, in choices[].message.reasoning (non-streaming) and choices[].delta.reasoning (streaming). Calling vLLM directly with the same request returns it, so the loss happens in the gateway's translation step.
ChatCompletionResponseChoiceMessage and ChatCompletionResponseChunkChoiceDelta have no reasoning field, so the reasoning is discarded when the gateway parses vLLM's response.
Then appendThinkingBlocks (non-streaming) and the stream chunk handler build thinking blocks only from thinking_blocks and reasoning_content. vLLM sends neither, so no thinking block is produced and there is no error or log line.
vLLM removed reasoning_content from its responses (reasoning_content -> reasoning vllm-project/vllm#27752, Remove deprecated reasoning_content message field vllm-project/vllm#33402, Remove deprecated reasoning_content message field(part-2) vllm-project/vllm#37480) and has no option to bring it back, so every Anthropic → vLLM user through the gateway is likely affected.

This PR's response-side change addresses exactly that cause, and the precedence (thinking_blocks > reasoning_content > reasoning) is the one we'd expect. It also fixes the request-side history problem from #2937, which we hadn't looked into.

Related PR: #2945 by @hustxiayang fixes the same response-side problem in nearly the same way: it reads reasoning from the message and stream delta when reasoning_content is absent, and redacts it. It doesn't change the request side, though, so the multi-turn history problem from #2937 would remain. Since this PR fixes both halves of #2937, could it be considered as the fix for the issue?

@aabchoo @nacx @yuzisun, can we consider this one to be merged to fix the reasoning functionality for vLLM?

Thanks @anencore94!

codecov · 2026-11-05T17:36:12Z

Codecov Report

❌ Patch coverage is 68.00000% with 16 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/translator/openai_helper.go	65.95%	16 Missing ⚠️

📢 Thoughts on this report? Let us know!

aabchoo · 2026-11-05T17:42:13Z

Will take a look sometime this week or next. Could you fix the codecov for now?

aabchoo · 2026-11-05T18:05:26Z

-	if len(rc.ReasoningContent.RedactedContent) > 0 {
+
+	// Fall back to Reasoning (used by vLLM >= v0.30.0, which renamed reasoning_content to reasoning).
+	if msg.Reasoning != nil && *msg.Reasoning != "" {


fall back to reasoning in both non-streaming and streaming response translation when reasoning_content is unset or empty

This function doesn't actually do that? This only checks if reasoning is valid

Fixed in 9501f60. The nonempty string branch already returned early, but an empty structured reasoning_content object could also return early and suppress the reasoning fallback. The structured branch now returns only when it appends meaningful text/signature or redacted content. Added wire-format response regressions for absent/null/empty content and structured objects, plus precedence checks for signed and redacted content.

aabchoo · 2026-11-05T19:11:25Z

-		if text != "" {
-			assistantMsg.Content = openai.StringOrAssistantRoleContentUnion{Value: text}
-		}
+	text := anthropicContentToText(msg.Content)


side note, anthropicContentToText joins without a separator which is very different from the above.

Changed in 9501f60 to concatenate without inserting a newline, matching anthropicContentToText. The regression preserves whitespace supplied by the original blocks exactly, while retaining the individual signed thinking_blocks unchanged.

aabchoo · 2026-11-05T19:14:48Z

 	}

+	// Redact reasoning from backends that name the field "reasoning" (vLLM)
+	if msg.Reasoning != nil {


should this be treated as a fall back as well?

I kept the two redactions independent. This function sanitizes the object for logging rather than selecting the response field: both fields can coexist, so redacting only the preferred field could leave the other field’s private reasoning in the logged object. Added a regression in 9501f60 that supplies both fields, verifies both are redacted, and verifies the original object is unchanged.

aabchoo · 2026-11-05T19:15:37Z

+			if block.Thinking.Thinking != "" {
+				reasoningTexts = append(reasoningTexts, block.Thinking.Thinking)
+			}
+			thinkingBlocks = append(thinkingBlocks, openai.ThinkingBlock{


Worth checking if thinking and signature is empty?

Fixed in 9501f60: skip a thinking block when both thinking and signature are empty. Nonempty unsigned text and signature-only blocks remain preserved, as do redacted-thinking blocks. Added regression cases for each boundary.

Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>

anencore94 · 2026-11-06T12:24:13Z

@aabchoo pushed 9501f60 with the review fixes and coverage additions; I replied to each inline thread.

Added 17 boundary/regression cases. Four cases fail before the fixes and pass afterward. make precommit, final lint (0 issues), and make test-coverage GO_TEST_ARGS='-race -json' pass locally: 76 packages, 8,287 test/subtest pass events, with the recording test skipped. Repository-filtered coverage is 89.7%; appendThinkingBlocks and redactResponseMessage each have 100% statement coverage. Local mapping covers all 52 changed executable lines across the PR, but this is not an uploaded Codecov result; upstream CI/Codecov for the new head remains to be verified.

No live-provider or GPU E2E pass is claimed. This follow-up used OpenAI Codex for implementation, review and testing.

translator: preserve reasoning in both directions for Anthropic-to-Op…

2266110

…enAI with vLLM Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>

anencore94 requested a review from a team as a code owner November 1, 2026 07:35

hustxiayang mentioned this pull request Nov 5, 2026

fix: fix the message api translation of "reasoning" for vLLM backend #2945

Closed

Merge branch 'main' into fix/issue-2937-vllm-reasoning

bee15ef

aabchoo self-assigned this Nov 5, 2026

aabchoo reviewed Nov 5, 2026

View reviewed changes

fix: preserve reasoning fallback and cover translation boundaries

9501f60

Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>

Conversation

anencore94 commented Nov 1, 2026

Uh oh!

netlify Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for theagentrouter canceled.

Uh oh!

anencore94 commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinkswSiloAI commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Nov 5, 2026

Codecov Report

Uh oh!

aabchoo commented Nov 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anencore94 commented Nov 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

netlify Bot commented Nov 1, 2026 •

edited

Loading

anencore94 commented Nov 2, 2026 •

edited

Loading

kevinkswSiloAI commented Nov 5, 2026 •

edited

Loading