Skip to content

feat(qairt): native stop sequences in the geniex-qairt pipeline - #1342

Open
MAN$I VERMA (mansiverma897993) wants to merge 2 commits into
qualcomm:mainfrom
mansiverma897993:fix/completions-host-stop
Open

feat(qairt): native stop sequences in the geniex-qairt pipeline#1342
MAN$I VERMA (mansiverma897993) wants to merge 2 commits into
qualcomm:mainfrom
mansiverma897993:fix/completions-host-stop

Conversation

@mansiverma897993

@mansiverma897993 MAN$I VERMA (mansiverma897993) commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Fixes #1341

FIM clients always send a stop list, /v1/completions forwarded it to the plugin unchanged, and the qairt plugin rejects any generate call with stop sequences (-100016), so completions failed on every QAIRT model.

For non-llama_cpp runtimes the stop list is no longer forwarded. A stopScanner enforces it host-side: it matches the stop strings against the streamed tokens, holds back the longest tail that could still start a match so a stop spanning token boundaries never leaks, cancels generation through the token callback on a match (stop_reason: "user", mapped to finish_reason: "stop"), and truncates the output at the match. Applied to both the streaming and blocking paths. llama_cpp keeps its native stop handling, so existing behavior is unchanged there.

Test report

TestStopScanner (added in completion_test.go), 7 cases: no match, match inside one token, match spanning tokens, earliest of multiple stops wins, stop at start, empty stop strings pass through, held tail flushed at end.

Verified end-to-end without the SDK toolchain via a pure-Go stub harness (exact-signature geniex_sdk stubs, real gin HTTP server, fake plugin that emits tokens through OnToken and honors cancellation), same pattern as #1317:

--- PASS: TestE2EQairtStreamingStopSpanningTokens   stop split across tokens; SSE text truncated, finish_reason stop, plugin got Stop=nil, generation cancelled early
--- PASS: TestE2EQairtStreamingNoMatchFlushesTail   held-back tail emitted when generation ends without a match
--- PASS: TestE2EQairtStreamingLength               finish_reason length preserved when max_tokens is hit
--- PASS: TestE2EQairtBlockingStop                  blocking JSON response truncated at the stop, finish_reason stop
--- PASS: TestE2EQairtBlockingNoStopsUnchanged      no stops: no OnToken registered, response unchanged
--- PASS: TestE2EQairtIssue1341Replay              exact request from #1341 (10-entry FIM stop list, streaming): 200, truncated completion, no -100016
--- PASS: TestE2ELlamaCppStopsForwardedToPlugin     llama_cpp still receives the stop list natively
--- PASS: TestCompletionPrompt / TestCompletionStop / TestStopScanner / TestCompletionUnsupported
ok      github.com/qualcomm/GenieX/cli/server/handler

go vet clean, gofmt clean.

The qairt plugin rejects any generate call carrying stop sequences
(PARAM_NOT_SUPPORTED, -100016), and FIM autocompletion clients always
send a stop list, so /v1/completions failed on every QAIRT model.

For non-llama_cpp runtimes the stop list is no longer forwarded to the
plugin. A stopScanner matches the stop strings against the streamed
tokens instead, holding back the longest tail that could still start a
match so a stop spanning token boundaries never leaks, cancels
generation through the token callback on a match, and truncates the
returned text at the match. llama_cpp keeps its native stop handling.

Fixes qualcomm#1341

Signed-off-by: mansiverma897993 <vmansi756@gmail.com>
@RemiliaForever

Copy link
Copy Markdown
Contributor

The clean fix is native stop support in the geniex-qairt pipeline — generateTokens in llm_pipeline.cpp already decodes per-token, accumulates full_text, and reports stop_reason, so it'd match llama_cpp and cover every consumer. In the plugin or handler it's just a byte-level patch outside the decode loop.

Implements stop sequences natively in the geniex-qairt pipeline instead
of enforcing them host-side, as requested in review: the pipeline's
generateTokens now matches stop strings byte-wise against the streamed
output (mirroring llama_cpp), so every consumer gets the same behavior.

The qairt plugin adapter now forwards the FFI stop list into
GenerationConfig::stop_sequences instead of rejecting it with
PARAM_NOT_SUPPORTED (-100016), and maps the pipeline's new
"stop_sequence" stop_reason through to profile_data. The temporary
host-side stopScanner in the /v1/completions handler is removed — the
handler forwards the stop list to every plugin as it already did before,
and llama_cpp keeps its native handling unchanged.

Bumps third-party/geniex-qairt to e90b66b (feat/code):
native stop-sequence support in the LLM pipeline.

Fixes qualcomm#1341

Signed-off-by: mansiverma897993 <vmansi756@gmail.com>
@mansiverma897993

Copy link
Copy Markdown
Contributor Author

Thanks for the review RemiliaForever (@RemiliaForever)! Addressed - stop is now handled natively in the geniex-qairt pipeline instead of being enforced host-side.

What changed:

  • generateTokens in llm_pipeline.cpp now matches the stop sequences byte-wise against the streamed output. A StopMatcher holds back the longest tail that could still start a match (so a stop spanning token boundaries never leaks through the token callback), cancels generation on a match, truncates the output at the match, and reports stop_reason: "stop_sequence" - matching llama_cpp and covering every consumer of the pipeline.
  • The qairt plugin adapter (sdk/plugins/qairt/src/llm.cpp) now forwards the FFI stop list into GenerationConfig::stop_sequences instead of rejecting it with PARAM_NOT_SUPPORTED (-100016), and maps the new stop_sequence stop reason through to profile_data.
  • The temporary host-side stopScanner in the /v1/completions handler is removed; the handler forwards the stop list to every plugin as it did before the workaround. llama_cpp keeps its native handling unchanged.
  • Bumped third-party/geniex-qairt to the plugin commit with native stop support (mansiverma897993/geniex-qairt-plugin feat/native-stop), which includes the StopMatcher unit tests (byte-level matching, boundary-spanning stops, earliest-stop-wins, held-back tail flush).

Fixes #1341 either way - no more -100016 on QAIRT models.

@mansiverma897993 MAN$I VERMA (mansiverma897993) changed the title fix(server): enforce completion stop sequences host-side for qairt feat(qairt): native stop sequences in the geniex-qairt pipeline Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

/v1/completions fails on QAIRT models because the qairt plugin rejects stop sequences

2 participants