feat(qairt): native stop sequences in the geniex-qairt pipeline - #1342
Open
MAN$I VERMA (mansiverma897993) wants to merge 2 commits into
Open
feat(qairt): native stop sequences in the geniex-qairt pipeline#1342MAN$I VERMA (mansiverma897993) wants to merge 2 commits into
MAN$I VERMA (mansiverma897993) wants to merge 2 commits into
Conversation
The qairt plugin rejects any generate call carrying stop sequences (PARAM_NOT_SUPPORTED, -100016), and FIM autocompletion clients always send a stop list, so /v1/completions failed on every QAIRT model. For non-llama_cpp runtimes the stop list is no longer forwarded to the plugin. A stopScanner matches the stop strings against the streamed tokens instead, holding back the longest tail that could still start a match so a stop spanning token boundaries never leaks, cancels generation through the token callback on a match, and truncates the returned text at the match. llama_cpp keeps its native stop handling. Fixes qualcomm#1341 Signed-off-by: mansiverma897993 <vmansi756@gmail.com>
Contributor
|
The clean fix is native |
Implements stop sequences natively in the geniex-qairt pipeline instead of enforcing them host-side, as requested in review: the pipeline's generateTokens now matches stop strings byte-wise against the streamed output (mirroring llama_cpp), so every consumer gets the same behavior. The qairt plugin adapter now forwards the FFI stop list into GenerationConfig::stop_sequences instead of rejecting it with PARAM_NOT_SUPPORTED (-100016), and maps the pipeline's new "stop_sequence" stop_reason through to profile_data. The temporary host-side stopScanner in the /v1/completions handler is removed — the handler forwards the stop list to every plugin as it already did before, and llama_cpp keeps its native handling unchanged. Bumps third-party/geniex-qairt to e90b66b (feat/code): native stop-sequence support in the LLM pipeline. Fixes qualcomm#1341 Signed-off-by: mansiverma897993 <vmansi756@gmail.com>
Contributor
Author
|
Thanks for the review RemiliaForever (@RemiliaForever)! Addressed - What changed:
Fixes #1341 either way - no more |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1341
FIM clients always send a
stoplist,/v1/completionsforwarded it to the plugin unchanged, and the qairt plugin rejects any generate call with stop sequences (-100016), so completions failed on every QAIRT model.For non-llama_cpp runtimes the stop list is no longer forwarded. A
stopScannerenforces it host-side: it matches the stop strings against the streamed tokens, holds back the longest tail that could still start a match so a stop spanning token boundaries never leaks, cancels generation through the token callback on a match (stop_reason: "user", mapped tofinish_reason: "stop"), and truncates the output at the match. Applied to both the streaming and blocking paths. llama_cpp keeps its native stop handling, so existing behavior is unchanged there.Test report
TestStopScanner(added incompletion_test.go), 7 cases: no match, match inside one token, match spanning tokens, earliest of multiple stops wins, stop at start, empty stop strings pass through, held tail flushed at end.Verified end-to-end without the SDK toolchain via a pure-Go stub harness (exact-signature
geniex_sdkstubs, real gin HTTP server, fake plugin that emits tokens throughOnTokenand honors cancellation), same pattern as #1317:go vetclean,gofmtclean.