feat(sdk): return PROMPT_TOO_LONG when the prompt itself overflows - #1349
Draft
RemiliaForever (RemiliaForever) wants to merge 1 commit into
Draft
feat(sdk): return PROMPT_TOO_LONG when the prompt itself overflows#1349RemiliaForever (RemiliaForever) wants to merge 1 commit into
RemiliaForever (RemiliaForever) wants to merge 1 commit into
Conversation
Overlong input previously always mapped to CONTEXT_LENGTH, so callers could not tell a prompt that never fit from a window exhausted mid-generation. Split them: - Prefill overflow (the prompt alone exceeds the context window, even after a context shift, or the model can't shift) now returns GENIEX_ERROR_LLM_GENERATION_PROMPT_TOO_LONG in both llama_cpp and qairt LLM/VLM plugins. - Mid-decode overflow keeps returning CONTEXT_LENGTH. The qairt plugins read the pipeline's new "prompt_too_long" stop_reason (bumped via the geniex-qairt submodule) and map it accordingly. Downstream: - Go binding exposes ErrLlmGenerationPromptTooLong. - Server: prompt-too-long is a 400 client error (no partial output); a window exhausted mid-generation is a normal truncated completion (finish_reason=length, HTTP 200) on both the blocking and streaming paths, for chat and completion. - CLI: a distinct hint and a reset-and-continue path for prompt-too-long. Sliding-window support is unchanged: the error is reported only when prefill actually fails, never via a pre-check. Signed-off-by: RemiliaForever <remilia@koumakan.cc>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Overlong input previously always mapped to
GENIEX_ERROR_LLM_TOKENIZATION_CONTEXT_LENGTH, so callers could not distinguish a prompt that never fit the context window from a window that filled up mid-generation. This PR splits the two:GENIEX_ERROR_LLM_GENERATION_PROMPT_TOO_LONGfrom both thellama_cppandqairtLLM/VLM plugins.CONTEXT_LENGTH.Downstream behavior:
llama_cppreports the phase-appropriate error from its sharedprocess()(prefill → prompt-too-long, decode → context-length);qairtreads the pipeline's new"prompt_too_long"stop_reason.ErrLlmGenerationPromptTooLong.400client error with no partial output; a window exhausted mid-generation is a normal truncated completion (finish_reason=length, HTTP 200) on both blocking and streaming paths, for chat and completion.Sliding-window support is unchanged: the error is reported only when prefill actually fails, never via a pre-check.
The
qairtplugin behavior depends on the companion submodule change (qualcomm/geniex-qairt-plugin#34); this PR bumps thethird-party/geniex-qairtpointer to include it.Test plan
llama_cppend-to-end (local): prompt-too-long, sliding-window intact, server 400 for prompt-too-long, server 200finish_reason=lengthfor mid-generation fill (blocking + streaming)qairtruntime (qcs, Qwen3-1.7B): CLI prompt-too-long vs context-length messages,--sliding-windowcontinues, server 400, server 200finish_reason=lengthqairtunit tests (qcs):ctest -L unit—ThrowsWhenPromptExceedsContext,SlidingWindowDisabledByDefaultStillThrows, newThrowsWhenGenerationExceedsContext, 8/8 passprocess_test.go: prompt-too-long resets and continuesDepends on qualcomm/geniex-qairt-plugin#34