Skip to content

feat(sdk): return PROMPT_TOO_LONG when the prompt itself overflows - #1349

Draft
RemiliaForever (RemiliaForever) wants to merge 1 commit into
mainfrom
feat/prompt-too-long-error
Draft

feat(sdk): return PROMPT_TOO_LONG when the prompt itself overflows#1349
RemiliaForever (RemiliaForever) wants to merge 1 commit into
mainfrom
feat/prompt-too-long-error

Conversation

@RemiliaForever

Copy link
Copy Markdown
Contributor

Summary

Overlong input previously always mapped to GENIEX_ERROR_LLM_TOKENIZATION_CONTEXT_LENGTH, so callers could not distinguish a prompt that never fit the context window from a window that filled up mid-generation. This PR splits the two:

  • Prompt too long — the prompt alone exceeds the context window (even after a context shift, or the model can't shift). Now returns the new GENIEX_ERROR_LLM_GENERATION_PROMPT_TOO_LONG from both the llama_cpp and qairt LLM/VLM plugins.
  • Context length exceeded — the in-flight generation fills the window mid-decode. Keeps returning CONTEXT_LENGTH.

Downstream behavior:

  • SDK pluginsllama_cpp reports the phase-appropriate error from its shared process() (prefill → prompt-too-long, decode → context-length); qairt reads the pipeline's new "prompt_too_long" stop_reason.
  • Go binding — exposes ErrLlmGenerationPromptTooLong.
  • Server — prompt-too-long is a 400 client error with no partial output; a window exhausted mid-generation is a normal truncated completion (finish_reason=length, HTTP 200) on both blocking and streaming paths, for chat and completion.
  • CLI — a distinct hint and a reset-and-continue path for prompt-too-long.

Sliding-window support is unchanged: the error is reported only when prefill actually fails, never via a pre-check.

The qairt plugin behavior depends on the companion submodule change (qualcomm/geniex-qairt-plugin#34); this PR bumps the third-party/geniex-qairt pointer to include it.

Test plan

  • llama_cpp end-to-end (local): prompt-too-long, sliding-window intact, server 400 for prompt-too-long, server 200 finish_reason=length for mid-generation fill (blocking + streaming)
  • qairt runtime (qcs, Qwen3-1.7B): CLI prompt-too-long vs context-length messages, --sliding-window continues, server 400, server 200 finish_reason=length
  • qairt unit tests (qcs): ctest -L unitThrowsWhenPromptExceedsContext, SlidingWindowDisabledByDefaultStillThrows, new ThrowsWhenGenerationExceedsContext, 8/8 pass
  • CLI process_test.go: prompt-too-long resets and continues
  • CI: lint → build-sdk → CLI/Python/Android → docker
  • Depends on feat(core): distinguish prompt-too-long from mid-generation overflow geniex-qairt-plugin#34 merging first (submodule pointer)

Depends on qualcomm/geniex-qairt-plugin#34

Overlong input previously always mapped to CONTEXT_LENGTH, so callers
could not tell a prompt that never fit from a window exhausted
mid-generation. Split them:

- Prefill overflow (the prompt alone exceeds the context window, even
  after a context shift, or the model can't shift) now returns
  GENIEX_ERROR_LLM_GENERATION_PROMPT_TOO_LONG in both llama_cpp and qairt
  LLM/VLM plugins.
- Mid-decode overflow keeps returning CONTEXT_LENGTH.

The qairt plugins read the pipeline's new "prompt_too_long" stop_reason
(bumped via the geniex-qairt submodule) and map it accordingly.

Downstream:
- Go binding exposes ErrLlmGenerationPromptTooLong.
- Server: prompt-too-long is a 400 client error (no partial output);
  a window exhausted mid-generation is a normal truncated completion
  (finish_reason=length, HTTP 200) on both the blocking and streaming
  paths, for chat and completion.
- CLI: a distinct hint and a reset-and-continue path for prompt-too-long.

Sliding-window support is unchanged: the error is reported only when
prefill actually fails, never via a pre-check.

Signed-off-by: RemiliaForever <remilia@koumakan.cc>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant