Skip to content

add native llama.cpp Web UI integration - #1311

Draft
GlennLin wants to merge 1 commit into
qualcomm:mainfrom
GlennLin:agent/add-native-llamacpp-webui
Draft

add native llama.cpp Web UI integration#1311
GlennLin wants to merge 1 commit into
qualcomm:mainfrom
GlennLin:agent/add-native-llamacpp-webui

Conversation

@GlennLin

@GlennLin GlennLin commented Aug 8, 2026

Copy link
Copy Markdown

Summary

  • serve an unmodified upstream llama.cpp Web UI build directly from geniex serve
  • add the compatible /props, static asset, SPA fallback, model-default, and explicit unsupported-feature behavior required by the UI
  • adapt chat requests and responses for llama.cpp aliases, OpenAI terminal stream semantics, Gemma 4 reasoning content, authoritative GenieX timing data, and model metadata
  • serialize SDK model acquisition, prompt formatting, and generation on one persistent OS thread so the OpenMP CPU backend retains stable repeat-request throughput
  • align llama_cpp CPU context defaults with llama-server while preserving accelerator-specific settings

Why

The upstream UI assumes llama-server has a single loaded model, omits model in normal requests, and only retains assistant statistics when the stream identifies the resolved model. GenieX previously rejected the empty model and omitted the terminal metadata the UI consumes.

Separately, Go could move consecutive cgo inference calls between OS threads. The CPU backend's OpenMP worker team remained associated with the first calling thread, causing later requests to fall from approximately 17 tokens/s to 2-3 tokens/s. A persistent locked inference thread keeps those calls on the same OS thread.

Impact

Users can point geniex serve --webui-dir at a genuine upstream llama.cpp UI distribution and run text chat through the embedded GenieX llama_cpp plugin without starting llama-server or a proxy. The UI receives aggregate prefill/decode statistics, Gemma reasoning content, and a resolved model ID. ttft_ms is also exposed in the timing payload for API consumers.

Unsupported llama-server administration features such as /slots and /tools return HTTP 403 rather than being simulated.

Validation

  • bazelisk test //cli/server/... --test_output=errors --spawn_strategy=local --strategy=GoTestGenTest=local --strategy=GoCompilePkg=local
    • 6 tests passed; 3 Windows targets skipped on the Linux host
  • bazelisk build //cli/cmd/geniex:geniex --spawn_strategy=local --strategy=GoCompilePkg=local
  • cmake --build sdk/build-native -j 2
  • end-to-end checks against the unmodified upstream UI assets:
    • root HTML and JS bundle were byte-identical to the source distribution
    • /props, /v1/models, streaming and blocking completions returned expected data
    • two sequential 64-token CPU streams decoded at 18.221 and 18.116 tokens/s
    • omitted-model browser requests returned HTTP 200
    • terminal streams contained finish_reason, model ID, timings, optional usage, and [DONE]

Limitations

The validated path is text chat with the IQ-9075 CPU package and a 4K context. The Web UI assets remain an external runtime input. GPU/NPU/hybrid, vision, audio, and 16K Web UI operation were not revalidated in this change. The unmodified upstream UI does not display its own TTFT badge or separate reasoning/final-answer decode rates.

Signed-off-by: GlennLin <12709970+GlennLin@users.noreply.github.com>
@GlennLin
GlennLin force-pushed the agent/add-native-llamacpp-webui branch from 7a04c61 to 98ffa6e Compare August 8, 2026 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant