feat(llm): expose Lemonade's global_timeout so long-document prefill stops hitting a fixed ceiling

[kovtcharov]

Lemonade v11.9.0 replaced its fixed 120-second prefill timeout with a 600-second default that is configurable via global_timeout. GAIA never had a lever for this, and long-document Q&A is exactly the workload that hits it.

Why this matters here. The canonical GAIA failure of this shape is #1030 — document Q&A on long PDFs. A large PDF turned into a single large prefill is precisely what a fixed 120s ceiling truncates, and until now the only response was to make the prompt smaller. A configurable ceiling turns "long PDF silently fails" into a policy decision GAIA can make per profile.

Worth scoping:

• Decide where the value belongs. It likely pairs with the existing per-device context pins (GPU_CTX_SIZE / NPU_CTX_SIZE in src/gaia/llm/lemonade_client.py), since a machine running a 64K context needs a longer prefill budget than one capped at 32K — the NPU and GPU profiles should probably not share one number.
• Surface a real error when the budget is exceeded rather than a truncated answer, per the fail-loudly rule.

Should be validated against the rag_quality eval category on a genuinely long document, not a synthetic prompt.

Proposed from the Lemonade v11.9.0/v2026.39.1 release scan; maintainer triage decides scope. cc @kovtcharov-amd

[github-actions]

Makes sense, and it fits how GAIA already handles per-machine LLM settings — the model-load call is the place where GAIA pins other per-profile values, so a prefill budget belongs right next to them, split per device profile as you suggest.

One thing that has to land first: GAIA still installs and ships Lemonade 11.8.1, and supports servers as old as 10.2.0. global_timeout doesn't exist on either, so sending it unconditionally would break users on an older server — the version bump comes first, and the knob needs to be skipped on servers that predate it.

Second, GAIA has its own request ceiling of 15 minutes. Anything above that gets cut off by GAIA before the server's own budget ever applies, so the two numbers need to be picked together rather than one at a time.

On fail-loudly: the timeout does surface as a server error, so the work is making sure it's reported as "this document exceeded the prefill budget, here's the knob" rather than a generic request failure.

Agreed on validating against rag_quality with a real long PDF — a synthetic prompt won't reproduce the prefill shape that causes this.

🔍 Technical details

Where the value goes. LemonadeClient._load_model_impl already builds the /load payload from optional per-load knobs (src/gaia/llm/lemonade_client.py:3944-3950 — llamacpp_args, ctx_size, save_options). global_timeout slots in the same way, resolved per device alongside profile_ctx_size() (lemonade_client.py:315-322), so GPU_CTX_SIZE=65536 and NPU_CTX_SIZE=32768 can carry different budgets.

Version gate. LEMONADE_VERSION = "11.8.1" and LEMONADE_MIN_VERSION = "10.2.0" (src/gaia/version.py:12-17). Sending an unknown load parameter to an older server is the #1655 failure mode — a 400 that only appears on a machine that isn't already warm. Whatever guard is used, it needs a test that asserts the outgoing payload shape per server version, not just that the client was called.

Client-side ceiling. DEFAULT_REQUEST_TIMEOUT = 900 (lemonade_client.py:415) is the requests timeout on /chat/completions. It's above Lemonade's new 600s default, so today the server budget wins — but a global_timeout >900 would be pre-empted by a client-side read timeout, which surfaces as a connection error rather than an actionable one. Either cap the configured value against it or raise both together.

Error classification. The timeout arrives as a non-200 from /chat/completions; for the fail-loudly requirement it needs a LemonadeError subclass carrying what failed, the current budget, and how to raise it — not the generic request-failure path.