Repository navigation
Sync with upstream llama.cpp master (2026-10-01) - #41
Merged
Merged
Conversation
This commit tweaks the Toaster element to include a close button. These toasts often cover other UI elements like the model selector, and this change avoids having to wait for them to disappear on their own (e.g. after a load failure).
The snapdragon CI builds packages only to feed the QDC device tests, so Hexagon NPU binaries never reached the releases page. Build both targets in release.yml and attach them as release assets.
* Fix Muse Glimmer tool-call first parser error * Add test to verify * Condense patterns * remove test for trigger patterns
* convert: add MiMo-V2.6 support Hoist the K3 mxfp4 conversion repack into base.py so it can be reused Remove decoder from mmproj convert * Update conversion/mimo.py * fix: use autoparser --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
* model : add DFlash layer-input taps for HunyuanVL
DFlash speculative decoding needs the target graph to expose the residual
stream entering each layer (res->t_layer_inp[il]) - the draft model reads
those tensors to build its cross-context. Qwen3 and the other DFlash-capable
targets register them, but the Hunyuan graphs do not, so serving a DFlash
draft against a HunyuanOCR target aborts during the first graph build:
GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null")
Register the tensor at the top of the layer loop, mirroring qwen3. The
layer input is the residual stream entering layer il, i.e. the output of
layer il-1, which is what the draft's target_layers metadata refers to
(the converter writes target_layer_ids+1). hunyuan-dense.cpp reuses this
graph, so it is covered as well; hunyuan-moe has a separate graph and is
untouched.
The vector is only read when a speculative implementation enables those
layer ids, so there is no behaviour change without a draft model.
Tested with tencent/HunyuanOCR 1.5 and its DFlash draft: image requests now
run, draft acceptance is ~0.5 and the OCR output is byte-identical to the
non-speculative run.
Co-authored-by: wendadawen <wendadawen@qq.com>
* convert : fix DFlash draft conversion against HunYuan targets
Converting a DFlash draft with a HunYuan target failed in two ways.
1. DFlashModel.set_vocab() reuses the target class' vocab handling by
calling it unbound with the draft instance, but HunYuanModel.set_vocab()
called self._fix_special_tokens(), a method that only exists on
HunYuanModel, so the conversion always aborted with
AttributeError: 'DFlashModel' object has no attribute '_fix_special_tokens'
Make the vocab helpers module-level functions taking the model
explicitly, so they do not depend on the instance being a HunYuanModel.
They have no other callers, so the two id lookups are folded into
_fix_special_tokens().
2. The delegated call runs with self.dir_model pointed at the target but
keeps the draft's self.hparams, so config lookups inside the target's
vocab code (the pad_token_id < 0 guard, eod_token_id) read the draft's
config instead of the target's. That aborts on targets with
pad_token_id = -1 (e.g. the HunyuanOCR v1.0 checkpoint) and otherwise
writes special token ids that disagree with the target.
Add _vocab_hparams(): it returns the target's config (with text_config
merged to the root, as TextModel does) when the model is a draft
converted with --target-model-dir, and the model's own hparams
otherwise, so a normal conversion is unaffected.
Tested: converting tencent/HunyuanOCR/dflash succeeds with both the 1.5 and
the v1.0 target; converting the base model without --target-model-dir
produces a byte-identical GGUF to before.
Co-authored-by: wendadawen <wendadawen@qq.com>
* convert : fix DFlash draft vocab against HunYuan targets
Switch hparams to the target config for the duration of the borrowed
set_vocab(), matching the existing dir_model swap, instead of teaching
HunYuanModel::set_vocab about draft models.
* convert : fix HunYuan special token ids for DFlash drafts
* convert : use load_hparams for HunYuan special token ids
* Add support for binding llama-server to multiple addresses Assisted-by: Codex * remove redundant thread handler * make it clear about overlapping addr * reject --port 0 with multiple tcp addr * improve arg handler * nits * fix test * nits 2 * nits * nits 2 --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* metal : gate mul_mm_id src1 rescale behind ggml_prec Assisted-by: Claude Fable 5.1 * ggml-webgpu: reject MUL_MAT_ID when src1 precision is F32 * cuda/vulkan: reject MUL_MAT_ID in supports_op when src1 prec is F32 fix `supports_op` to return false for failing backends when the specified src1 precision is f32 Assisted-by: Claude Fable 5.1 --------- Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
…G Plus/Xe2/Xe3) (ggml-org#24406) * vulkan : Intel FA kernel optimization for split k path * vulkan : Host code update for Intel split k FA kernel path selection, fix A770 Linux op test failures * vulkan : use symmetric coopMatMulAdd() in flash_attn_decode_phase_1 shader to resolve test op failre on A770 Linux with 26.2.3 mesa driver * vulkan : fix editorconfig issue in flash_attn_decode_phase_2.comp --------- Co-authored-by: Liu, Russell <russell.liu@intel.com>
…gml-org#22575) * server: support input_image in function_call_output (ggml-org#20663) * server: fix if statement spacing * server: avoid repeated type lookup
…9217) * server: route every model load through the queue A model loaded by the fast path has no queue entry, so tick() evicts it at its LOADED transition before its own request is proxied. Every load now joins the queue, whose entry protects the model until its waiters leave. * server: do not admit requests into a stopping model A request for a model that is being stopped still sees it LOADED and is proxied into the dying child. Such a request now joins the queue and is served by the next instance. The stopping mark is cleared under the same lock that sets UNLOADED, so no request can see a model that is neither stopping nor unloaded while its child is gone.
…in` (ggml-org#29056) * opencl: add A8 Q4_K non-MoE dp4a binary kernel * opencl: rename binary kernel selection helpers
Avoid returning references through lambdas that hold a local cast pointer, which triggers -Werror=dangling-reference in some CI compilers. Reuse the precomputed select_expr pointer directly. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
…7962) * HIP : use bit manipulation for __vcmpne4 * HIP : use bit manipulation for __vsub4
…HVX FA mask handling (ggml-org#29282)
…uction (ggml-org#29139) Since ggml-org#28732 our internal symbols are exported. A duplicate copy dlopened and dlclosed by ggml_backend_load_all() then interposes them, so its destructors destroy the live vk_instance and later device queries hit the GGML_ASSERT on vk_instance.device_indices. Hidden visibility exports only GGML_BACKEND_API, as before ggml-org#28732. Fixes ggml-org#29138 Assisted-by: henk:claude-fable-5
* resovle confict * support gedt_rows_back, update the ops.md
…sions (ggml-org#28931) * sycl : extend MMVQ GLU fusion to mixed quant types; add rms_norm+scale and ssm_conv+silu fusions * fixing spacing issue and macro converted to template function
* ggml-meta: resolve multi buffer views * add TODO to revisit if graph allocator gets refactored
* vulkan: optimize IQ4_XS matmul kernels Assisted-by: OpenAI Codex * vulkan: address IQ4_XS review nits - drop the dead LOAD_VEC_A != 8 branch in the IQ4_XS shmem load; iq4_xs is in lut_load_vec_a()'s "8" list, so that path is never generated - disable MMVQ for IQ4_XS on Intel (27.3% tg regression on A770) - remove a stray empty line in types.glsl Assisted-By: Claude Opus 5 <noreply@anthropic.com>
LLM-jp-4.1 uses the GPT-OSS format, but its tokenizer decodes a space after every special token and parallel tool calls are separated by <|end|>. The GPT-OSS handler rejects this output, so add a dedicated handler, selected by the chat_format=llm-jp-harmony-v1 declaration in the chat template. Assisted-by: Claude Fable 5.1
…org#29640) * BLAS : Document AOCL-BLAS build and label the device AOCL-BLAS * AOCL-Blas : Add an AOCL-BLAS Quick Start and drop the fixed version path * AOCL-BLAS doc : Note on ZenDNN
…-io (ggml-org#29749) Assisted-by: Claude Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
…29792) Pinning to >= 3.4.3 is required to enable DeviceTopK, which was affected by a race condition NVIDIA/cccl#10627. We will relax this for future CTK versions which will bundle CCCL > 3.4.X (CTK 13.5 will bundle CCCL 3.5.0 for example)
* llama: fix qwen4exp * qwen4exp: keep kq_mask input the same shape
* Qwen4Exp: add MTP * remove has_state member, check via ctx_bufs being non-empty * consistent naming + less verbose comments * cont : clean-up recurrent memory * cont : clean-up comments --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* CUDA: Handle compute type for NVFP4 on cublass path Signed-off-by: ynankani <ynankani@nvidia.com> * Use BF16 compute type for quantized models if HW allows Signed-off-by: ynankani <ynankani@nvidia.com> * Set acc prec to bf16 for nvfp4 as it needs atleast bf16 range Signed-off-by: ynankani <ynankani@nvidia.com> * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * preserve op_params for per-expert matmul Signed-off-by: ynankani <ynankani@nvidia.com> --------- Signed-off-by: ynankani <ynankani@nvidia.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* metal : release temporary private transfer buffers Assisted-by: OpenAI Codex * metal : fix order and formatting --------- Co-authored-by: Niklas Wenzel <dev@nikwen.de>
* cuda : route sm70 to the Turing MMVQ nwarps table Volta (sm_70) has no MMVQ parameter table of its own and falls through to GENERIC, which launches K-quant batch-1 decode (ncols_dst == 1) at nwarps=4. sm_70 shares TURING's tuning: the K-quant vec_dot prefers nwarps=2 there. Route sm_70 to the existing MMVQ_PARAMETERS_TURING table in both the device and the host table selector. Measured on one Tesla V100 32GB PCIe (PG500-216, driver 580.178.04, CUDA 12.0.140) with Qwen3.8-27B Q4_K_M, tg128, interleaved A/B in 6 ABBA blocks with paired per-block deltas: +1.091 t/s = +3.17 % (t = +49.0, all six per-block deltas positive); perplexity bit-identical (6.3697 +/- 0.04066 both builds, wiki.test.raw). The patched build's K-quant mul_mat_vec_q kernels launch at nwarps=2 (cubin EIATTR_MAX_THREADS) while Q4_0/Q8_0 stay at nwarps=4, and the same measurement on the September master base gave +3.84 % (t = 85). The tuning originates from the V100-focused fork anyei/llamacpp-v100 (MIT), commit b912d1b1e, which carries a dedicated MMVQ_PARAMETERS_VOLTA table; a cubin-level comparison confirmed that routing sm_70 to the existing TURING table is equivalent for the K-quant batch-1 path this change affects, so this is the minimal 2-line form. anyei/llamacpp-v100@b912d1b1e Original-patch-by: anyei <angelyoelroblesmercedes@gmail.com> * Update ggml/src/ggml-cuda/mmvq.cu --------- Co-authored-by: tkittich <tkittich@gmail.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* convert : write Gemma embedding scale for DFlash drafts A DFlash draft shares the target's token embeddings. Gemma scales them by sqrt(hidden_size) in the forward pass, and the draft config does not state that scale, so the converted draft read unscaled embeddings. Take the scale from the target config when the draft config has none. Assisted-by: Claude * convert : check with get_model_architecture for gemma models
…ia DMA (ggml-org#29685) * hexagon: shared strided DMA copy for CPY and CONCAT, any-dim CONCAT via DMA * hex-cpy: various fixes on top of the concat optimizations Removed CONCAT_DMA_MIN_ROW logic, it was broken with 64-bit DMA. While it's kinda silly to use DMA for tiny stuff if that tensor gets mapped to an extended buffer the only way to read it is DMA. Added missing dma_queue_flush() calls. Added additional guards for conditions we don't support. --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* tests : simplify function signature * llama : clamp kpool re-pool bound to existing pools The n_tokens/kpool + n_seqs_unq bound on n_new_g overshoots when a batch fills the whole cache: n_ctx tokens complete exactly n_ctx/kpool pools, so the +1 pads new_pool_idxs/new_pool_rep one entry past n_pool_real. Graph reserve only covers n_pool_real entries, so the first full-context decode builds bigger tensors than reserved and ggml-alloc demands a graph reallocation (abort under GGML_SCHED_DEBUG_REALLOC=1). Clamp the bound to n_pool_real: a ubatch can never mark more pools than the cache holds, and reserve's n_pool_max already covers that. Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD * cont : cap to n_pool_max
…9808) * skill: note about adding model-specific CLI arguments * add testing instructions
…++ (ggml-org#29816) See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101510 Close ggml-org#29759 Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…61001 # Conflicts: # common/json-schema-to-grammar.cpp
- upstream moved $ref resolution into common_chat_schema_from_json, so the nested $defs hoisting for MCP tool schemas now runs there - server_http_context dropped thread and listening_address in favor of join() and listening_addresses
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges upstream ggml-org/llama.cpp master (ec7630a, 449 commits) into the fork.
One conflict, in common/json-schema-to-grammar.cpp. Upstream replaced the $ref resolver with the new common_chat_schema builder (common/json-schema.cpp), so the file now matches upstream and the fork's nested $defs hoisting for MCP tool schemas moved into common_chat_schema_from_json.
Follow-up adaptation for upstream API changes:
threadandlistening_address. llama-agent-server now usesjoin()and logs each entry oflistening_addresses.Verified locally on macOS: