Repository navigation
perf: SIMD fixes (ARM ODR, x86 AVX2 runtime dispatch), opt-in ONNX, Linux fixes - #187
Merged
Merged
Conversation
The header was deleted with the symbol_extractor_v1 ABI; builds only passed where an installed copy under the system include path shadowed it.
cross_rerank_simeon_outer_maxsim read SearchEngineConfig, but the daemon resolves the Simeon scoring mode from reranker policy (ConfigResolver), so the stat said 0 for every outer-MaxSim run and the scifact ablation could not tell which scorer ran. setCrossReranker() now takes the mode label the wiring resolved; the engine reports it as cross_rerank_scoring_mode and derives the outer-MaxSim stat from it.
static_cosine_ann_index.cpp instantiates sqlite-vec's header-only HNSW and inner_product_distance templates but was compiled in yams_vector without SQLITE_VEC_ENABLE_NEON, while the sqlite-vec library and backend TU were compiled with it. The inline kernels therefore had two bodies (an ODR violation): the ANN TU's copy was a serial scalar fadd chain, and the linker could keep either. Build it in yams_sqlite_vec_backend, which already carries the feature defines, and fail the build on AArch64 if the defines are missing.
./setup.sh Profiling configured Meson as buildtype=debug (-O0) and turned TSAN on by default, so Tracy zone timings described unoptimized, instrumented code. Use debugoptimized and leave TSAN off unless --tsan is passed.
Bump simeon to c846635: - fix(tokenizer): locale-independent character classes - perf(pq): batched/gathered inner-product scans, bit-identical - perf(projection): Achlioptas apply visits only non-zero sketch buckets (5.6x on short texts at 4096->384, bit-identical) The PQ search scan now calls inner_product_batch/_gather instead of one pimpl call per code. Scores are bit-identical, so ranking and tie-breaks are unchanged (1.6-2.0x lower ADC scan time in simeon's pq_query_bench).
looksLikePathQuery and nodeWithinScope in graph_context_service.cpp lost their last callers with the symbol ABI and only produced -Wunused-function.
YAMS_POST_SYMBOL_CONCURRENT and YAMS_SYMBOL_WEIGHT no longer have readers, so configuration_authority_policy flagged them as stale.
Bump simeon to 38a0b39 (active_simd_tier() checks CPUID; pq.cpp kernels follow it) and sqlite-vec-cpp to d4ce0cc (target-attribute AVX2+FMA float distance kernels behind a CPUID check). x86 builds now compile simeon's arch/avx2.cpp alone with -mavx2 -mfma and define SIMEON_HAS_AVX2; every other TU keeps the baseline ISA, so generic x86-64 binaries stay portable while AVX2 hosts use SIMD for encoding, PQ training, and sqlite-vec distances (3-4x per distance call at 384d on a Zen 2 host). Full unit suite on Linux x86 (clang, Debug): no new failures.
Default builds no longer pull onnxruntime: conan enable_onnx and the conan profiles default to False, meson enable-onnx defaults to disabled and skips onnxruntime discovery entirely, and setup.sh/setup.ps1 only enable ONNX (and the ONNX-backed Glint plugin) with --with-onnx / -WithOnnx. YAMS_DISABLE_ONNX=true still forces it off. This removes a build blocker on hosts where conan tries to build onnxruntime from source (e.g. ROCm/MIGraphX on Linux). Release, Docker, and Arch packaging pass the flag explicitly, so shipped artifacts are unchanged until ONNX is dropped.
~VerificationThreadPool set stop without holding queueMutex, so a worker that had evaluated the wait predicate but not yet blocked could miss notify_all() and join() hung forever (integrity_catch2 timed out under parallel load; gdb showed join() waiting on workers parked in wait()). Set the flag under the mutex. The new shutdown churn test hung in 35/40 parallel runs before the fix and passes 200/200 full-suite runs after.
- config_resolver: several [tuning] defaults derive from core count (e.g. onnx_max_concurrent is 7 on a 48-thread host), so a reload is now checked against the values observed before the override instead of != sentinel. - service_manager: write the P2P trust files with explicit permissions so a umask of 002 (Debian user-private groups) no longer trips the group-writable check before the symbolic-link check under test.
0fec31c: sqlite3_mprintf error paths used %zu, which SQLite parses as a string conversion and crashed on invalid vec0 dimensions or corrupt HNSW persistence; also corrects a RaBitQ test expectation.
Merged
1 of 3 tasks
trvon
added a commit
that referenced
this pull request
Sep 23, 2026
perf: carry #187 (SIMD fixes, x86 AVX2 dispatch, opt-in ONNX, Linux fixes) into experimental
4 of 6 tasks
trvon
added a commit
that referenced
this pull request
Sep 24, 2026
* docs(agents): document PR stacking, promotion, and submodule merge rules A stacked PR (#187) merged into its parent branch after the parent (#186) had already merged into experimental, so its commits missed experimental and the promotion PR until #189 carried them over. Record the merge order, a git cherry check for stranded commits, and the no-squash rule for submodule PRs. * fix(metadata): scope migration 40 deletes to symbol extractor output Migration 40 deleted every kg_edges row whose relation was calls, implements, includes, ... and every node of a symbol type such as class or interface. Graph ingest through MCP and IPC accepts any type and relation, so an upgrade from v0.19 irreversibly removed user-modelled data like a 'class' node or an 'implements' edge. Select the extractor's nodes by provenance and key shape instead (<kind>:<qualified>@<file>[@snap:<hash>], symbol_ref:<name>, or nodes its kg_doc_entities rows reference) and delete edges that are tagged with the extractor or touch those nodes. The new test fails before the change. * fix(release): build Arch packages with the ONNX plugin scripts/build-arch-pkg.sh still ran a plain ./setup.sh Release, which no longer enables ONNX, so both Arch release targets would have shipped without the ONNX embedding provider and Glint. Pass --with-onnx like the other release lanes and fail packaging if the plugin is missing from the stage, since the Arch lane skips the ONNX smoke test. * fix(search): keep topology cluster edges out of relational graph richness Topology rebuilds now materialize member_of/overlaps/medoid_of/ subcluster_of edges in the KG. They derive from embeddings, but counted toward kgRelationalEdgeCount, so building topology alone could push hasRichGraphTopology() over its threshold and reopen graph-weight gates that stay closed for corpora without real relations. Exclude them. * fix(topology): keep rebuilding when the latest snapshot is undecodable storeBatch loads the previous snapshot only to find documents that left the topology, but returned early when it could not decode it, so one corrupt payload blocked every future topology write. On a decode error, log and store the new snapshot without clearing dropped documents' keys; other load errors still propagate. * fix(graph): make isolated-node queries usable after the symbol graph removal Isolated mode lost its function/calls default in v0.20 and instead required a node type, but MCP only sets the node type through list_type, which also switched the request to list-by-type (checked first), so isolated:true either errored or silently listed nodes. Default isolated mode to document nodes (with the existing semantic_neighbor relation default: documents no other document lists as an embedding neighbour), keep MCP list_type from overriding isolated mode, and update the tool schema examples that still named calls edges. * fix(search): keep operator queries valid FTS5 instead of passing them through Queries containing FTS5 operator syntax (a quote, uppercase AND/OR/NOT, parentheses, column filters) were passed to MATCH almost verbatim, so 'foo "bar', '(foo OR bar', 'foo AND', 'NOT foo', 'IL-6 OR TNF' or 'ratio:2' failed with an fts5 syntax or no-such-column error instead of returning results; MCP clients and the CLI reach this path. Rebuild operator queries token by token: keep phrases, parentheses, AND/OR/NOT, well-formed NEAR(...) groups and filters on real FTS columns, quote any other non-bareword term, and fall back to the literal word query when the structure is invalid. The existing assertion that 'hello NEAR/2 world' passes through encoded a query FTS5 rejects. * fix(common): make the lenient UTF-8 sanitizers emit well-formed UTF-8 sanitizeUtf8 and ensureValidUtf8 only checked continuation-byte bits, so overlong encodings (E0 80 80), UTF-16 surrogates (ED A0 80) and values above U+10FFFF (F4 90 80 80) passed through. They exist to satisfy proto3 string fields (proto_serializer.cpp calls them for every string), which reject exactly those sequences, so a document containing them could make a peer fail to parse the message. Share one well-formedness check with sanitizeUtf8Strict; the lenient variant still replaces bytes with '?'. * fix(extraction): tighten UTF-8 detection and keep units after bad surrogates EncodingDetector::detectEncoding accepted C0/C1 and F5-F7 lead bytes, surrogates and overlong sequences as UTF-8, so such files skipped conversion and reached the index as invalid UTF-8. It now uses the shared well-formedness check. convertToUtf8 consumed the unit after an unpaired high surrogate before checking it, so UTF-16 'D800 0041' lost the 'A'. Only consume it when it is a low surrogate. * fix(extraction): keep HTML script/style and tag scans linear removeScriptAndStyle re-searched <script, <style and <!-- from the cursor on every iteration and, for an unclosed opener, scanned to the end for its closing tag once per occurrence; convertBlockTagsToNewlines rescanned to the end for '>' at every '<'. An ingested ~1 MB file of unclosed '<script' openers, empty comments or bare '<' took over a minute. Cache each next-occurrence position until the cursor passes it and remember when a closing token is absent from the rest of the document. The same three inputs now finish in well under a second. * fix(compression): check stored zstd sizes against the frame before allocating decompress() trusted a caller-provided expected size (from the stored block header, which has no checksum of its own) and allocated that many bytes, up to 16 GB, before decoding. A corrupt size field could therefore drive a huge allocation. When the frame records its content size, the expected size must match it. When it does not (streaming compressor output), decode incrementally so the buffer grows with the real output and fail once it passes the expected size. * fix(setup): find the Conan toolchain for Fuzzing builds ./setup.sh Fuzzing runs Conan with the Debug build type, which writes its toolchain under build/fuzzing/build-debug, but setup only looked under build-fuzzing and failed with 'Conan meson toolchain file not found'. This is the path the Forgejo fuzz-smoke lane uses. Look under build-debug as the Profiling mode already does. * fix(search): make grouped FTS5 operator queries explicit (found by fuzzing) fuzz_fts5_query found two inputs the operator-query normalizer still rejected at MATCH time: - FTS5 applies implicit AND only between phrases, so '(foo OR bar) baz', 'bar (foo)' and 'NEAR(a b) c' are syntax errors; insert an explicit AND next to a parenthesized or NEAR group. - 'NEAR/<x>(...)' was treated as a NEAR group; FTS5 accepts only 'NEAR('. * fix(build): link libFuzzer builds on Linux With -Dbuild-fuzzers=true every object carries ASan/UBSan and SanitizerCoverage, whose symbols only the fuzzer executable provides, but Meson's default b_lundef=true links shared libraries such as libyams_onnx_resource with --no-undefined, so ./setup.sh Fuzzing failed to link on Linux (the Forgejo fuzz-smoke path). Set -Db_lundef=false for Fuzzing builds and warn when a libFuzzer configuration keeps it on. * test(fuzz): harnesses for search queries, UTF-8, text extraction, compression - fuzz_fts5_query: sanitized MCP/CLI search text must execute as MATCH on an FTS5 table shaped like documents_fts, also after re-sanitizing. - fuzz_utf8: sanitizer output is well-formed and preserves valid input; a UTF-8 verdict implies well-formed bytes; UTF-16/Latin-1 conversions produce well-formed UTF-8. - fuzz_text_extractors: HTML/plain-text extraction, title and section detection on arbitrary bytes, with an output-size bound; run with -timeout to catch super-linear scans. - fuzz_compressed_block: stored blocks decode within the header's size and -rss_limit_mb; zstd/LZMA round-trip exactly. Seeds in generate_corpus.sh; the Forgejo fuzz-smoke lane builds and runs them and triggers on the parsers' sources. Five minutes per harness on a Linux x86 host after the fixes in this series: no findings. * fix(search): strip matched XML-style tag pairs in linear time stripXmlTags removed '<name>...</name>' pairs with a lazy backreference std::regex; every unmatched opener rescanned the rest of the query, so a 32 KB query of '<a>x' took 4.4 s (libstdc++, -O2) and time grew 4x per doubling. Search text comes from MCP clients and the CLI. Index closing tags once and find each opener's first closer by binary search, preserving the regex's left-to-right shortest-match semantics. The suspected stack overflow did not reproduce (libc++ or libstdc++, down to 256 KB thread stacks under ASan); the new test guards the time bound. * fix(repair): refuse embedding repair without a resolved data directory repairMissingEmbeddings built its database path as dataPath/'vectors.db' with create_if_missing, so an unresolved (empty) data directory created a stray vectors.db in the process working directory. The dispatcher passes an empty path when the service manager has not resolved its data dir. Return an error instead. Tests that exercised repair or placed plugins under the unresolved data directory now give the service manager a temp dir; the dispatcher suite no longer leaves vectors.db or dispatcher_external_* in the working directory. * refactor(compression): remove the never-built LZMA compressor lzma_compressor.cpp targets the 7-Zip LZMA SDK (LzmaEnc.h/Alloc.h), but the build only finds xz/liblzma, which does not ship those headers, so every build (release logs: 'Has header "LzmaEnc.h" ... NO') compiled it out. No build could read or write LZMA blocks, and its decoder never bounded output by the expected size. The policy still chose LZMA for cold/archival and large inactive data; compression then failed ('Compressor not available') and batch stores kept those blocks uncompressed. Select Zstandard, which already uses the archive level for old data. Remove the compressor, its test, the enable-lzma option and the build flags that passed it. The LZMA algorithm id stays in the header format, and compression.lzma_level remains accepted in config (now unused). * ci(mirror): expect the git.trevon.dev Forgejo SSH host ssh.git.trevon.dev no longer resolves (the experimental mirror run failed with 'Could not resolve hostname'), and main already switched the mirror remote to git.trevon.dev, where the mirror push succeeds. The policy test still required the old host and forbade the new one, failing Tests on every platform for main. Require git.trevon.dev, forbid the dead name, and update the mirroring doc. --------- Co-authored-by: trvon <me@trevon.dev> Co-authored-by: Trevon <git@trevon.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #186. Depends on trvon/simeon#1 and trvon/sqlite-vec-cpp#1 (submodule pointers reference those branches).
Problem
Follow-ups from the v0.20 review: an ODR violation left the topology ANN index on a scalar distance kernel, x86 builds had no SIMD at all, ONNX Runtime was on by default and blocks builds on some hosts, several Linux-only failures (including a real shutdown hang), and a misleading rerank debug stat.
Solution
static_cosine_ann_index.cppbuilt withoutSQLITE_VEC_ENABLE_NEONwhile the rest of sqlite-vec had it → two bodies for an inline kernel (ODR). Moved into the backend lib; AArch64 build now fails if the defines are missing. Disassembly: 0 → 8fmla, 404 → 44 scalarfadd.avx2.cppgets AVX flags, CPUID-gated), 5.6x short-text encoding, locale-independent tokenizer,%zucrash fix.setup.sh --with-onnx,setup.ps1 -WithOnnx); meson skips onnxruntime discovery when disabled. Release, Docker and Arch packaging pass the flag explicitly, so shipped artifacts still include ONNX.stopwithout the queue mutex → lost wakeup,join()hung forever (sporadicintegrity_catch2timeouts). New churn test hung 35/40 before, 200/200 pass after.cross_rerank_scoring_modereports the scorer the daemon actually wired (the MaxSim stat previously read engine config and always said 0)../setup.sh Profilingbuilds debugoptimized without TSAN so Tracy times optimized code.Release Note
--with-onnx.Breaking Changes
Tests
feat/graph-topology: quality identical; nfcorpus p50 -17.5%, p95 -22%; setup -13% / -36%.Documentation
--with-onnxAI usage
Sign-off