Repository navigation
feat: sync zvec to v0.8.0-beta2 and add macOS x64 / Windows ARM64 support - #16
Merged
Merged
Conversation
Move the zvec submodule from v0.7.0 (8321c13) to the upstream main tip c133f20, which git describe resolves to v0.8.0-beta. Upstream #807 embeds the jieba dictionaries in the C-API library (LZ4) and removes the dict-directory entry points, so the Go-side discovery and registration layer is obsolete: - Drop ConfigData.SetJiebaDictDir/GetJiebaDictDir and the package-level SetDefaultJiebaDictDir/GetDefaultJiebaDictDir from the cgo and purego backends, along with the four purego symbol registrations. - Remove the auto-registration from Initialize and delete jieba_dict*.go. - Stop bundling dictionaries in package-libs.sh and in the release archives. - Raise the purego ABI requirement from 0.7 to 0.8 so the loader rejects older libraries instead of failing on missing symbols. - Bind the new ZVEC_DATA_TYPE_VECTOR_UINT8 as DataTypeVectorUint8. - Add TestFTSJiebaTokenizerEmbeddedDicts, an end-to-end regression test proving the jieba tokenizer works with no external dictionary files. BREAKING CHANGE: the jieba dict-directory APIs are removed. The `jieba` tokenizer now uses dictionaries embedded in the zvec C-API library; custom dictionaries are still selectable via the per-field `jieba_dict_dir` extra param or the ZVEC_JIEBA_DICT_DIR environment variable.
Upstream now publishes osx-amd64 and windows-arm64 SDKs (macos-15-intel and windows-11-arm runners), so cover the same platforms here. - Add cgo vendor directives for darwin/amd64 and windows/arm64. Both were previously unbuildable in the default vendor mode because no cgo_vendor_* file matched those GOOS/GOARCH pairs. - Teach download-libs about zvec-libs-darwin-x64.tar.gz and zvec-libs-windows-arm64.zip, and list them in the unsupported-platform hint. - release.yml: build both targets, pin CMAKE_OSX_ARCHITECTURES for the Intel job, verify the packaged dylib architecture with lipo, and install cmake/ninja from pip on Windows ARM64 because lukka/get-cmake ships no arm64 binaries. The ARM64 job builds with Ninja, which the existing packaging step already supports. - ci.yml: run the source-mode suite on both platforms so they get real test coverage, not just a compile check. - Update the platform lists in the READMEs and the copilot instructions.
Source (windows-arm64) failed at "Run Go unit tests (Windows)":
gcc_arm64.S:30: Error: no such instruction: `stp x29,x30,[sp,'
FAIL github.com/zvec-ai/zvec-go [build failed]
The windows-11-arm image adds an x86_64 MinGW-w64 toolchain (C:\mingw64\bin)
to PATH, and cgo's default CC=gcc resolves to it, so Go's runtime/cgo ARM64
assembly was handed to an x86 assembler.
Probing the runner confirmed the native ARM64 clang at
C:\Program Files\LLVM\bin (target aarch64-pc-windows-msvc) is not usable
either: Go unconditionally adds -mthreads for cgo on Windows, which clang
rejects for that target. The MSYS2 clangarm64 toolchain
(aarch64-w64-windows-gnu) builds, links against the MSVC-produced
zvec_c_api.lib, runs, and passes go test/go vet.
- Install mingw-w64-clang-aarch64-gcc from the pre-installed MSYS2 and point
CC/CXX at it in ci.yml (source and vendor jobs) and release.yml. CC/CXX are
exported only after the C-API build so CMake keeps selecting cl.exe, and no
PATH change is needed since the driver locates its own assembler and linker.
- Skip the race detector on windows/arm64: Go's RaceDetectorSupported only
reports windows/amd64, so -race fails with "-race is not supported on
windows/arm64".
- Document the Windows ARM64 toolchain requirement in both READMEs.
Moves the pinned zvec commit from c133f20 (v0.8.0-beta) to 8397071, which is exactly the upstream v0.8.0-beta2 tag. The four upstream commits in between are internal and leave the C API untouched — c_api.h, the C bindings and CMakeLists.txt are all unchanged — so no Go binding changes are needed: 8397071 fix(sqlengine): keep the exact FLOAT and DOUBLE bound in a forward filter 89ba922 ci: strip debug symbols from Android prebuilt SDK libraries f8cdeaf perf(flat): isolate query metadata from segment cache locking 1a57305 refactor: migrate index storage reads to MemoryBlock cmake/version.cmake parses v0.8.0-beta2 as 0.8.0, so the purego ABI gate (>= 0.8.0) is unaffected. Verified locally on darwin/arm64: rebuilt libzvec_c_api.dylib reports v0.8.0-beta2, and the source, vendor and purego suites all pass, including -race, go vet and six-platform purego cross-compilation.
The Quick Start in both READMEs and the generated release notes still told users to run `download-libs -version v0.6.0` and to `require` v0.6.0. Since release.yml renders that snippet into the v0.8.0 release body, bump it before tagging so the published notes point at the release they ship with. The `sync-zvec.sh v0.6.0` example is left alone: that argument is an upstream zvec tag, not a zvec-go version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Four commits.
1.
feat!: bump zvec submodule to v0.8.0-beta and adapt C-API bindingsSubmodule moves off
8321c13(v0.7.0) onto upstreammain, tracking the 0.8 C-API.Upstream #807 embeds the jieba dictionaries in the C-API library (LZ4) and removes four C-API symbols:
zvec_config_data_set/get_jieba_dict_dirandzvec_set/get_default_jieba_dict_dir. Adaptation:ConfigData.SetJiebaDictDir/GetJiebaDictDirand package-levelSetDefaultJiebaDictDir/GetDefaultJiebaDictDirfrom both the cgo and purego backends, plus their purego symbol registrations.Initializeand deletedjieba_dict*.go.scripts/package-libs.shand in the release archives.ZVEC_DATA_TYPE_VECTOR_UINT8asDataTypeVectorUint8.TestFTSJiebaTokenizerEmbeddedDicts, an end-to-end regression test proving jieba FTS works with no external dictionary files.Breaking change for SDK users of the four removed functions. Dictionary selection is still possible via the per-field
jieba_dict_dirextra param orZVEC_JIEBA_DICT_DIR(priority: field > env > embedded defaults).Symbol-level audit of
c_api.hv0.7.0 → v0.8.0-beta2: upstream removed exactly those 4 functions and added no new ones. The Go bindings drop exactly those same 4 and reference no symbol that is absent from the header. The only additions are macros —ZVEC_DATA_TYPE_VECTOR_UINT8(bound) andZVEC_IO_BACKEND_TYPE_WINDOWS_OVERLAPPED(the SDK does not expose IO-backend constants).2.
feat(ci): build and test macOS x64 and Windows ARM64Upstream now publishes
zvec-sdk-osx-amd64andzvec-sdk-windows-arm64; this adds the matching SDK support.cgo_vendor_darwin_amd64.go/cgo_vendor_windows_arm64.go. Both platforms were previously unbuildable in the default vendor mode — nocgo_vendor_*file matched those GOOS/GOARCH pairs.download-libslearnedzvec-libs-darwin-x64.tar.gzandzvec-libs-windows-arm64.zip.release.yml:macos-15-intel+windows-11-armjobs,CMAKE_OSX_ARCHITECTURESpinning, alipo -archsguard on the packaged dylib, and pip-installed cmake/ninja for Windows ARM64 (lukka/get-cmakehas no arm64 binaries;cmake==3.30.0andninja==1.11.1both ship nativewin_arm64wheels). The ARM64 job builds with Ninja, which the existing packaging step already handles.ci.yml: source-mode suite now runs on both new platforms, so they get real test coverage rather than a compile-only check.3.
fix(ci): build and test Windows ARM64 with a native ARM64 cgo toolchainThe first CI run failed
Source (windows-arm64)at the Go test step:The
windows-11-armimage puts an x86_64 MinGW-w64 toolchain (C:\mingw64\bin) onPATH, and cgo's defaultCC=gccresolves to it — so Go'sruntime/cgo/gcc_arm64.Swas handed to an x86 assembler.A short-lived probe workflow on the runner evaluated every candidate:
gcc(image default, x86_64 MinGW-w64 14.2.0)clang(native ARM64,aarch64-pc-windows-msvc)error: unsupported option '-mthreads' for target— Go adds-mthreadsunconditionally for cgo on Windowsclang-clunknown argument ignored in clang-cl: '-dM'clangarm64gcc (aarch64-w64-windows-gnu)zvec_c_api.lib, runs, passesgo test+go vetFix:
mingw-w64-clang-aarch64-gccfrom the image's pre-installed MSYS2 (~35 s) and pointCC/CXXat it — inci.yml(source and vendor jobs) and inrelease.yml, whosego buildverification would otherwise fail identically.CC/CXXare exported only after the C-API build, because CMake honours$CC/$CXXand would otherwise stop selectingcl.exe. NoPATHchange is needed: the driver finds its own assembler and linker, and the produced binaries depend only onzvec_c_api.dll+ the system UCRT (MinGW runtime is linked statically).windows/arm64. This is a second, independent blocker: Go'splatform.RaceDetectorSupportedreports onlywindows/amd64, so the step fails with-race is not supported on windows/arm64.4.
chore: bump zvec submodule to v0.8.0-beta2Re-pins
c133f20→8397071, which is exactly the upstreamv0.8.0-beta2tag. The four intervening commits are internal and leave the C API untouched (c_api.h, the C bindings andCMakeLists.txtare all unchanged), so no binding changes were needed:Upstream validated
8397071withBuild Prebuilt Librariesgreen on every platform includingwindows-arm64andosx-amd64.cmake/version.cmakeparsesv0.8.0-beta2as0.8.0, so the purego ABI gate is unaffected.Verification
Locally on darwin/arm64 against a library rebuilt from
8397071(reportsv0.8.0-beta2, 0.8.0):-racepasses.go vetclean in all three modes plus./cmd/...;gofmtclean.actionlintreports no new warnings vsmainfor either workflow.CI (previous run, identical workflows, submodule at
v0.8.0-beta): 12/12 jobs green, with 859 passing tests and 0 failures on each ofSource (macos-x64),Source (windows-x64)andSource (windows-arm64)— covering both cgo and purego integration paths.TestFTSJiebaTokenizerEmbeddedDictspasses on both new platforms. CMake resolveszvec version: v0.8.0-beta (0.8.0)in CI, confirming submodule tags are available to the version logic.Notes
Vendor (*)jobs currently no-op:lib/is gitignored, so a fresh checkout has no pre-built libraries and every step skips. Real coverage comes from theSource (*)jobs. Wiring the vendor jobs to release artifacts would be a separate change.cmd/download-libsresolves/releases/latest, which excludes prereleases. Until a non-prerelease zvec-go tag is cut from this branch,go generatestill fetches the v0.7.0 archives — cgo vendor mode keeps working, but purego mode will (correctly) reject a 0.7 library.