Skip to content

feat: sync zvec to v0.8.0-beta2 and add macOS x64 / Windows ARM64 support - #16

Merged
chinaux merged 5 commits into
mainfrom
feat/zvec-0.8-platforms
Oct 21, 2026
Merged

chinaux merged 5 commits into
mainfrom
feat/zvec-0.8-platforms

Conversation

@chinaux

@chinaux chinaux commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

What

Four commits.

1. feat!: bump zvec submodule to v0.8.0-beta and adapt C-API bindings

Submodule moves off 8321c13 (v0.7.0) onto upstream main, tracking the 0.8 C-API.

Upstream #807 embeds the jieba dictionaries in the C-API library (LZ4) and removes four C-API symbols: zvec_config_data_set/get_jieba_dict_dir and zvec_set/get_default_jieba_dict_dir. Adaptation:

  • Removed ConfigData.SetJiebaDictDir/GetJiebaDictDir and package-level SetDefaultJiebaDictDir/GetDefaultJiebaDictDir from both the cgo and purego backends, plus their purego symbol registrations.
  • Removed the dict auto-registration from Initialize and deleted jieba_dict*.go.
  • Stopped bundling dictionaries in scripts/package-libs.sh and in the release archives.
  • Raised the purego ABI requirement 0.7 → 0.8 so the loader rejects an older library with a clear message instead of failing on missing symbols.
  • Bound the new ZVEC_DATA_TYPE_VECTOR_UINT8 as DataTypeVectorUint8.
  • Added TestFTSJiebaTokenizerEmbeddedDicts, an end-to-end regression test proving jieba FTS works with no external dictionary files.

Breaking change for SDK users of the four removed functions. Dictionary selection is still possible via the per-field jieba_dict_dir extra param or ZVEC_JIEBA_DICT_DIR (priority: field > env > embedded defaults).

Symbol-level audit of c_api.h v0.7.0 → v0.8.0-beta2: upstream removed exactly those 4 functions and added no new ones. The Go bindings drop exactly those same 4 and reference no symbol that is absent from the header. The only additions are macros — ZVEC_DATA_TYPE_VECTOR_UINT8 (bound) and ZVEC_IO_BACKEND_TYPE_WINDOWS_OVERLAPPED (the SDK does not expose IO-backend constants).

2. feat(ci): build and test macOS x64 and Windows ARM64

Upstream now publishes zvec-sdk-osx-amd64 and zvec-sdk-windows-arm64; this adds the matching SDK support.

  • New cgo_vendor_darwin_amd64.go / cgo_vendor_windows_arm64.go. Both platforms were previously unbuildable in the default vendor mode — no cgo_vendor_* file matched those GOOS/GOARCH pairs.
  • download-libs learned zvec-libs-darwin-x64.tar.gz and zvec-libs-windows-arm64.zip.
  • release.yml: macos-15-intel + windows-11-arm jobs, CMAKE_OSX_ARCHITECTURES pinning, a lipo -archs guard on the packaged dylib, and pip-installed cmake/ninja for Windows ARM64 (lukka/get-cmake has no arm64 binaries; cmake==3.30.0 and ninja==1.11.1 both ship native win_arm64 wheels). The ARM64 job builds with Ninja, which the existing packaging step already handles.
  • ci.yml: source-mode suite now runs on both new platforms, so they get real test coverage rather than a compile-only check.

3. fix(ci): build and test Windows ARM64 with a native ARM64 cgo toolchain

The first CI run failed Source (windows-arm64) at the Go test step:

gcc_arm64.S:30: Error: no such instruction: `stp x29,x30,[sp,'
FAIL github.com/zvec-ai/zvec-go [build failed]

The windows-11-arm image puts an x86_64 MinGW-w64 toolchain (C:\mingw64\bin) on PATH, and cgo's default CC=gcc resolves to it — so Go's runtime/cgo/gcc_arm64.S was handed to an x86 assembler.

A short-lived probe workflow on the runner evaluated every candidate:

Compiler Result
gcc (image default, x86_64 MinGW-w64 14.2.0) fails as above
clang (native ARM64, aarch64-pc-windows-msvc) error: unsupported option '-mthreads' for target — Go adds -mthreads unconditionally for cgo on Windows
clang-cl unknown argument ignored in clang-cl: '-dM'
MSYS2 clangarm64 gcc (aarch64-w64-windows-gnu) builds, links the MSVC-produced zvec_c_api.lib, runs, passes go test + go vet

Fix:

  • Install mingw-w64-clang-aarch64-gcc from the image's pre-installed MSYS2 (~35 s) and point CC/CXX at it — in ci.yml (source and vendor jobs) and in release.yml, whose go build verification would otherwise fail identically.
  • CC/CXX are exported only after the C-API build, because CMake honours $CC/$CXX and would otherwise stop selecting cl.exe. No PATH change is needed: the driver finds its own assembler and linker, and the produced binaries depend only on zvec_c_api.dll + the system UCRT (MinGW runtime is linked statically).
  • Skip the race detector on windows/arm64. This is a second, independent blocker: Go's platform.RaceDetectorSupported reports only windows/amd64, so the step fails with -race is not supported on windows/arm64.
  • Documented the Windows ARM64 toolchain requirement in both READMEs.

4. chore: bump zvec submodule to v0.8.0-beta2

Re-pins c133f20 → 8397071, which is exactly the upstream v0.8.0-beta2 tag. The four intervening commits are internal and leave the C API untouched (c_api.h, the C bindings and CMakeLists.txt are all unchanged), so no binding changes were needed:

8397071 fix(sqlengine): keep the exact FLOAT and DOUBLE bound in a forward filter
89ba922 ci: strip debug symbols from Android prebuilt SDK libraries
f8cdeaf perf(flat): isolate query metadata from segment cache locking
1a57305 refactor: migrate index storage reads to MemoryBlock

Upstream validated 8397071 with Build Prebuilt Libraries green on every platform including windows-arm64 and osx-amd64. cmake/version.cmake parses v0.8.0-beta2 as 0.8.0, so the purego ABI gate is unaffected.

Verification

Locally on darwin/arm64 against a library rebuilt from 8397071 (reports v0.8.0-beta2, 0.8.0):

  • Full integration suite passes in source, vendor and purego modes; -race passes.
  • go vet clean in all three modes plus ./cmd/...; gofmt clean.
  • purego cross-compiles for darwin/amd64, darwin/arm64, linux/amd64, linux/arm64, windows/amd64, windows/arm64.
  • actionlint reports no new warnings vs main for either workflow.

CI (previous run, identical workflows, submodule at v0.8.0-beta): 12/12 jobs green, with 859 passing tests and 0 failures on each of Source (macos-x64), Source (windows-x64) and Source (windows-arm64) — covering both cgo and purego integration paths. TestFTSJiebaTokenizerEmbeddedDicts passes on both new platforms. CMake resolves zvec version: v0.8.0-beta (0.8.0) in CI, confirming submodule tags are available to the version logic.

Notes

  • The 6 Vendor (*) jobs currently no-op: lib/ is gitignored, so a fresh checkout has no pre-built libraries and every step skips. Real coverage comes from the Source (*) jobs. Wiring the vendor jobs to release artifacts would be a separate change.
  • cmd/download-libs resolves /releases/latest, which excludes prereleases. Until a non-prerelease zvec-go tag is cut from this branch, go generate still fetches the v0.7.0 archives — cgo vendor mode keeps working, but purego mode will (correctly) reject a 0.7 library.

Move the zvec submodule from v0.7.0 (8321c13) to the upstream main tip
c133f20, which git describe resolves to v0.8.0-beta.

Upstream #807 embeds the jieba dictionaries in the C-API library (LZ4) and
removes the dict-directory entry points, so the Go-side discovery and
registration layer is obsolete:

- Drop ConfigData.SetJiebaDictDir/GetJiebaDictDir and the package-level
  SetDefaultJiebaDictDir/GetDefaultJiebaDictDir from the cgo and purego
  backends, along with the four purego symbol registrations.
- Remove the auto-registration from Initialize and delete jieba_dict*.go.
- Stop bundling dictionaries in package-libs.sh and in the release archives.
- Raise the purego ABI requirement from 0.7 to 0.8 so the loader rejects
  older libraries instead of failing on missing symbols.
- Bind the new ZVEC_DATA_TYPE_VECTOR_UINT8 as DataTypeVectorUint8.
- Add TestFTSJiebaTokenizerEmbeddedDicts, an end-to-end regression test
  proving the jieba tokenizer works with no external dictionary files.

BREAKING CHANGE: the jieba dict-directory APIs are removed. The `jieba`
tokenizer now uses dictionaries embedded in the zvec C-API library; custom
dictionaries are still selectable via the per-field `jieba_dict_dir` extra
param or the ZVEC_JIEBA_DICT_DIR environment variable.
Upstream now publishes osx-amd64 and windows-arm64 SDKs (macos-15-intel and
windows-11-arm runners), so cover the same platforms here.

- Add cgo vendor directives for darwin/amd64 and windows/arm64. Both were
  previously unbuildable in the default vendor mode because no cgo_vendor_*
  file matched those GOOS/GOARCH pairs.
- Teach download-libs about zvec-libs-darwin-x64.tar.gz and
  zvec-libs-windows-arm64.zip, and list them in the unsupported-platform hint.
- release.yml: build both targets, pin CMAKE_OSX_ARCHITECTURES for the Intel
  job, verify the packaged dylib architecture with lipo, and install
  cmake/ninja from pip on Windows ARM64 because lukka/get-cmake ships no
  arm64 binaries. The ARM64 job builds with Ninja, which the existing
  packaging step already supports.
- ci.yml: run the source-mode suite on both platforms so they get real test
  coverage, not just a compile check.
- Update the platform lists in the READMEs and the copilot instructions.
Source (windows-arm64) failed at "Run Go unit tests (Windows)":

    gcc_arm64.S:30: Error: no such instruction: `stp x29,x30,[sp,'
    FAIL github.com/zvec-ai/zvec-go [build failed]

The windows-11-arm image adds an x86_64 MinGW-w64 toolchain (C:\mingw64\bin)
to PATH, and cgo's default CC=gcc resolves to it, so Go's runtime/cgo ARM64
assembly was handed to an x86 assembler.

Probing the runner confirmed the native ARM64 clang at
C:\Program Files\LLVM\bin (target aarch64-pc-windows-msvc) is not usable
either: Go unconditionally adds -mthreads for cgo on Windows, which clang
rejects for that target. The MSYS2 clangarm64 toolchain
(aarch64-w64-windows-gnu) builds, links against the MSVC-produced
zvec_c_api.lib, runs, and passes go test/go vet.

- Install mingw-w64-clang-aarch64-gcc from the pre-installed MSYS2 and point
  CC/CXX at it in ci.yml (source and vendor jobs) and release.yml. CC/CXX are
  exported only after the C-API build so CMake keeps selecting cl.exe, and no
  PATH change is needed since the driver locates its own assembler and linker.
- Skip the race detector on windows/arm64: Go's RaceDetectorSupported only
  reports windows/amd64, so -race fails with "-race is not supported on
  windows/arm64".
- Document the Windows ARM64 toolchain requirement in both READMEs.
Moves the pinned zvec commit from c133f20 (v0.8.0-beta) to 8397071, which is
exactly the upstream v0.8.0-beta2 tag.

The four upstream commits in between are internal and leave the C API untouched
— c_api.h, the C bindings and CMakeLists.txt are all unchanged — so no Go
binding changes are needed:

  8397071 fix(sqlengine): keep the exact FLOAT and DOUBLE bound in a forward filter
  89ba922 ci: strip debug symbols from Android prebuilt SDK libraries
  f8cdeaf perf(flat): isolate query metadata from segment cache locking
  1a57305 refactor: migrate index storage reads to MemoryBlock

cmake/version.cmake parses v0.8.0-beta2 as 0.8.0, so the purego ABI gate
(>= 0.8.0) is unaffected. Verified locally on darwin/arm64: rebuilt
libzvec_c_api.dylib reports v0.8.0-beta2, and the source, vendor and purego
suites all pass, including -race, go vet and six-platform purego
cross-compilation.
@chinaux chinaux changed the title feat: sync zvec to v0.8.0-beta and add macOS x64 / Windows ARM64 support feat: sync zvec to v0.8.0-beta2 and add macOS x64 / Windows ARM64 support Oct 20, 2026
The Quick Start in both READMEs and the generated release notes still told
users to run `download-libs -version v0.6.0` and to `require` v0.6.0. Since
release.yml renders that snippet into the v0.8.0 release body, bump it before
tagging so the published notes point at the release they ship with.

The `sync-zvec.sh v0.6.0` example is left alone: that argument is an upstream
zvec tag, not a zvec-go version.
@chinaux
chinaux merged commit b4066ff into main Oct 21, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant