English | 中文
A high-performance, production-grade multi-task inference service written in Rust. VecBoost provides six inference capabilities — text embedding, reranking, decision answering, translation, language detection, and speech transcription — with multiple inference engines, GPU acceleration, and enterprise-grade features.
✨ Features • 🚀 Quick Start • 📚 Documentation • 💻 Examples • 🤝 Contributing
Text embedding, reranking, decision answering, translation, language detection, and speech transcription all load models through a single EngineFactory, sharing configuration, caching, and rate-limiting infrastructure.
| 🧠 Text Embedding Batch vectors · Matryoshka · OpenAI-compatible |
🔁 Reranking Bi-encoder document relevance |
🎲 Decision Answering choice / score / noul tasks |
| 🌐 Translation Marian language pairs · auto source detection |
🔤 Language Detection Confidence and reliability flags |
🎙️ Speech Transcription Qwen3-ASR with VAD segmentation |
📑 Table of Contents
| 🚀 High Performance Optimized Rust codebase with batching and concurrent request processing; jemalloc global allocator enabled by default on Linux |
🔧 Multiple EnginesONNX Runtime (default) and Candle (opt-in fallback) inference engines, switchable via EngineFactory |
| 🔁 Rerank Bi-encoder based document reranking over HTTP/gRPC/CLI |
🌍 Internationalization (i18n) ICU+Fluent bilingual (English/Chinese) error responses with Accept-Language per-request negotiation |
| 🌐 Translation & Language Detection Marian/opus-mt multi-pair translation ((src,tgt) routing + auto detection) and whatlang language detection, onnx feature |
🎮 Decision Answering Laya decision pipeline: choice / score / noul with temperature-calibrated probabilities |
| 🎮 GPU Acceleration Native NVIDIA CUDA and Apple Metal support; opt-in mkl/accelerate CPU backends |
🌐 Multi-Protocol Interface HTTP/REST, gRPC, MCP, and CLI generated from a single source by sdforge |
| 🧩 7-Library Ecosystem Modular ecosystem: trait-kit/confers/inklog/oxcache/limiteron/dbnexus/sdforge |
📊 Smart Caching High-performance caching via oxcache (LRU/LFU/FIFO + TTL) and three-stage semantic cache lookups |
| 🔐 Enterprise Security JWT authentication, CSRF protection, role-based access control, TOTP, account lockout, and audit logging |
⚡ Rate Limiting Token-bucket rate limiting via limiteron (global/IP/user/API-key dimensions) |
| 📈 Priority Queue Configurable request priorities, weighted fair scheduling, and time-window dynamic batching |
🧊 Matryoshka Support Dynamic dimensionality reduction (auto re-normalization after truncation) for smaller, faster embeddings (OpenAI-compatible) |
| 🔍 Observability Prometheus metrics, health checks, structured logging (inklog console + file rotation) |
📦 Cloud-Native Deployment Docker image (linux/amd64, single-arch by default build); Kubernetes deployment guidance (manifests bring-your-own) |
In addition to the core capabilities above, an OpenAI-compatible endpoint (POST /v1/embeddings, with encoding_format=base64 support), BF16 inference and SIMD vector similarity, GPU memory paging, read-only vecboost doctor diagnostics, Library SDK integration (library mode), and the config_full.toml / config_minimal.toml config presets are also available; the full feature-flag matrix is in 🔖 Feature Flags, and endpoint details are in 🔌 API Usage.
Prerequisites:
📋 Table
| Dependency | Version | Notes |
|---|---|---|
| Rust | 1.91+ | edition 2024 (rust-version in Cargo.toml is authoritative) |
| Cargo | 1.91+ | ships with Rust |
| CUDA Toolkit | 12.x | optional, NVIDIA GPU support (cuda feature) |
| Metal SDK | latest | optional, Apple Silicon GPU support (metal feature) |
| protobuf-compiler | latest | optional, required for gRPC E2E tests |
💡 Tip: run
rustc --versionto verify your Rust installation.
📄 Code · bash
# 1. Clone the repository
git clone https://github.com/Kirky-X/vecboost.git
cd vecboost
# 2. Default build (http + onnx features)
cargo build --release
# 3. Run (reads config/config.toml by default)
./target/release/vecboostMinimal build: cargo build --no-default-features --features http. For more build combinations (GPU / multi-protocol / MCP / the CI full-feature set), see the 📖 User Guide · Installation.
Configure and run:
📄 Code · bash
# Copy and customize the config (defaults to config/config.toml)
cp config/config.toml config/config_custom.toml
# Edit config/config_custom.toml
# Run with a custom config (--config; it is stripped before clap parsing, so position before or after the subcommand both work)
./target/release/vecboost --config config/config_custom.toml✅ Success: the service starts at
http://127.0.0.1:9002by default (secure default: loopback only).
🐳 Docker:
docker build -t vecboost:latest ., then run withconfig/andmodels/mounted; Docker Compose and Kubernetes deployment are covered in the 📖 User Guide · Docker deployment.
The following example is adapted from examples/http/embed_api.rs and generates embeddings over HTTP (full endpoints in the 📘 API Reference):
📄 Code · bash
curl -X POST http://localhost:9002/api/v1/embed \
-H "Content-Type: application/json" \
-d '{"text": "Hello, world!"}'Response:
📄 Code · json
{
"embedding": [0.123, 0.456, 0.789, ...],
"dimension": 1024,
"processing_time_ms": 15.5
}Request/response examples for the remaining endpoints — decision answering, translation, language detection, transcription — are in the 📘 API Reference; you can also use the CLI (cli feature) or the library SDK (library mode), with subcommands and tool lists in 💻 CLI tools.
VecBoost generates its four protocol interfaces (HTTP/gRPC/MCP/CLI) from single-source handlers via sdforge (one source per domain under src/api/); all endpoints, parameters, request/response examples, the gRPC method table, MCP tool lists, and error codes are documented in the 📘 API Reference, and the interactive OpenAPI docs in 📚 OpenAPI documentation (/swagger-ui/ and /redoc, requiring the swagger-ui feature).
The default config path is config/config.toml (--config <path> selects another; a missing explicit path fails fast with an error; the repo ships config_full.toml / config_minimal.toml presets). Environment variables with the VECBOOST_ prefix override the config file, and secrets (VECBOOST_JWT_SECRET / VECBOOST_ADMIN_PASSWORD) must be provided via env vars; config file changes are validated and logged, taking effect after restart.
Per-key options and defaults for every config section (server / model / embedding / rerank / monitoring / auth / rate_limit / audit / database / logging / pipeline.worker / semantic_cache / device / translation / transcribe), the full environment-variable table, and a complete example config are documented in the 📖 User Guide · Configuration; you can also read config/config.toml directly.
⚠️ Note: the[flow_control]and[cache]TOML sections are not parsed in the current version (legacy section names); rate limiting uses[rate_limit], caching uses[embedding]and[semantic_cache]. See the ❓ FAQ.
📋 Table
| Document | Description |
|---|---|
| 📖 User Guide | Complete tutorial from installation to advanced usage (incl. deployment) |
| 📘 API Reference | Full REST / gRPC / OpenAI-compatible interface documentation |
| 🏗️ Architecture | Design principles, module layout, and data flow |
| ⚡ Performance Guide | Benchmark data, tuning-switch registry, and experiment discipline |
| 🔒 Security | Security design, supported versions, and vulnerability reporting |
| ❓ FAQ | Frequently asked questions |
| 🧪 Test Scenarios | Test-stack responsibilities and the scenario matrix |
| 📋 Changelog | Release-by-release change log |
| 🤝 Contributing | How to contribute |
| 🧹 Maintenance Norms | Cross-change standing constraints: transcribe consumption surface, CLI contract, retired-switch discipline, deprecation-doc governance |
ℹ️ Note:
docs/benchmarks/(benchmark archive),docs/reviews/(review ledgers) anddocs-transcribe/are local-only directories (see.gitignore); they are not committed and do not exist in a clean clone. Key benchmark numbers are inlined in the ⚡ Performance Guide.
All examples live in examples/, an independent workspace-member crate vecboost-examples (12 categories, 31 runnable binaries) covering basic embedding, HTTP/CLI calls, engine switching, auth, caching, rate limiting, monitoring, audit, semantic cache, and Library SDK integration; the per-category listing is in examples/README.md.
VecBoost uses a modular ecosystem architecture: trait-kit wires modules through a typestate module registry (Kit<Unbuilt> → Kit<Ready>); src/registry/ has 20 production ModuleMeta implementations plus 3 test fixtures (several are feature-gated, e.g. under auth), sdforge generates the four protocol bindings from the single source src/api/embedding.rs, inference is abstracted behind EngineFactory to the Candle / ONNX engines, and requests flow through a priority queue and time-window batching into the inference pipeline.
The 7-library ecosystem (trait-kit / confers / inklog / oxcache / limiteron / dbnexus / sdforge) versions and responsibilities, the module dependency graph, data flow, and the cache / security / deployment architecture and extension points are documented in the 🏗️ Architecture document.
The test stack has six layers: inline unit tests in src/, integration tests in tests/integration/, specialized integration tests (doctor / gRPC E2E / model snapshot regression / quantization quality gate / SDK matrix), real-service scenario tests in tests/scenario/*.py (16 pytest suites), performance regression thresholds in tests/perf/, and 6 Criterion microbenchmark suites in benches/. Layer responsibilities, the exhaustive scenario matrix, and the run commands (identical to CI) are in 🧪 Test Scenarios; environment setup and quality gates are in 🤝 Contributing.
Benchmark data comes from the measured docs/benchmarks/ archive (criterion; that directory is local-only and git-ignored, so it does not exist in a clean clone): SIMD vector similarity is up to 3.06x faster than the scalar baseline (~341.7 ns for 1024-dim cosine), exact semantic-cache hits take ~10 ns, and candle single-text embedding is 4.0x faster with --features mkl. Full benchmark tables, measured engine-selection evidence, performance design highlights (time-window batching / in-batch dedup / SIMD / thread tuning / jemalloc), GGUF quantization and the tuning-switch registry are in the ⚡ Performance Guide.
VecBoost is secure by default: the factory config binds to loopback only, and non-loopback binding with auth.enabled=false refuses to start (the VECBOOST_ALLOW_INSECURE=1 escape hatch logs an ERROR); authentication and authorization build on garrison (JWT + CSRF + RBAC admin role + TOTP + account lockout), plus XFF trust inversion, file-path allowed roots, input length limits, AES-256-GCM config encryption, audit logging, and i18n bilingual error masking. Mechanism-level details are in the 🔒 Security document.
cargo audit, cargo deny check, CodeQL, Trivy/Checkov image scanning, gitleaks secret scanning, and pre-commit hooks run both in CI and locally; the full list and triage policy are in the 🔒 Security document · Supply chain & gates.
Please do not report security vulnerabilities through public issues. Use the GitHub Security Advisories private disclosure channel as the primary route (fall back to Kirky.X at kirky-x@outlook.com only if that channel is unavailable). Dependency advisories are gated by CI cargo-audit. For the full policy and supported versions, see SECURITY.md.
Roadmap items (core service / ecosystem integration / security & i18n / performance foundation / tuning switches / multi-replica shared state, etc.; ✅ done / 📋 planned) are maintained with version evolution in the 📋 Changelog · Roadmap.
For the detailed workflow and code standards, see the 🤝 Contributing guide. Found a problem? Open an issue; have an idea? Start a discussion; fork and open a PR.
The full release history is in the 📋 Changelog (following Keep a Changelog, semantic versioning).
0.3.0-rc.1 contains multiple breaking behavioral changes (secure defaults hardening, login convergence, XFF trust inversion, RBAC wiring, cache-key/tokenizer changes, etc.) — read before upgrading: the per-item "old behavior → new behavior → migration" table is in the 📋 Changelog · 0.3.0-rc.1. The multi-replica boundary (auth sessions default to process memory and can be externalized with session_backend = "database"; rate limiting and caches can now be externalized to Redis via distributed-redis) and hot-reload semantics (config changes take effect after restart) are covered in the ❓ FAQ.
This project is licensed under the Apache License 2.0 - see the LICENSE file for details. Copyright © 2025-2026 Kirky.X🌠.
The core-dependency list (onnxruntime / candle / tokenizers / hf-hub / the 7-library ecosystem / garrison / axum / tokio / utoipa / prometheus / criterion / jemallocator) is in 🏗️ Architecture · Core dependency acknowledgements.
Thanks to the Rust community, Hugging Face (model & tokenizer ecosystem), and all contributors.
|
Issues Report problems and bugs |
Discussions Ask questions and share ideas |
GitHub Browse the source |
© 2026 Kirky.X. All rights reserved.