Skip to content
Kirky-XPublic

About

Production embedding & decision inference service in Rust — Candle/ONNX dual engine, CUDA/Metal acceleration, REST/gRPC/MCP/CLI generated via sdforge, rerank, Matryoshka dimensions, OpenAI-compatible API, and semantic caching

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

720 Commits

Folders and files

Repository files navigation

VecBoost Logo

Rust 2024 Rustc 1.91+ release CI security coverage License: Apache-2.0

English | 中文

A high-performance, production-grade multi-task inference service written in Rust. VecBoost provides six inference capabilities — text embedding, reranking, decision answering, translation, language detection, and speech transcription — with multiple inference engines, GPU acceleration, and enterprise-grade features.

✨ Features • 🚀 Quick Start • 📚 Documentation • 💻 Examples • 🤝 Contributing


🎯 Six Inference Capabilities, One Service

Text embedding, reranking, decision answering, translation, language detection, and speech transcription all load models through a single EngineFactory, sharing configuration, caching, and rate-limiting infrastructure.

🧠
Text Embedding
Batch vectors · Matryoshka · OpenAI-compatible
🔁
Reranking
Bi-encoder document relevance
🎲
Decision Answering
choice / score / noul tasks
🌐
Translation
Marian language pairs · auto source detection
🔤
Language Detection
Confidence and reliability flags
🎙️
Speech Transcription
Qwen3-ASR with VAD segmentation

📋 Table of Contents

📑 Table of Contents

✨ Features

🚀 High Performance
Optimized Rust codebase with batching and concurrent request processing; jemalloc global allocator enabled by default on Linux
🔧 Multiple Engines
ONNX Runtime (default) and Candle (opt-in fallback) inference engines, switchable via EngineFactory
🔁 Rerank
Bi-encoder based document reranking over HTTP/gRPC/CLI
🌍 Internationalization (i18n)
ICU+Fluent bilingual (English/Chinese) error responses with Accept-Language per-request negotiation
🌐 Translation & Language Detection
Marian/opus-mt multi-pair translation ((src,tgt) routing + auto detection) and whatlang language detection, onnx feature
🎮 Decision Answering
Laya decision pipeline: choice / score / noul with temperature-calibrated probabilities
🎮 GPU Acceleration
Native NVIDIA CUDA and Apple Metal support; opt-in mkl/accelerate CPU backends
🌐 Multi-Protocol Interface
HTTP/REST, gRPC, MCP, and CLI generated from a single source by sdforge
🧩 7-Library Ecosystem
Modular ecosystem: trait-kit/confers/inklog/oxcache/limiteron/dbnexus/sdforge
📊 Smart Caching
High-performance caching via oxcache (LRU/LFU/FIFO + TTL) and three-stage semantic cache lookups
🔐 Enterprise Security
JWT authentication, CSRF protection, role-based access control, TOTP, account lockout, and audit logging
⚡ Rate Limiting
Token-bucket rate limiting via limiteron (global/IP/user/API-key dimensions)
📈 Priority Queue
Configurable request priorities, weighted fair scheduling, and time-window dynamic batching
🧊 Matryoshka Support
Dynamic dimensionality reduction (auto re-normalization after truncation) for smaller, faster embeddings (OpenAI-compatible)
🔍 Observability
Prometheus metrics, health checks, structured logging (inklog console + file rotation)
📦 Cloud-Native Deployment
Docker image (linux/amd64, single-arch by default build); Kubernetes deployment guidance (manifests bring-your-own)

In addition to the core capabilities above, an OpenAI-compatible endpoint (POST /v1/embeddings, with encoding_format=base64 support), BF16 inference and SIMD vector similarity, GPU memory paging, read-only vecboost doctor diagnostics, Library SDK integration (library mode), and the config_full.toml / config_minimal.toml config presets are also available; the full feature-flag matrix is in 🔖 Feature Flags, and endpoint details are in 🔌 API Usage.


🚀 Quick Start

📦 Installation

Prerequisites:

📋 Table
Dependency Version Notes
Rust 1.91+ edition 2024 (rust-version in Cargo.toml is authoritative)
Cargo 1.91+ ships with Rust
CUDA Toolkit 12.x optional, NVIDIA GPU support (cuda feature)
Metal SDK latest optional, Apple Silicon GPU support (metal feature)
protobuf-compiler latest optional, required for gRPC E2E tests

💡 Tip: run rustc --version to verify your Rust installation.

📄 Code · bash
# 1. Clone the repository
git clone https://github.com/Kirky-X/vecboost.git
cd vecboost

# 2. Default build (http + onnx features)
cargo build --release

# 3. Run (reads config/config.toml by default)
./target/release/vecboost

Minimal build: cargo build --no-default-features --features http. For more build combinations (GPU / multi-protocol / MCP / the CI full-feature set), see the 📖 User Guide · Installation.

Configure and run:

📄 Code · bash
# Copy and customize the config (defaults to config/config.toml)
cp config/config.toml config/config_custom.toml
# Edit config/config_custom.toml

# Run with a custom config (--config; it is stripped before clap parsing, so position before or after the subcommand both work)
./target/release/vecboost --config config/config_custom.toml

✅ Success: the service starts at http://127.0.0.1:9002 by default (secure default: loopback only).

🐳 Docker: docker build -t vecboost:latest ., then run with config/ and models/ mounted; Docker Compose and Kubernetes deployment are covered in the 📖 User Guide · Docker deployment.

💡 Minimal Example

The following example is adapted from examples/http/embed_api.rs and generates embeddings over HTTP (full endpoints in the 📘 API Reference):

📄 Code · bash
curl -X POST http://localhost:9002/api/v1/embed \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello, world!"}'

Response:

📄 Code · json
{
  "embedding": [0.123, 0.456, 0.789, ...],
  "dimension": 1024,
  "processing_time_ms": 15.5
}

Request/response examples for the remaining endpoints — decision answering, translation, language detection, transcription — are in the 📘 API Reference; you can also use the CLI (cli feature) or the library SDK (library mode), with subcommands and tool lists in 💻 CLI tools.


🔌 API Usage

VecBoost generates its four protocol interfaces (HTTP/gRPC/MCP/CLI) from single-source handlers via sdforge (one source per domain under src/api/); all endpoints, parameters, request/response examples, the gRPC method table, MCP tool lists, and error codes are documented in the 📘 API Reference, and the interactive OpenAPI docs in 📚 OpenAPI documentation (/swagger-ui/ and /redoc, requiring the swagger-ui feature).


🔧 Configuration

The default config path is config/config.toml (--config <path> selects another; a missing explicit path fails fast with an error; the repo ships config_full.toml / config_minimal.toml presets). Environment variables with the VECBOOST_ prefix override the config file, and secrets (VECBOOST_JWT_SECRET / VECBOOST_ADMIN_PASSWORD) must be provided via env vars; config file changes are validated and logged, taking effect after restart.

Per-key options and defaults for every config section (server / model / embedding / rerank / monitoring / auth / rate_limit / audit / database / logging / pipeline.worker / semantic_cache / device / translation / transcribe), the full environment-variable table, and a complete example config are documented in the 📖 User Guide · Configuration; you can also read config/config.toml directly.

⚠️ Note: the [flow_control] and [cache] TOML sections are not parsed in the current version (legacy section names); rate limiting uses [rate_limit], caching uses [embedding] and [semantic_cache]. See the ❓ FAQ.


📚 Documentation

📋 Table
Document Description
📖 User Guide Complete tutorial from installation to advanced usage (incl. deployment)
📘 API Reference Full REST / gRPC / OpenAI-compatible interface documentation
🏗️ Architecture Design principles, module layout, and data flow
⚡ Performance Guide Benchmark data, tuning-switch registry, and experiment discipline
🔒 Security Security design, supported versions, and vulnerability reporting
❓ FAQ Frequently asked questions
🧪 Test Scenarios Test-stack responsibilities and the scenario matrix
📋 Changelog Release-by-release change log
🤝 Contributing How to contribute
🧹 Maintenance Norms Cross-change standing constraints: transcribe consumption surface, CLI contract, retired-switch discipline, deprecation-doc governance

ℹ️ Note: docs/benchmarks/ (benchmark archive), docs/reviews/ (review ledgers) and docs-transcribe/ are local-only directories (see .gitignore); they are not committed and do not exist in a clean clone. Key benchmark numbers are inlined in the ⚡ Performance Guide.


💻 Examples

All examples live in examples/, an independent workspace-member crate vecboost-examples (12 categories, 31 runnable binaries) covering basic embedding, HTTP/CLI calls, engine switching, auth, caching, rate limiting, monitoring, audit, semantic cache, and Library SDK integration; the per-category listing is in examples/README.md.


🧱 Architecture

VecBoost uses a modular ecosystem architecture: trait-kit wires modules through a typestate module registry (Kit<Unbuilt> → Kit<Ready>); src/registry/ has 20 production ModuleMeta implementations plus 3 test fixtures (several are feature-gated, e.g. under auth), sdforge generates the four protocol bindings from the single source src/api/embedding.rs, inference is abstracted behind EngineFactory to the Candle / ONNX engines, and requests flow through a priority queue and time-window batching into the inference pipeline.

The 7-library ecosystem (trait-kit / confers / inklog / oxcache / limiteron / dbnexus / sdforge) versions and responsibilities, the module dependency graph, data flow, and the cache / security / deployment architecture and extension points are documented in the 🏗️ Architecture document.


🧪 Testing

The test stack has six layers: inline unit tests in src/, integration tests in tests/integration/, specialized integration tests (doctor / gRPC E2E / model snapshot regression / quantization quality gate / SDK matrix), real-service scenario tests in tests/scenario/*.py (16 pytest suites), performance regression thresholds in tests/perf/, and 6 Criterion microbenchmark suites in benches/. Layer responsibilities, the exhaustive scenario matrix, and the run commands (identical to CI) are in 🧪 Test Scenarios; environment setup and quality gates are in 🤝 Contributing.


📊 Performance

Benchmark data comes from the measured docs/benchmarks/ archive (criterion; that directory is local-only and git-ignored, so it does not exist in a clean clone): SIMD vector similarity is up to 3.06x faster than the scalar baseline (~341.7 ns for 1024-dim cosine), exact semantic-cache hits take ~10 ns, and candle single-text embedding is 4.0x faster with --features mkl. Full benchmark tables, measured engine-selection evidence, performance design highlights (time-window batching / in-batch dedup / SIMD / thread tuning / jemalloc), GGUF quantization and the tuning-switch registry are in the ⚡ Performance Guide.


🔒 Security

🔒 Security Design

VecBoost is secure by default: the factory config binds to loopback only, and non-loopback binding with auth.enabled=false refuses to start (the VECBOOST_ALLOW_INSECURE=1 escape hatch logs an ERROR); authentication and authorization build on garrison (JWT + CSRF + RBAC admin role + TOTP + account lockout), plus XFF trust inversion, file-path allowed roots, input length limits, AES-256-GCM config encryption, audit logging, and i18n bilingual error masking. Mechanism-level details are in the 🔒 Security document.

🔗 Supply Chain & Gates

cargo audit, cargo deny check, CodeQL, Trivy/Checkov image scanning, gitleaks secret scanning, and pre-commit hooks run both in CI and locally; the full list and triage policy are in the 🔒 Security document · Supply chain & gates.

🚨 Reporting a Vulnerability

Please do not report security vulnerabilities through public issues. Use the GitHub Security Advisories private disclosure channel as the primary route (fall back to Kirky.X at kirky-x@outlook.com only if that channel is unavailable). Dependency advisories are gated by CI cargo-audit. For the full policy and supported versions, see SECURITY.md.


🧭 Roadmap

Roadmap items (core service / ecosystem integration / security & i18n / performance foundation / tuning switches / multi-replica shared state, etc.; ✅ done / 📋 planned) are maintained with version evolution in the 📋 Changelog · Roadmap.


🤝 Contributing

For the detailed workflow and code standards, see the 🤝 Contributing guide. Found a problem? Open an issue; have an idea? Start a discussion; fork and open a PR.


📋 Changelog

The full release history is in the 📋 Changelog (following Keep a Changelog, semantic versioning).

0.3.0-rc.1 contains multiple breaking behavioral changes (secure defaults hardening, login convergence, XFF trust inversion, RBAC wiring, cache-key/tokenizer changes, etc.) — read before upgrading: the per-item "old behavior → new behavior → migration" table is in the 📋 Changelog · 0.3.0-rc.1. The multi-replica boundary (auth sessions default to process memory and can be externalized with session_backend = "database"; rate limiting and caches can now be externalized to Redis via distributed-redis) and hot-reload semantics (config changes take effect after restart) are covered in the ❓ FAQ.


📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details. Copyright © 2025-2026 Kirky.X🌠.


🙏 Acknowledgements

The core-dependency list (onnxruntime / candle / tokenizers / hf-hub / the 7-library ecosystem / garrison / axum / tokio / utoipa / prometheus / criterion / jemallocator) is in 🏗️ Architecture · Core dependency acknowledgements.

Thanks to the Rust community, Hugging Face (model & tokenizer ecosystem), and all contributors.


📞 Contact & Support

Issues
Report problems and bugs
Discussions
Ask questions and share ideas
GitHub
Browse the source

© 2026 Kirky.X. All rights reserved.

About

Production embedding & decision inference service in Rust — Candle/ONNX dual engine, CUDA/Metal acceleration, REST/gRPC/MCP/CLI generated via sdforge, rerank, Matryoshka dimensions, OpenAI-compatible API, and semantic caching

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages