English | 简体中文
vllm-oxide is a large language model inference engine written in Rust. It uses Candle and custom CUDA kernels, with an architecture informed by nano-vllm and vLLM. The engine supports continuous batching, prefix caching, a paged KV cache, and recompute preemption.
v0.2.0 is available as a source release, validated with Qwen3-0.6B in BF16 on one CUDA GPU.
Getting started · Documentation · Architecture · Releases · Contributing
Use the Rust library to generate from a batch of text or token-ID prompts. Each result contains the generated text and token IDs. Reusing one LLM across calls keeps the model loaded and its cache allocation available. The CLI uses the same generation path for one prompt at a time.
| Area | v0.2.0 support |
|---|---|
| Model and execution | Qwen3-0.6B, BF16, one CUDA GPU, synchronous offline generation |
| Request handling | Continuous batching within a generation call, chunked prefill, stable request IDs, input-order results |
| KV cache | Paged storage, shared-prefix caching, copy-on-write, recompute preemption |
| Token selection | Greedy, temperature, top-k, top-p, presence/frequency/repetition penalties, EOS and token-limit stopping |
| Distribution | Tagged source, a Rust library, and a CLI; no prebuilt binary or published crates.io package |
Serving, async streaming, multi-GPU/TP/NCCL, other model families, quantization, LoRA, speculative decoding, and CUDA Graphs are outside this release.
GPU inference needs Linux or WSL2, Rust 1.94 or newer, an NVIDIA GPU with compute capability 8.9 or newer, a working NVIDIA driver, and a CUDA Toolkit containing nvcc. The release was tested on RTX 4080 with driver 595.71 and Toolkit 13.2.51; other hardware and toolkit combinations have not received the same release validation.
Clone the released source and run the provided example with the exact model revision used for validation:
git clone --branch v0.2.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.git
cd vllm-oxide
cargo run --locked --release -p vllm_oxide --example generate_qwen3 --features cuda -- \
hub:Qwen/Qwen3-0.6B@7e4ae267688d671ddfca3122e4528ee980cf3234The first run builds the CUDA backend and downloads the model. The example continues the prompt "The meaning of life is". Later runs reuse the downloaded files. To use a local checkpoint, replace the hub: argument with its directory.
To choose your own prompt, use the CLI with a local checkpoint:
export QWEN3_MODEL_DIR=/absolute/path/to/Qwen3-0.6B
cargo run --locked --release -p vllm_oxide_cli --features cuda -- \
--model "$QWEN3_MODEL_DIR" --max-tokens 2 "The capital of France is"The CLI also accepts a Hub repo ID, such as Qwen/Qwen3-0.6B, but does not expose a revision flag. See getting started for CLI options, local checkpoints, and offline use.
Pin the source release in your Cargo.toml:
[dependencies]
vllm_oxide = { git = "https://github.com/RedHeartSecretMan/vllm-oxide.git", tag = "v0.2.0", features = ["cuda"] }
anyhow = "1"use vllm_oxide::{EngineOptions, Prompt, SamplingParams, Source, LLM};
fn main() -> anyhow::Result<()> {
let mut llm = LLM::new(
Source::Hub {
repo: "Qwen/Qwen3-0.6B".into(),
revision: Some("7e4ae267688d671ddfca3122e4528ee980cf3234".into()),
},
EngineOptions::default(),
)?;
let outputs = llm.generate(
&[Prompt::Text("The capital of France is".into())],
&[SamplingParams {
max_tokens: 2,
..Default::default()
}],
)?;
println!("{}", outputs[0].text);
Ok(())
}The supported public types are LLM, EngineOptions, Prompt, SamplingParams, RequestOutput, and Source. generate accepts one sampling policy per prompt and returns outputs in input order with stable request IDs. Text and token-ID prompts can share a batch. See the usage guide and v0.2 migration guide for details.
The validation summary records the release's numerical, behavior, and public CUDA checks. In the fixed RTX 4080/WSL2 workload, median decode throughput was 29.7601 token/s for one request and 115.8207 token/s across four requests. See benchmarks for the workload, measurement boundaries, and memory usage. These results apply to the measured workload and hardware; they do not establish performance parity with vLLM.
CPU development does not require CUDA:
cargo build --locked --workspace
cargo test --locked --workspaceUse CPU builds for development and tests. Inference requires --features cuda and a GPU. GitHub CI checks compilation, formatting, Clippy, Rust tests, Python CPU validation tools, and dependencies. Model accuracy, CUDA API/CLI behavior, and performance have separate checks on a local GPU.
For contribution setup and complete checks, read CONTRIBUTING.md and the development guide. The architecture overview maps a generation call to the current implementation, and the validation guide separates routine CI from real-model checks.
| Looking for | Start here |
|---|---|
| Usage help, bugs, or a feature proposal | Support and GitHub Issues |
| A contribution | Contributing and open pull requests |
| Community expectations | Community standards |
| A private security report | Security policy |
| Version history or an upgrade | Releases, changelog, and v0.2 migration |
| Implementation decisions and experiments | Architecture, ADRs, and research |