Skip to content

About

A Rust inference engine for large language models, built on Candle and custom CUDA kernels.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

2 stars

Watchers

0 watching

Forks

vllm-oxide

English | 简体中文

CI Release License: Apache-2.0

vllm-oxide is a large language model inference engine written in Rust. It uses Candle and custom CUDA kernels, with an architecture informed by nano-vllm and vLLM. The engine supports continuous batching, prefix caching, a paged KV cache, and recompute preemption.

v0.2.0 is available as a source release, validated with Qwen3-0.6B in BF16 on one CUDA GPU.

Getting started · Documentation · Architecture · Releases · Contributing

What you can build

Use the Rust library to generate from a batch of text or token-ID prompts. Each result contains the generated text and token IDs. Reusing one LLM across calls keeps the model loaded and its cache allocation available. The CLI uses the same generation path for one prompt at a time.

Area v0.2.0 support
Model and execution Qwen3-0.6B, BF16, one CUDA GPU, synchronous offline generation
Request handling Continuous batching within a generation call, chunked prefill, stable request IDs, input-order results
KV cache Paged storage, shared-prefix caching, copy-on-write, recompute preemption
Token selection Greedy, temperature, top-k, top-p, presence/frequency/repetition penalties, EOS and token-limit stopping
Distribution Tagged source, a Rust library, and a CLI; no prebuilt binary or published crates.io package

Serving, async streaming, multi-GPU/TP/NCCL, other model families, quantization, LoRA, speculative decoding, and CUDA Graphs are outside this release.

Quick start

GPU inference needs Linux or WSL2, Rust 1.94 or newer, an NVIDIA GPU with compute capability 8.9 or newer, a working NVIDIA driver, and a CUDA Toolkit containing nvcc. The release was tested on RTX 4080 with driver 595.71 and Toolkit 13.2.51; other hardware and toolkit combinations have not received the same release validation.

Clone the released source and run the provided example with the exact model revision used for validation:

git clone --branch v0.2.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.git
cd vllm-oxide
cargo run --locked --release -p vllm_oxide --example generate_qwen3 --features cuda -- \
    hub:Qwen/Qwen3-0.6B@7e4ae267688d671ddfca3122e4528ee980cf3234

The first run builds the CUDA backend and downloads the model. The example continues the prompt "The meaning of life is". Later runs reuse the downloaded files. To use a local checkpoint, replace the hub: argument with its directory.

To choose your own prompt, use the CLI with a local checkpoint:

export QWEN3_MODEL_DIR=/absolute/path/to/Qwen3-0.6B
cargo run --locked --release -p vllm_oxide_cli --features cuda -- \
    --model "$QWEN3_MODEL_DIR" --max-tokens 2 "The capital of France is"

The CLI also accepts a Hub repo ID, such as Qwen/Qwen3-0.6B, but does not expose a revision flag. See getting started for CLI options, local checkpoints, and offline use.

Use from Rust

Pin the source release in your Cargo.toml:

[dependencies]
vllm_oxide = { git = "https://github.com/RedHeartSecretMan/vllm-oxide.git", tag = "v0.2.0", features = ["cuda"] }
anyhow = "1"
use vllm_oxide::{EngineOptions, Prompt, SamplingParams, Source, LLM};

fn main() -> anyhow::Result<()> {
    let mut llm = LLM::new(
        Source::Hub {
            repo: "Qwen/Qwen3-0.6B".into(),
            revision: Some("7e4ae267688d671ddfca3122e4528ee980cf3234".into()),
        },
        EngineOptions::default(),
    )?;
    let outputs = llm.generate(
        &[Prompt::Text("The capital of France is".into())],
        &[SamplingParams {
            max_tokens: 2,
            ..Default::default()
        }],
    )?;
    println!("{}", outputs[0].text);
    Ok(())
}

The supported public types are LLM, EngineOptions, Prompt, SamplingParams, RequestOutput, and Source. generate accepts one sampling policy per prompt and returns outputs in input order with stable request IDs. Text and token-ID prompts can share a batch. See the usage guide and v0.2 migration guide for details.

Scope and performance

The validation summary records the release's numerical, behavior, and public CUDA checks. In the fixed RTX 4080/WSL2 workload, median decode throughput was 29.7601 token/s for one request and 115.8207 token/s across four requests. See benchmarks for the workload, measurement boundaries, and memory usage. These results apply to the measured workload and hardware; they do not establish performance parity with vLLM.

Development and validation

CPU development does not require CUDA:

cargo build --locked --workspace
cargo test --locked --workspace

Use CPU builds for development and tests. Inference requires --features cuda and a GPU. GitHub CI checks compilation, formatting, Clippy, Rust tests, Python CPU validation tools, and dependencies. Model accuracy, CUDA API/CLI behavior, and performance have separate checks on a local GPU.

For contribution setup and complete checks, read CONTRIBUTING.md and the development guide. The architecture overview maps a generation call to the current implementation, and the validation guide separates routine CI from real-model checks.

Community and project information

Looking for Start here
Usage help, bugs, or a feature proposal Support and GitHub Issues
A contribution Contributing and open pull requests
Community expectations Community standards
A private security report Security policy
Version history or an upgrade Releases, changelog, and v0.2 migration
Implementation decisions and experiments Architecture, ADRs, and research

License

Apache-2.0.

About

A Rust inference engine for large language models, built on Candle and custom CUDA kernels.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages