Skip to content
@openinferencefabric

Open Inference Fabric

Open source, bare metal to tokens: Run:ai, KAI Scheduler and Dynamo on NVIDIA and AMD GPUs

Open Inference Fabric

Open source, bare metal to tokens: one platform to schedule, serve, benchmark and observe LLMs on NVIDIA and AMD GPUs.

Website: https://openinferencefabric.github.io

oif installs, with one command, a Kubernetes platform on GPU servers: open-source NVIDIA Run:ai and the KAI Scheduler for projects, quotas and fractional GPUs, NVIDIA Dynamo for serving, AIPerf for benchmarks, and Grafana dashboards for every GPU. NVIDIA and AMD GPUs run side by side in one cluster, and every step has a tested guide. Apache-2.0.

Upstream work: AMD GPUs in the open-source NVIDIA stack

Our changes go back to the upstream projects, not into private copies. Everything below is open for review.

Project What we sent
Run:ai PRs #3950 AMD GPUs in the node inventory · #3948 AMD icon for ROCm images · #3949 GPU fractions readiness; issue #4274 AMD GPU metrics
KAI Scheduler PRs #2557 reservation pod resources · #2594 topology required level; design for AMD GPU fractions #2556
KAI gpu-fractioning PR #175 hard VRAM limit per pod on AMD GPUs (Linux dmem cgroup)
NVIDIA Dynamo ROCm images for vLLM: co-developing #12537 (AMD) on branch rocm, image builds on vLLM 0.33.0, benchmark on the Radeon PRO W7900 next
ROCm Issues rocm-systems#14860 VRAM limits in hipMemGetInfo · #14861 per-process GPU busy time · device-metrics-exporter#722 per-pod metrics for shared GPUs
KWOK Issue #1802 AMD Instinct example nodes

Every item, with lab evidence: docs/UPSTREAM.md · live status: https://openinferencefabric.github.io/upstream/.

Get involved

  • Test on NVIDIA, AMD or mixed GPU servers and tell us what breaks.
  • Review the upstream pull requests above, or try the Dynamo ROCm branch on your AMD GPU.
  • Contribute Go, Kubernetes, ROCm/CUDA or docs: start with CONTRIBUTING.md.

Pinned Loading

  1. oif oif Public

    One open-source platform to schedule, serve, benchmark and observe LLMs on NVIDIA and AMD GPUs (Run:ai, Dynamo, AIPerf)

    Shell 2

Repositories

Showing 5 of 5 repositories

People

This organization has no public members. You must be a member to see who’s a part of this organization.

Top languages

Loading…

Most used topics

Loading…