feat: add sticky least-loaded session routing

[mikasenghaas]

Summary

• squash the two commits from feat(router): add sticky least-loaded session routing vllm-project/router#235 into one upstream-authored feature commit
• keep fork-specific compatibility and image fixes in a separate integration commit
• add the sticky_least_loaded routing policy and explicit /finish_session lifecycle endpoint
• preserve legacy session identifiers while accepting configured request-ID headers
• include the upstream policy, API, parser, and routing coverage
• adapt the discovered P/D test to the fork-specific optional run_id argument
• upgrade runtime Debian packages to pick up current security fixes
• bound sticky session state and session-ID size with stateless fallback, sweeping expired entries before degrading affinity
• resolve the effective model policy before choosing typed-body routing input
• preserve supported legacy session fields across typed request schemas
• keep per-request headers available to consistent hashing without letting them mask sticky session IDs
• round-robin stateless requests across equally least-loaded workers

Motivation: Better load-balancing across engines compared to previous default consistent_hashing (green)

Note: The drop on the purple curve was a node failure and job restart, and not caused by the routing scheme.

Validation

• cargo +1.95.0 fmt --check

• cargo +1.95.0 clippy --all-targets -- -D warnings

• cargo +1.95.0 test

• uv run --isolated --extra dev pytest py_test/unit -q (105 passed, 4 skipped)

• GitHub Actions: all checks pass, including image build and Trivy scan

Context

prime-rl#3553 compares consistent hashing with sticky least-loaded routing. The experiment needs published manylinux wheels from this fork so cluster uv sync --all-extras --all-packages does not compile the Rust extension on compute nodes.

Upstream: vllm-project#235

Note

Add StickyLeastLoadedPolicy for session routing

• Introduces StickyLeastLoadedPolicy to route requests to the least-loaded replica using a session identifier, preserving affinity until sessions expire or workers become unhealthy.
• Adds extract_session_id in src/policies/hash_key.rs to read session IDs from specific headers and request bodies, and updates the HTTP router to pass full request bodies and headers to policies that require them.
• Adds a POST /finish_session endpoint in src/server.rs to release session assignments, and extends LoadBalancingPolicy with finish_session and needs_request_body default methods.
• Updates PolicyFactory, PolicyRegistry, config enums, CLI arguments, and Python bindings to accept the sticky_least_loaded policy.
• Behavioral Change: Router::route_typed_request in src/routers/http/router.rs now serializes the full request for body-dependent policies; serialization failures return HTTP 500. Dockerfile.router now upgrades Debian packages and installs ca-certificates without recommendations.

^{Macroscope summarized f64492f.}

---

Note

Medium Risk
Changes core load-balancing and routing input selection (full-body serialization, mutex-backed session state); misconfigured session IDs or missing finish_session can skew load until TTL, and multi-router deployments need coordinated ingress and release.

Overview
Introduces sticky_least_loaded, a session-aware policy that pins ongoing sessions to one worker while assigning new sessions to the replica with the fewest active sessions (ties broken with rendezvous hashing). Session IDs come from the same headers as consistent hashing plus JSON fields (session_params.session_id, user, session_id, user_id); requests without an ID still load-balance but are not tracked.

Adds POST /finish_session?session_id=... (auth-gated) and registry fan-out so default, per-model, and PD prefill/decode policy instances can release reservations. Idle sessions expire (default 2h, env-configurable) with bounded state (max sessions / ID length, stateless fallback when limits are hit).

Routing plumbing changes: policies can require the full serialized typed body (not just prompt text), PD selection passes headers into policies, and request schemas gain optional legacy session fields. CLI, config, Python bindings, docs, tests, and the runtime Dockerfile (apt upgrade) are updated accordingly.

^{Reviewed by Cursor Bugbot for commit 8cdb9f0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a substantial stateful session-routing policy, session-release API, PD routing integration, and production container package upgrades rather than a small isolated change. Its session lifecycle and cross-scope routing behavior require human review, including an unresolved identifier-collision risk.

^{Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 8cdb9f0. Configure here.}