feat(router): add sticky least-loaded session routing

[bvolpato]

Purpose

Add opt-in sticky_least_loaded routing for multi-turn workloads. New sessions reserve the healthy worker with the fewest active sessions, with deterministic rendezvous tie-breaking. Existing sessions keep their worker until explicit release, idle expiry, or worker unavailability. Selection and reservation share one mutex so concurrent arrivals do not all observe the same free capacity.

This addresses the session-policy gap described in NovaSky-AI/SkyRL#2070. It balances active sessions, not concurrent requests, queued tokens, or GPU utilization.

• Add Rust/Python configuration and CLI support for regular, prefill, and decode policies.
• Preserve header/body session IDs through typed routes and service-discovered PD routing. Parse JSON IDs without losing escapes or treating unrelated nested fields as sessions.
• Add authenticated POST /finish_session?session_id=..., with idempotent release across registered policies. Check revisited sessions for expiry before renewing their affinity.

Adapted from the public Apache-2.0 implementation in SumanthRH/router@b50d926. The existing license is preserved. AI assistance was used for adaptation and regression coverage.

This change does not retire the downstream fork by itself. Migration still needs the typed routing paths in #199, consolidated request-accounting/response-lifetime work from #200/#216 (including #215/#176), and the endpoint cancellation work in NovaSky-AI/SkyRL#2121. This PR does not change request-load guards or cancel backend generation.

Test Plan

Run with Rust 1.95 and the compiled Python extension:

cargo test --locked --offline --lib
cargo test --locked --offline --test api_endpoints_test
pytest py_test/unit -q
cargo fmt --all -- --check
black --check --target-version py38 py_src/vllm_router/router.py py_src/vllm_router/router_args.py py_test/unit/test_arg_parser.py
ruff check py_src/vllm_router/router.py py_src/vllm_router/router_args.py py_test/unit/test_arg_parser.py

Test Result

• Rust library: 504 passed. API endpoints: 48 passed.
• Python unit tests: 105 passed, 4 skipped. Formatting and lint checks passed.
• Reproduced failures before the fixes: typed-body sessions were not tracked, escaped IDs were truncated, unrelated nested/null IDs were accepted, expired affinity was revived before the sweep, and header-only discovered PD requests reserved no session. These regressions pass after the fixes.
• Local macOS CPU/mock-worker coverage only. No GPU, tensor-parallel abort latency, or throughput claim.

Downsides

• State is router-process-local and lost on restart. Multiple router replicas need consistent ingress and release on each router that tracked the session.
• Shared/default policy instances require model-scoped session IDs across disjoint model pools. Separate per-model policies already isolate their state. Release fans out by ID, so IDs should be globally unique.
• Default idle TTL is two hours, configurable with VLLM_ROUTER_SLL_SESSION_EXPIRATION_IN_S. Idle entries are swept on routing requests at most once per minute; a revisited entry is checked immediately. Callers must release finished sessions for prompt balancing.
• One entry is retained per active session. Assignment uses a mutex, and typed requests using this policy incur JSON serialization/parsing. Session counts do not estimate session length or request/GPU utilization.

Risk and rollback

Opt-in only; existing policy defaults remain unchanged. To roll back, choose the previous policy and restart the router. No persisted data migration is needed.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-03T04:26:05.112018Z	3810b63	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3810b631d7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[hsliuustc0106]

Reviewed at f2ae9ea4. This is the complete version of the sticky-session design (relevant given the overlap with #306, whose own TODO cites this approach):

• TTL expiry (2 h default, VLLM_ROUTER_SLL_SESSION_EXPIRATION_IN_S) with sweeps bounded to once per 60 s under the lock.
• Selection + recording are atomic under one mutex, so concurrent new sessions can't all observe the same free capacity; rendezvous hashing gives deterministic tie-breaks.
• Header-less requests are load-balanced without being recorded as sessions — extract_session_id returns None instead of falling back to hashing the request body, avoiding the per-prompt map-growth problem.
• /finish_session is properly behind authorize_request.

One documentation note (P3): the idle TTL silently breaks stickiness for sessions whose turns are more than 2 hours apart — sticky KV-cache reuse quietly degrades and the session may be reassigned. It's tunable, but worth stating in user-facing docs.

Process note: the branch predates the #283 merge (touches protocols/spec.rs and hash_key.rs), so expect a routine rebase. No findings beyond that.

---

From an automated daily review pass over new/updated PRs (head SHA frozen at f2ae9ea4).