Add Lynkr router submission#167
Conversation
Lynkr is an open-source (Apache-2.0), self-hosted LLM gateway that routes requests via embedding-anchor tiering. Pool: openai/gpt-oss-120b, qwen3-235b-a22b-2507, z-ai/glm-4.7 (all via OpenRouter). Includes router config, full-split predictions with generated_result, robustness predictions, the router adapter class, and additive entries to model_cost/universal_model_names for the pool models. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
/evaluate |
…t router 13 entries where the model responded but no answer text was extractable now carry generated_answer="" and success=false with an explanatory error field (previously null answer with success=true). Validator passes locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Optimality Metrics
Evaluation completed by RouterArena automated workflow |
All previously-failed entries (credit exhaustion during original run) now have real generated results — 176/176 successful. Validator passes locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
/evaluate |
1 similar comment
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Optimality Metrics
Evaluation completed by RouterArena automated workflow |
|
@yl231 The full-split evaluation is now clean — 0 abnormal entries (down from 6 on the previous run; root cause was the truncated entries in the regenerated prediction file). Final numbers from the run above: Arena 0.6765, Accuracy 68.41%, $0.2924/1K queries, Robustness 0.9238, 8,400/8,400 scored, all CI checks green. The submitted config is Lynkr's untuned production default, per the evaluation-only policy. Ready for review whenever you have a moment — happy to adjust anything needed for the leaderboard entry. |
Router: Lynkr
Lynkr is an open-source (Apache-2.0), self-hosted LLM gateway that routes requests across local and cloud models. Routing here uses its embedding-anchor tier classifier (no LLM classifier call — routing decisions are local and deterministic).
openai/gpt-oss-120b,qwen/qwen3-235b-a22b-2507,z-ai/glm-4.7Files
router_inference/config/lynkr.json— router configurationrouter_inference/predictions/lynkr.json— full-split predictions withgenerated_resultpopulatedrouter_inference/predictions/lynkr-robustness.json— robustness predictionsrouter_inference/router/lynkr_router.py+ registration in__init__.py— adapter for reproducibilitymodel_cost/model_cost.json(z-ai/glm-4.7 pricing) anduniversal_model_names.pyfor the pool modelsLocal evaluation results (for reference; please re-run officially)
Arena 67.65 · Accuracy 68.41% · $0.29/1K queries · Robustness 92.38 · Opt.Sel 11.13 / Opt.Cost 17.00 / Opt.Acc 85.37
Disclosures
gpt-oss-120b(~96% of queries) — a consequence of the tier boundaries, not tuning on RouterArena data.Happy to adjust anything to match submission conventions.
Leaderboard row details (for the manifest/README, to save you a lookup)
lynkr/lynkr-robustness(this PR)