LLM Compare — free multi-model comparison via Merge Gateway

Send one prompt to several LLMs through Merge Gateway and compare their answers side by side. Each run saves Markdown, JSON, and a self-contained HTML report with stats, chain-of-thought, and a neutral analyzer write-up.

Built following the gateway-implement skill from merge-gateway-skills (Merge Gateway SDK, provider/model IDs, MERGE_GATEWAY_API_KEY).

Setup — free

pip install -r requirements.txt

All you need is a Merge Gateway key — free to start, no credit card, ~2 minutes:

1. Sign up at https://gateway.merge.dev/signup (free tier included)
2. Create a key in the dashboard (starts with mg_)
3. Pass it to the script — first hit wins: --api-key, the MERGE_GATEWAY_API_KEY env var, or a .gateway-key file next to compare.py. The script prints these steps if it can't find a key.

Runs themselves cost fractions of a cent on cheap models (a trivial 2-model test runs ~$0.0001); every report itemizes cost per model, and --reanalyze / --rerender reuse saved runs for free.

Free-tier notes (per Merge Gateway pricing, Sep 2026): new accounts start with $10 in free LLM credits, and the free plan covers free-tier LLMs with no credit card. Stepping up to Pro requires putting a card on file, but Pro includes a recurring $10/month credit plus all major frontier models — stay within the $10 and you pay nothing. That Pro-level access is what this project's models.txt assumes (a full 9-model run costs ~$0.55).

Usage

# Prompt as argument, interactive input, file, or pipe
python compare.py "Explain recursion in one sentence"
python compare.py --prompt-file prompt.txt --models-file my_models.txt
Get-Content prompt.txt | python compare.py

# Repeat each request, skip analysis, pick analyzer
python compare.py -p "..." --repeat 3 --no-analysis
python compare.py -p "..." --analyzer openai/gpt-5-mini

# Custom analyzer question (answered first, then standard comparison)
python compare.py -p "..." --ask "Which answer is most actionable?"
# ... or ONLY the custom question, no standard comparison
python compare.py -p "..." --ask-only "Which answer is most actionable?"

# Reuse saved runs: analyzer-only re-run, or free re-render
python compare.py --reanalyze results/run.json --output-name run_v2
python compare.py --rerender results/run.json --output-name run_v2

# Resume an interrupted run, or add models to a past run (old runs are kept
# untouched; progress checkpoints after every model, so Ctrl+C is safe)
python compare.py --resume results/run.partial.json --output-name run
python compare.py --resume results/run.json --models xai/grok-4.6 --output-name run_v2

python compare.py --list-models   # what your key can access

Flag	Default	Meaning
--models-file / --models, -m	models.txt	One provider/model ID per line (# comments ok, friendly names mapped)
--drop	—	Exclude models (fresh runs, --resume, --reanalyze, --rerender)
--repeat, -n	1	Query each model N times
--analyzer / --no-analysis	zai/glm-5.3-flash	Model writing the comparison; or skip it
--ask / --ask-only	—	Custom analyzer question + standard output / instead of it
--max-run-chars	3000	Chars per response sent to the analyzer
--out-dir / --output-name	results/ / timestamp	Output location
--formats	md,json,html	Subset to write
--temperature / --max-tokens / --top-p	—	Passed through to every request
--workers / --sequential	4	Parallelism across models
--reanalyze / --rerender	—	Analyzer-only rerun / free rewrite from saved JSON

Outputs (results/<name>.md|.json|.html)

• Stats per model and totals: runs ok, latency avg/min/max, input/output/ reasoning/cache-read tokens, cost, CoT block count, wall time.
• Responses: every run's answer, its chain-of-thought blocks (when the gateway returns them), and usage JSON.
• Analysis: neutral, no scores or verdicts — custom answer first (if asked), then Comparison summary, per-model overview, comparison table, differences, consensus/divergence. The exact analyzer instruction is stored in the JSON and shown in the HTML.
• The HTML is a single file (no dependencies): stat cards, model-stats table linking to answers, run tabs, copy buttons, rendered Markdown.

Version control (jj)

This repo uses Jujutsu on top of git (jj git init — pushes to GitHub normally). Basics:

jj st                    # working-copy status
jj describe -m "msg"     # set change description
jj new                   # start a new change
jj log --limit 5
jj git push -b main

Notes

• The gateway SDK (v0.2.0) mis-parses live payloads, so calls go through the SDK client's raw request path with defensive parsing — same endpoint/auth.
• What your key can access varies: a limited key sees ~82 models while the catalog lists 270+. Unknown/inaccessible models are recorded as per-run errors, not crashes (--list-models shows your key's set).
• Cost driver is usually the analyzer on long outputs, plus hidden reasoning tokens on reasoning models — both are itemized in the report.
• Suggested next runs live in models.txt (edit freely).