chore(ci): 👷 benchmark the ladder rungs, one testbed per shape and silicon by diagonal-hamiltonian · Pull Request #323 · Algorithmiq/monoprop

diagonal-hamiltonian · 2026-09-01T11:48:18Z

🤖 AI text below 🤖

Stacked on #317, which defines the rungs. This is the continuous-benchmarking side only.

What was wrong

The benchmark lanes declared no shape and no problem. monoprop_PARTITIONS was unset everywhere,
so resolve_partition_count_'s ranks == 1 ? cores : 1 decided it — every point in the
ubuntu-26.04 and aws-c7i-4xlarge series was one partition per core of whatever runner it landed
on, unrecorded. A runner resize would have moved every series without moving a testbed name.

The first revision of this PR fixed the shape and left the problem: the rungs were named after
benches/LADDER.md but ran the suite's dataclass defaults, so the sizes had nothing to do with the
rungs they were named after, and nobody had measured what they cost.

What it does now

Each rung declares a shape, a round count and a problem, and the tables below are measured, not
derived. rungs is one rung per line, <name> <ranks> <partitions> <rounds> | <pytest args>.

rung	`R` × `P`	launch	`monoprop_PARTITIONS`	rounds	testbed
L1	1 × 1	`just bench-ci`	1	3	`<cpu>-8c-L1`
L2a	1 × 8	`just bench-ci`	8	1	`<cpu>-8c-L2a`
L2b	4 × 2	`mpiexec -n 4 --map-by slot:PE=2 --bind-to core`	2	1	`<cpu>-8c-L2b`

R × P = 8 at both L2 rungs, so L2a → L2b moves the process count and nothing else. Rounds is 1
above L1 because pedantic builds round k+1's propagator before releasing round k's — at these
sizes that is a memory setting, not a statistics setting.

L1 — LADDER.md's own rows, one thread

row	flags	`-k`	terms	~s	~GiB
hubbard `propagate`	`--hubbard-cutoff=10 --hubbard-lower-atol=4.2e-05`	`test_model_propagate and hubbard`	9,953,109	25	0.87
pauli `propagate`	`--pauli-cutoff=12 --pauli-lower-atol=1.22e-04`	`test_model_propagate and pauli`	10,069,308	21	1.06
random `gradient`	`--num-generators=1000 --num-modes=142 --cutoff=6 --obs-terms=295000`	`test_random_gradient and heisenberg`	19,902,244	11	2.60

One process, one selector — the three flag sets use disjoint options. Whole-rung peak 2.60 GiB in
3:53. The term counts reproduce LADDER.md's L1 table exactly. LADDER.md's fourth L1 row, the same
gradient pared at 1e-10, is dropped: --pare-threshold is session-wide and cannot share a
process with the unpared row.

L2a and L2b — one operator, three operations

build_graph publishes the graph energy and gradient evaluate, so the three share one operator
and the peak is the max over them, not the sum. propagate is excluded: it holds its own. Both
rungs take the same flags.

-k "(test_random_build_graph or test_random_energy or test_random_gradient) and heisenberg"
--num-generators=1000 --num-modes=142 --cutoff=6 --obs-terms=2500000

row	terms	~s L2a	~GiB L2a	~s L2b	~GiB/node L2b	~GiB worst rank
random `build_graph`	167,515,463	83	19.11	68	23.71	6.14
random `energy`	167,515,463	13	17.54	6	20.36	5.17
random `gradient`	167,515,463	39	18.85	16	21.68	5.50

Peak 19.11 GiB at L2a and 23.71 GiB/node at L2b, in 135 s and 90 s. ~GiB/node is the sum over the
node's ranks; it errs high, because a page shared between ranks is charged to each.

--obs-terms=2500000 is the largest measured point that fits, not an interpolation: the next
step tried, 3.5M, reaches 35.49 GiB/node against the instance's 32 GiB. A two-point fit over 1.5M
and 2.5M predicted 31.1 GiB there and under-called it by 3.9, so this axis is not interpolated.

The term count is identical at both shapes — the geometry-independence check, which the terms
measure holds to 0%. L2b's node sum is 1.24x L2a's peak here against 1.29x at 104M terms, so the
per-rank multiplier falls as the operator grows.

Review findings fixed

--bench-rounds 3 was a 2× memory error, not a statistics choice: pedantic holds round
k+1's propagator alongside round k's. Rounds is now per rung, 1 on the sized ones.
Placement was printed, never asserted, and from rank 0's fields rather than the *_min
reductions that exist to catch a partial collapse. check_shape.py now fails a rung when
partitions > 1 and single_cpu_threads_min == 0. It does not fire on an all-zero summary,
which means /proc was unreadable rather than nothing pinned.
Pinning was the caller's option. --map-by slot:PE=$monoprop_PARTITIONS --bind-to core moved
into bench-ci-mpi, which cannot now run unpinned and refuses without monoprop_PARTITIONS.
*ARGS is freed for the pytest args the rungs need.
cpuinfo was being recorded twice. machine.py, test_machine.py, the py-cpuinfo2
dependency and a project-wide [tool.uv] override-dependencies are gone: time-<label>.json's
machine_info.cpu, which pytest-benchmark already writes, is a superset of the nine fields
machine.py collected. The override also dropped py-cpuinfo from the whole project, changing
what produces pytest-benchmark's own machine_info — verified absent in a synced venv.
Comments cut to one line each; benchmarks.mdx's prose replaced by the tables above.

What is not measured

On this silicon, not the runner's. Every cell was measured on an 8-core mask of a Deucalion
x86 node (znver2, SMT off). Term counts are deterministic at --seed=0 and transfer exactly;
memory transfers well; the ~s column is indicative for Sapphire Rapids.

SMT is off on the calibration machine. PE=2 resolved against cores there, so each rank got a
2-CPU mask. On the runner (SMT on) --bind-to core should give each rank 2 cores / 4 hwthreads and
the engine 2 physical cores; the assertion holds either way, but the mask figure will differ.

L2b's 4 ranks is a judgement, not an optimum. The instance is one NUMA domain, so LADDER.md's
one-rank-per-domain rule picks no number. 4 × 2 makes L2b the memory-worst rung, which is the right
place to hang the ceiling, and keeps more than one partition per rank where 8 × 1 would not.

Nothing benchmarks until a maintainer sets vars.BENCH_BARE_METAL. bench_main.yml is deleted
and the README badge with it, rather than pointing at a workflow that never runs: a GitHub-hosted
runner has ~2 physical cores, making L2a 1 × 2 and L2b 2 × 1 — restatements of L1 through a noise
floor no threshold survives.

Verification

check	result
L1 / L2a / L2b at the flags and shapes above, on an 8-core mask	11/11 runs rc=0, job COMPLETED in 23:01
L1 term counts against LADDER.md's L1 table	exact: 9,953,109 / 10,069,308 / 19,902,244
terms identical at 1x8 and 4x2	167,515,463 both
`--obs-terms` bracketing	2.5M = 23.71 GiB/node fits; 3.5M = 35.49 does not
placement at 1x8 (`P` == physical cores)	8 of 11 threads on a CPU each, 8-CPU mask — the boundary does not collapse
placement at 4x2 under the CI `mpiexec` line verbatim	2 of 5 threads pinned, 2-CPU mask per rank — `PE=2` resolved against cores, not hwthreads
`mpiexec -n 4 --map-by slot:PE=2 --bind-to core` vs `srun --cpu-bind=cores`	identical shape and pinning
rung parser, 10 cases	real block, rung-after-MPI (fd 3), unbalanced quote, quotes-only, missing rounds, non-numeric rounds, oversubscribed, `per-rank`→0, no args, comments/blanks — all as intended; harness generated from `bench.yml` itself
`-k` expression through `just`	arrives as one argv element; `bench-ci-mpi` refuses without `monoprop_PARTITIONS` (rc=1)
`cpu_slug.py`	8/8 brand strings from the deleted test file; `AMD EPYC 7742 64-Core Processor` → `amd-epyc-7742-64-core` end-to-end
`uv lock --check` after dropping the override	rc=0; `py-cpuinfo` restored as pytest-benchmark's dep, `py-cpuinfo2` and `overrides` gone
`prek run --from-ref <base> --to-ref HEAD`	all 24 hooks pass, rc=0 — the gate `lint` runs
`check_shape.py` matrix	7/7: the three real results pass; wrong partitions, wrong ranks and a collapsed placement each fail; an all-zero pinning summary does not false-alarm
`pytest packages/monoprop-bench-tools/tests`	38 passed
all three workflow YAML files	parse

github-actions · 2026-09-01T11:52:16Z

Docs preview: https://pr-323.monoprop-docs.pages.dev

codecov · 2026-09-01T11:57:38Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.70%. Comparing base (d531dc6) to head (12db4e0).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #323   +/-   ##
=======================================
  Coverage   97.70%   97.70%           
=======================================
  Files          14       14           
  Lines         742      742           
  Branches       98       98           
=======================================
  Hits          725      725           
  Misses         12       12           
  Partials        5        5

Flag	Coverage Δ
cpp	`97.70% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🤖 _AI text below_ 🤖 `benches/LADDER.md` — *the benchmarking ladder for sensitive PRs* — is five groups of benchmark configurations at the sizes the library is actually used at. Each row gives the exact pytest flags and the `-k` selector that produce it, so a group is a handful of ordinary `just bench` invocations. Nothing runs these, and no benchmark gates a PR. ```bash just bench L1-hubbard-branch --hubbard-cutoff=10 --hubbard-lower-atol=4.2e-05 \ -k "test_model_propagate and hubbard" ``` | group | shape | what it is for | | --- | --- | --- | | **L1** — one thread | 1 rank, 1 partition | hubbard and pauli `propagate` at ~10M terms; random `gradient` at ~20M, paired with and without `--pare-threshold` | | **L2a** — one node, one rank, ~1B terms | 1 rank, all cores | all four operations over three models and both pictures | | **L2b** — one node of ranks, same problems | `N`=1, `R × P` = cores | MPI with no network in it | | **L3** — several nodes, same problems | L2b's `R` and `P`, your `N` | the same shape at `N` > 1 isolates the network | | **L4** — strong and weak scaling | your `R`, `P`; `N` = 1…64 | hubbard `propagate` on one size sequence; both ladders built from it | ## What the review asked for **Gradient rows at 20M terms, with and without `pare_threshold=1e-10`.** `--pare-threshold` did not exist; it is now a pytest option threaded into `expectation_value_functional` and `expectation_value_and_gradient_functional` for both the random and the fixed-model benchmarks. Unset reproduces today's behaviour exactly, so the Bencher series are unbroken. Measured at 1 rank / 1 partition / 1 thread (`_core.so` md5 `b201ec4`), `--obs-terms=295000` landing 19,902,244 terms: | row | ~s (2 reps) | ~GiB | | --- | ---: | ---: | | random `gradient` | 11.05, 11.02 | 2.5 | | random `gradient`, `--pare-threshold=1e-10` | 0.75, 0.75 | 2.8 | Hubbard is not a candidate — 29 successive `build_graph` calls retain 29 layer-sets and `skip_if_graph_will_not_fit` skips it — so the gradient rows use the random problem, whose size is directly dialable. Proved the knob reaches the engine rather than being swallowed: `1e-10` leaves the energy bit-identical (the point of a retention cutoff), `1e10` moves it 0.8049 → 0.8307. **L3's ratios removed.** The "against L2" column is gone; L3 reads like L2 and L1 — flags, `-k`, terms, `~s`, `~GiB/node` — with the shape those cells were measured at stated underneath (`N`=4, `R`=8, `P`=16, medians of two reps) rather than as a prescription. **L3 and L4 generalised, and the shape made mandatory.** `N`, `R` and `P` are the caller's throughout, and `benches/conftest.py` now **refuses to start a session of more than one rank with `monoprop_PARTITIONS` unset**. The engine's own default is `ranks == 1 ? cores : 1`, so an unset knob measured one partition per rank at a plausible wall time — a trap the document used to merely warn about. Every rank raises after the collective `_nodes()`, so nothing is left in a collective. One-rank runs — `just bench`, `bench-smoke`, `bench-ci`, every CI workflow — are untouched. **The shape recorded where the benchmarks record.** `meta` already carried `ranks`, `nodes`, `ranks_per_node` and `monoprop_threads`, with report columns for each — but `partitions_env` was written *only when the env var was set*, so an undeclared partition count and an unrecorded one both rendered `—`. It is now recorded unconditionally, `"unset"` when absent. ## Model sizes are inputs now | was | now | | --- | --- | | `--pauli-num-qubits` accepted any value while `HEAVY_HEX_TOPOLOGY` stayed the fixed 127-qubit IBM Eagle map — silently a different model, or an index past the operator | raises, naming the topology as the reason and pointing at `--pauli-lower-atol` for sizing | | `--hubbard-observable-site` above `--hubbard-num-sites` — a documented "trap" that produced a wrong observable | raises | | a mode count above the extension's compile-time `MAX_NUM_MODES` failed deep inside the extension | raises, naming the limit and the cmake define | | `--pauli-observable-qubit` outside the register | raises | `LADDER.md` gains a *Model size knobs* table covering these plus `--obs-terms`, which is an upper bound rather than an exact count: monomials are drawn independently and duplicates collapse, by about `obs_terms / 2·C(2·num_modes, gen_length)`. Checked against a direct count at 142 modes — 0.03/0.19/0.37% measured at 200k/1M/2M draws against 0.04/0.19/0.38% predicted — giving 0.06% at L1's 295k and 2.8% at L2's 14.75M. Deterministic for a fixed `--seed`, so the calibration reproduces. ## Also - `just bench-mpi` never forwarded `monoprop_PARTITIONS` or `monoprop_NUM_THREADS` through `mpiexec -x`, though its own doc comment showed `monoprop_NUM_THREADS=2 just bench-mpi …`. Both are forwarded now, guarded so an unset one is not an mpiexec error. ## Traps, each from a failure it caught during calibration - **`srun --cpu-bind=cores` with no `--cpus-per-task` on the `srun` confines each task to one core.** Measured in one allocation: no flags → 128, `--cpus-per-task=128` → 128, `--cpu-bind=cores` alone → 1. ~100x, with nothing in the timing to say so. - **The report runs outside `srun`.** Under it, one process per rank races to write the same file. The two JSON artifacts are safe: only rank 0 writes them. - **A two-operation row's peak is the MAX over its operations, never the sum.** `HighWaterMark` resets `VmHWM` per benchmark, so both windows contain the same resident operator. - `--cpu-bind=none` cost 1.45x; `nproc` lies inside a job. - The `monoprop_PARTITIONS` and observable-site traps are no longer traps — both now raise. ## The L2a / L2b split `L2` → `L3` moved two things at once — multiple processes *and* the network — so anything that appeared at L3 was attributable to neither. L2 splits: **L2a** is the old L2 unchanged (one rank, `partitions = threads = cores`), and **L2b** is L3's shape at `N`=1. Each edge of the ladder now moves one thing: partitions, processes, the network, then node count. L2b is measured at `N`=1, `R`=8, `P`=16 on a 128-core node, two reps with the cell order flipped (job 1862701, `_core.so` md5 `b201ec44`). Term counts reproduce L2a's and L3's **exactly** — 1,001,661,534 / 985,970,588 / 948,937,993 / 597,445,055 — which is the geometry-independence check the rung relies on. `8 × 16` is stated as a choice, not derived as an optimum: one rank per NUMA domain on this machine, and L3's shape, so `L2b` → `L3` differs only in the node count. The section says so rather than implying a sweep. Two findings recorded next to the table: - **The per-rank cost that is not a share of the operator is 3.6–8.1 GiB on the Heisenberg rows and 0.2–0.6 GiB on the Schrödinger ones**, on one shape, one node and one binary. The flag sets differ in the observable — 14.75M terms against 200k. The raw observable is well under a GiB, so something indexed by it scales with it; that is not attributed further here. - **L2b, not L2a, is the memory-worst rung for the Heisenberg rows**, because the per-rank cost is paid `R` times with no extra nodes to spread it over. An earlier draft of this change claimed a row fitting L2a fits L2b; the measurement refutes it and the claim is gone. `~GiB/node` is a sum over ranks, as L3's column is, so reading it against L2a's single-process peak would be a sum-against-max comparison — the error that produced a phantom 2.3× in this project before. The table says so. Wall time is *not* compared across the two rungs: the same 1×128 work measured 140 s in L2a's job and 176 s in another allocation, which is exactly why that comparison needs both arms in one job. Also in this push: a new trap — `P` above the cores a rank can see, which is what carrying L2a's `monoprop_PARTITIONS` into an L2b launch does — and the multi-rank skeleton now names the interpreter directly, because `uv run` re-resolves the environment in every rank. **Stacked on top:** #323 rebuilds the continuous benchmarking around these rungs. ## Verification - `uv run pytest`: **622 passed, 8 skipped** — unchanged from #316. - `prek run --all-files` over every changed file: clean. - The multi-rank gate, on a 2-rank job: unset → `ERROR: monoprop_PARTITIONS is unset on a run of 2 ranks…` from both ranks, exit 4; set to 8 → runs, and `{"ranks": 2, "nodes": 1, "ranks_per_node": 2, "partitions_env": "8", "monoprop_threads": "8"}` lands in `results/<label>.json` and in the report's Configuration table. One rank with the knob unset still runs. - Each rejection exercised through the CLI: `--pauli-num-qubits=100` and `--hubbard-observable-site=70` each fail with the message naming the constraint. - Every pytest flag in `LADDER.md` checked against `pytest benches --help`; all present. Every `-k` run through `--collect-only`; each selects exactly the intended node ids. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Aaron Miller <61472721+diagonal-hamiltonian@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

`monoprop_PARTITIONS` was unset everywhere in CI, so `resolve_partition_count_`'s `ranks == 1 ? cores : 1` decided the shape: every point in the `ubuntu-26.04` and `aws-c7i-4xlarge` series was one partition per core of whatever runner it landed on, unrecorded, and a runner resize would have moved every series without moving a testbed name. Nothing declared a problem either -- the lanes ran the suite's dataclass defaults. A rung line now carries both: `<name> <ranks> <partitions> <rounds> | <pytest args>`. L1 is benches/LADDER.md's own L1 rows; L2a and L2b share one problem sized to this instance rather than to the 128-core node the ladder tables were measured on, and differ only in shape. Measured on an 8-core mask of a Deucalion x86 node: L1 1x1 9.95M / 10.07M / 19.90M terms 2.60 GiB 3:53 at 3 rounds L2a 1x8 167,515,463 terms 19.11 GiB 135 s at 1 round L2b 4x2 167,515,463 terms 23.71 GiB/node 90 s at 1 round `--obs-terms=2500000` is the largest measured point that fits: 3500000 reaches 35.49 GiB/node against the instance's 32 GiB, and a fit over 1.5M and 2.5M under-called it by 3.9 GiB, so the axis is read off measured points rather than interpolated. The term count is identical at both shapes, which is the geometry-independence check. Rounds is per rung because `pedantic` builds round k+1's propagator before releasing round k's -- at these sizes that is a memory setting, not a statistics setting. Placement is asserted rather than printed. `check_shape.py` fails a rung whose recorded shape differs from the one asked for, or whose `single_cpu_threads_min` is zero while threads were counted; an all-zero summary means /proc was unreadable, not that nothing was pinned, so it does not fire there. Both L2 rungs sit exactly on the `partitions > visible cores` boundary, where the only warning goes to C++ stderr and pytest's capture eats it. `--map-by slot:PE=$monoprop_PARTITIONS --bind-to core` moves into `bench-ci-mpi`, which can no longer run unpinned and refuses without `monoprop_PARTITIONS`; `*ARGS` is freed for the pytest args the rungs need. The testbed's CPU is slugified from the cpuinfo block pytest-benchmark already writes into `time-<label>.json`. bench_main.yml goes, and the README badge with it rather than pointing at a workflow that is inert until a maintainer sets `vars.BENCH_BARE_METAL`: a GitHub-hosted runner has about two physical cores, so L2a would be 1x2 and L2b 2x1 -- restatements of L1 through a noise floor no threshold survives. Assisted-by: ClaudeCode:claude-opus-5

The continuous-benchmarking section described the lanes in prose and quoted no sizes, because none had been measured. It now carries three tables -- the shapes, L1's rows and the L2 rows -- and the prose that justified each decision is gone. `~s` and `~GiB` are measured on an 8-core mask of a Deucalion x86 node (znver2, SMT off), not on the runner. Term counts are deterministic at `--seed=0` and transfer exactly; memory transfers well; the times are indicative for Sapphire Rapids. The `bench_main.yml` references follow the file. Assisted-by: ClaudeCode:claude-opus-5

Co-authored-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com> Signed-off-by: Aaron Miller <61472721+diagonal-hamiltonian@users.noreply.github.com>

Co-authored-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com> Signed-off-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com>

robertodr · 2026-09-03T11:34:31Z

 [![Documentation](https://github.com/Algorithmiq/monoprop/actions/workflows/docpages.yml/badge.svg)](https://docs.monoprop.algorithmiq.tech/)
 [![Test monoprop](https://github.com/Algorithmiq/monoprop/actions/workflows/test.yml/badge.svg)](https://github.com/Algorithmiq/monoprop/actions/workflows/test.yml)
 [![codecov](https://codecov.io/gh/Algorithmiq/monoprop/graph/badge.svg?token=1GgmPnNUxk)](https://codecov.io/gh/Algorithmiq/monoprop)
-[![Track benchmarks](https://github.com/Algorithmiq/monoprop/actions/workflows/bench_main.yml/badge.svg)](https://github.com/Algorithmiq/monoprop/actions/workflows/bench_main.yml)


Why did this badge disappear?

robertodr

The new recipes in justfile are used (and useful) exclusively in the CI job. They can be stated inline in the YAML file for the github workflow. The bench-ci recipe predates this change, but also that one should go away from the justfile.

Copilot

Pull request overview

This PR refactors the continuous benchmarking CI to run explicit “ladder rungs” (shape + rounds + problem) on dedicated bare-metal runners, and to upload each rung as its own silicon- and shape-specific Bencher testbed so histories remain stable across runner resizes.

Changes:

Replace the old single-series benchmark workflow with a rung-driven harness that enforces rank/partition shapes and validates placement via post-run JSON checks.
Add CI helper scripts to derive a CPU-based testbed slug and to fail runs that didn’t execute at the requested shape/pinning.
Update documentation and examples to reflect the new rung format and bare-metal workflow behavior.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
README.md	Removes the benchmark workflow badge now that `bench_main.yml` is deleted.
pyproject.toml	Excludes workflow helper scripts from Ruff (CI script linting scope change).
packages/monoprop-bench-tools/README.md	Updates BMF example to use the new bare-metal label naming.
justfile	Adds MPI benchmark runner, rung parsing/execution, core discovery, and Bencher upload per rung.
docs/content/docs/benchmarks.mdx	Updates continuous benchmarking docs to describe rung shapes, testbed naming, and measured tables.
.github/workflows/scripts/cpu_slug.py	Introduces CPU brand slugging to build Bencher testbed names.
.github/workflows/scripts/check_shape.py	Introduces shape/pinning verification against benchmark JSON metadata.
.github/workflows/bench.yml	Converts reusable workflow inputs from “markers/rounds/testbed” to “rungs”, runs rung harness, and uploads per-rung artifacts.
.github/workflows/bench_main.yml	Deletes the GitHub-hosted benchmark baseline workflow.
.github/workflows/bench_bare_metal.yml	Switches bare-metal benchmarking to rung-driven invocation with explicit L1/L2a/L2b definitions.
.github/runs-on.yml	Clarifies runner pinning rationale relative to testbed naming.

Suppressed comments (1)

justfile:468

bench-ci-rungs validates rounds but not ranks / partitions before doing arithmetic. A typo (e.g. missing ranks, non-numeric partitions) can silently coerce to 0 in bash arithmetic or crash later, leading to measuring the wrong shape.

      IFS=$' \t' read -r rung ranks partitions rounds <<<"$head"
      case "${rung:-}" in '' | '#'*) continue ;; esac
      # A missing field would otherwise fall through to a default and measure something else.
      case "${rounds:-}" in '' | *[!0-9]*) echo "::error::rung $rung has no round count"; exit 1 ;; esac

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+def main() -> str | None:
+    paths = sorted(Path("benches/results").glob("time-*.json"))
+    if not paths:
+        return "::error::no timing artifact to read the CPU from"
+    brand = json.loads(paths[0].read_text())["machine_info"]["cpu"]["brand_raw"]
+    print(slugify(str(brand)))
+    return None


+    print("affinity", len(os.sched_getaffinity(0)))
+    with open(os.environ["GITHUB_ENV"], "a") as env:
+        print(f"BENCH_CORES={cores}", file=env)


+            f"::error::{path.stem} placed no thread on a CPU of its own. "
+            "partitions above the rank's visible cores returns an empty placement order."
+        )


robertodr · 2026-09-04T07:43:10Z

Also, I still maintain that using py-cpuinfo2 would give more complete information on how the architecture you're running on

sonarqubecloud · 2026-09-04T11:14:46Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

diagonal-hamiltonian requested review from fpietra, ludmilaasb and robertodr as code owners September 1, 2026 11:48

github-actions Bot added documentation Improvements or additions to documentation ci labels Sep 1, 2026

diagonal-hamiltonian mentioned this pull request Sep 1, 2026

docs(benches): 🏗️ benchmarking ladder for sensitive PRs #317

Merged

diagonal-hamiltonian force-pushed the ci/bench-rungs branch from 8114981 to ef0947d Compare September 1, 2026 12:32

github-actions Bot added the dependencies Pull requests that update a dependency file label Sep 1, 2026

diagonal-hamiltonian changed the title ~~chore(ci): 👷 benchmark the ladder rungs, one testbed per shape~~ chore(ci): 👷 benchmark the ladder rungs, one testbed per shape and silicon Sep 1, 2026

diagonal-hamiltonian force-pushed the ci/bench-rungs branch 3 times, most recently from 979d117 to cf137bb Compare September 1, 2026 17:05

diagonal-hamiltonian force-pushed the ci/bench-rungs branch from cf137bb to 8aca5cf Compare September 2, 2026 10:39

Base automatically changed from chore/bench-rung-table to main September 2, 2026 12:09

diagonal-hamiltonian force-pushed the ci/bench-rungs branch from 8aca5cf to 090c214 Compare September 2, 2026 12:13

robertodr force-pushed the ci/bench-rungs branch from 090c214 to 060ab0b Compare September 3, 2026 09:58

diagonal-hamiltonian added 2 commits September 3, 2026 12:33

robertodr force-pushed the ci/bench-rungs branch from 060ab0b to c444ec7 Compare September 3, 2026 10:33

robertodr reviewed Sep 3, 2026

View reviewed changes

Comment thread .github/workflows/bench.yml Outdated

robertodr reviewed Sep 3, 2026

View reviewed changes

Comment thread .github/workflows/bench.yml Outdated

robertodr reviewed Sep 3, 2026

View reviewed changes

Comment thread .github/workflows/bench_bare_metal.yml Outdated

robertodr reviewed Sep 3, 2026

View reviewed changes

Comment thread .github/workflows/bench_bare_metal.yml Outdated

robertodr reviewed Sep 3, 2026

View reviewed changes

Comment thread docs/content/docs/benchmarks.mdx Outdated

robertodr reviewed Sep 3, 2026

View reviewed changes

Comment thread docs/content/docs/benchmarks.mdx Outdated

robertodr reviewed Sep 3, 2026

View reviewed changes

Comment thread pyproject.toml Outdated

diagonal-hamiltonian and others added 2 commits September 3, 2026 12:28

Update benchmarks.mdx

a05033b

Co-authored-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com> Signed-off-by: Aaron Miller <61472721+diagonal-hamiltonian@users.noreply.github.com>

Update pyproject.toml

add75a3

Co-authored-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com> Signed-off-by: Aaron Miller <61472721+diagonal-hamiltonian@users.noreply.github.com>

diagonal-hamiltonian requested a lite review from Copilot September 3, 2026 11:29

Copilot started reviewing on behalf of diagonal-hamiltonian September 3, 2026 11:29 View session

Apply batched suggestions from code review

8d1747f

Co-authored-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com> Signed-off-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com>

robertodr reviewed Sep 3, 2026

View reviewed changes

robertodr requested changes Sep 3, 2026

View reviewed changes

Copilot AI reviewed Sep 3, 2026

View reviewed changes

Merge branch 'main' into ci/bench-rungs

12db4e0

Navigation Menu

Uh oh!

chore(ci): 👷 benchmark the ladder rungs, one testbed per shape and silicon - #323

chore(ci): 👷 benchmark the ladder rungs, one testbed per shape and silicon#323
diagonal-hamiltonian wants to merge 6 commits into
mainfrom
ci/bench-rungs

diagonal-hamiltonian commented Sep 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

codecov Bot commented Sep 1, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robertodr Sep 3, 2026

Uh oh!

robertodr left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

robertodr commented Sep 4, 2026

Uh oh!

sonarqubecloud Bot commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

diagonal-hamiltonian commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was wrong

What it does now

L1 — LADDER.md's own rows, one thread

L2a and L2b — one operator, three operations

Review findings fixed

What is not measured

Verification

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

codecov Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robertodr Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

robertodr left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

robertodr commented Sep 4, 2026

Uh oh!

sonarqubecloud Bot commented Sep 4, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

diagonal-hamiltonian commented Sep 1, 2026 •

edited

Loading

codecov Bot commented Sep 1, 2026 •

edited

Loading