feat(benchmark): measure graphics changes with $bench

[sven-n]

Summary

Adds a segmented benchmark to the client, so a graphics change can be judged against a measurement instead of an impression.

$bench run measures a list of named segments, each on its own, and writes a run directory under bench/runs/. $bench run fx.* or #particles runs a subset for faster iteration; $bench quick is one short repeat for finding out whether a change did anything at all. Usage is documented in docs/benchmark.md.

Why segments. Splitting a run is what lets the result say where a change had an effect, rather than only whether the frame got faster. This first catalog varies the effect surfaces of whatever scene you are standing in, using the $effects toggles that already exist, so it needs no scripted content:

Segment	Isolates
scene.full	Nothing disabled — the reference, and the drift control
fx.all.off	Every effect surface at once, exactly as $effects off does it
fx.sprites.off / fx.particles.off / fx.joints.off	The three IR per-quad paths
fx.skillmodels.off	Skill effect models, impact effects still rendering
fx.boids.off	Ambient wildlife — an always-on town cost
fx.wingshadow.off / fx.winglayers.off	The extra wing shadow draw, the glow overlay passes

What it exports. run.json is canonical and schema-versioned; report.md, frames.csv and passes.csv are views rendered from it at fixed precision so two reports diff cleanly. The report header carries the build configuration, GPU and driver, the GL context version the chain actually settled on, every RHI::Caps flag, the resolution and the vsync state — the facts that decide which code path is running, which have so far had to be read out of a boot log by hand.

What it does to avoid lying.

• A repeat is a fixed frame count, never a fixed duration — a faster build has to render the same frames, not more of them.
• Each repeat discards a warmup window (shader compiles, texture uploads, streaming-ring growth). A frame over half a second discards its whole repeat rather than averaging an alt-tab in.
• The baseline is measured first and again last whatever the selection was; the difference is drift, and if it is large the report says nothing measured in between is trustworthy.
• The summary carries the 1% low, the percentile curve, the pacing metrics and the repeat spread — the noise band a delta has to clear before it means anything. This is the number GLP-29 did not have.
• Findings are rule-based and named, never prose. Two matter most: the attribution check (pass CPU ms against frame time — the automatic version of the blind spot GLP-24 found after it had already invalidated a phase of measurements), and segment-inert (a segment that submitted the same draw calls as the baseline disabled nothing here, so its timing is not evidence that what it targets is cheap).

Two supporting changes. $glstats used FrameProfiler::g_CountersEnabled as both "count things" and "draw the overlay", so a run turning the counters on would have switched a several-hundred-draw-call text overlay on inside its own measurement — the overlay now has its own flag, behaviour otherwise unchanged. The compile-time build info the $details overlay printed inline moved to Core/Utilities/BuildInfo.h, since the exports need the same facts.

Not in this PR, and worth doing next: scripted scenes (spawned characters, camera paths, weather), per-segment screenshots with an A/B image diff, and a benchdiff tool that compares two runs with a compatibility gate on the manifest hash.

Related issues

Verification

• Run in the client, on an RX 6800 in Lorencia, and that run found a real bug (#issuecomment-5461948274): fx.all.off set only SetDisableEffects and not the options switch $effects off also flips, so it left sprites, particles and joints rendering — 162 particle draws/frame against the baseline's 171. Fixed in 22bb352, with two tests covering it.
• The same run confirms the rest of the mechanism works: fx.particles.off took the Particles pass from 170.6 draws / 0.61 ms to 0.00 across 900 frames, and fx.joints.off reached 99.4 FPS against the baseline's 92.5.
• That run also had vsync on, which pinned every median between 9.89 and 10.63 ms and made the segments look alike. The report flagged it afterwards; it is now said in chat as the run starts, while there is still time to $vsync off. A clean uncapped re-run is still outstanding.
• 36 doctest cases / 167 assertions over the pure layer (stats, glob selection, findings rules, and the exact bytes of all four exports), passing in CI on Linux and MinGW.
• Full client build green on all six CI jobs: Linux and MinGW x86_64/i686, editor on and off.

Checklist

• I have read and followed docs/CODING_RULES.md.
• I have built the project locally (see docs/build/README.md) — full builds via CI on all six configurations; unit tests and per-file syntax checks locally.
• Changes are focused on one concern; the diff is as small as it reasonably can be.

---

Generated by Claude Code

[sven-n]

report.md

It does not seem to work correctly. It runs through, but it doesn't deactivate any effects and the results of each test looks similar.

[sven-n]

Thanks — the report had everything needed. It turned out to be two separate things, and one of them is a real bug.

fx.all.off was broken. $effects off is two actions:

SetDisableEffects(true);
if (g_pOption) g_pOption->SetRenderAllEffects(false);

My segment only did the first. RenderSprites() and RenderParticles() early-return on the options switch and never look at SetDisableEffects, and RenderJoints() looks at neither — so the one segment that promises "everything off" left the three heaviest paths running. Your own numbers show it:

Pass	scene.full	fx.all.off	fx.particles.off
Particles draws	170.6	162.4	0.0
Sprites draws	33.8	34.0	33.8
Joints draws	3.2	4.1	2.5

SceneConfig now carries the options switch too, and fx.all.off closes all three gates.

The per-surface segments were working the whole time. fx.particles.off took the Particles pass from 170.6 draws / 0.61 ms to 0.00 across 900 frames, and fx.joints.off is the best row in your run — 99.4 FPS against the baseline's 92.5. So the mechanism is sound; it was that one segment that lied.

What made every row look alike is vsync. The report has Vsync requested / effective: yes / yes, and every median sits between 9.89 and 10.63 ms — pinned at the refresh interval. Under that ceiling a genuine 0.6 ms saving reads as a rounding error. The report warned about it afterwards, which is precisely when the warning is useless, so it is now said in chat as the run starts, while there is still time to stop and $vsync off. Worth re-running that way — I'd expect fx.particles.off and fx.joints.off to separate clearly.

The deeper problem is that a broken segment produced a plausible-looking row instead of an obvious failure. New segment-inert rule: it compares each segment's draw calls per frame against the baseline and says so when nothing measurably changed. It would have flagged fx.all.off in your run, and it also catches the honest version of the same trap — fx.wingshadow.off with no wings equipped, whose 95.1 FPS is otherwise easy to read as "wing shadows are cheap" when the segment simply had nothing to disable.

Two tests cover the defect directly: every non-baseline segment must disable something, and fx.all.off must close all three gates.

Pushed as 22bb352.

---

Generated by Claude Code