Skip to content

feat(plan): saturated pools cut to measured parallelism (#14 phase 2) - #18

Merged
sylvesterdamgaard merged 3 commits into
mainfrom
feat/saturation-fill
Sep 10, 2026
Merged

sylvesterdamgaard merged 3 commits into
mainfrom
feat/saturation-fill

Conversation

@sylvesterdamgaard

@sylvesterdamgaard sylvesterdamgaard commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Completes #14. Phase 1 (#17) covered the histogram's blind spots with an aggregate CPU shape - but only in the report; cpuCeilingFor, the ceiling the allocator uses, still required per-request readings and stayed 0 for sub-50 ms workloads.

The design (after a live lesson): aggregateFill is the ONE place that reads the tick-delta aggregate, with a regime rule:

  • Calm host → the share (cores per busy worker) is honest; fill follows from per-worker cost, like the per-request path.
  • CPU-saturated host → the share is poisoned (every queued worker reads 'busy': 25 workers on 2 cores read share 5%, fill 40, ceiling 80) and fill IS the measured cores - tick deltas cannot be inflated by workers that only wait. ceil(1.9 cores) × headroom 2.0 = 4.

Both the allocator and the report call it, so every kind of round produces the same number. The first cut of this PR had the cores rule only in the starved-gate block and the share fallback in cpuCeilingFor - live, those two flapped against each other every other round (queue drained for one scrape → ceiling 80 → regrow toward memory's 30 → starved round → cut to 4 → SIGUSR2 reload per flip), collapsing throughput from 391 to ~110 rps. The flap round is now a regression test.

Safety:

  • Cut gated on --cpu AND the Trusted baseline (a cap below the configured ceiling IS a cut; cuts are opt-in). Without the flag: phase-1 hold, report shows what --cpu would do (SaturationMeasured).
  • A pool only claims the cores reading when it drives ≥ half the box: an io-shaped pool on a host made busy by a neighbor falls through to the share, whose fill is large and non-binding, exactly as io should be (regression-tested).
  • Aggregate not yet trusted (< MinAggCPURounds): hold, never cut on an unproven signal.

Live evidence (2 CPUs, 16→24-worker static pool, sub-3ms CPU flood, embedded in cbox-init):

phase workers rps
pre-trust plateau (30 min) 24 ~305
cut fires 4 368 → 380 → 391 (+28%)
(first cut of the PR: flap) 4↔25 collapsed to ~110 - the bug this revision removes

A fresh 45-minute run with the regime rule is queued; expect the cut to hold at 4 with ~390 rps to the end - numbers land as a comment.

Tests: oversized-starved pool converges to 4 (CPUBound); flap round holds at 4; io-neighbor not capped; untrusted aggregate holds; no-flag holds; saturated cpuCeilingFor returns 4 where the poisoned share said 80. Full gate green.

…ts the aggregate fallback (#14 phase 2)

Phase 1 (#17) gave the REPORT an aggregate CPU shape, but the ceiling the
allocator actually uses (cpuCeilingFor) still required per-request
readings - for sub-50ms workloads it stayed 0 and only the report told
the truth. It now falls back to the aggregate share under the same
Trusted gate.

And the saturated case gets the honest fill: under saturation every
worker reads 'busy' while queuing for a core, so the share's denominator
absorbs the oversubscription factor and share-based fill circles back to
the current size, whatever it is. The kernel's tick deltas cannot be
inflated by workers that only wait - the cores the pool actually drives
ARE its parallelism. A starved pool with a trusted baseline and a trusted
aggregate is now cut to ceil(measured cores) x headroom instead of merely
held: 16 workers saturating 2 cores converge to 4, where the worker-count
sweep measured the throughput optimum (~+30% over 10-20 workers on
CPU-bound endpoints).

Gated like every other cut: on the pool's Trusted baseline AND on --cpu -
a cap below the configured ceiling IS a cut, and cuts are opt-in. Without
the flag the starved pool is held exactly as in phase 1, and the report
row (SaturationMeasured) still shows what --cpu would do.
The first cut of phase 2 put the cores-based fill in the starved-gate
block and a share-based fallback in cpuCeilingFor. Live, on a 2-core
box with an oversized static pool, those two disagreed every other
round: starved rounds said ceiling 4 (honest cores), queue-drained
rounds said ceiling 80 (the share's busy denominator counts queued
workers, so 25 workers on 2 cores read share 5% and fill 40) - and the
pool flapped 4 -> 25 -> 4 with a SIGUSR2 reload per flip, collapsing
throughput from 391 to ~110 rps.

aggregateFill is now the ONE place that chooses: calm host -> the share
is honest, fill from per-worker cost; saturated host -> fill IS the
measured cores (tick deltas cannot be inflated by workers that only
wait). Both the allocator (cpuCeilingFor) and the report (cpuOf) call
it, so every kind of round produces the same number and the plan cannot
flap. The starved-gate block goes back to phase-1 hold-only - the cut
now arrives through the ceiling, in one place, still gated on --cpu and
the Trusted baseline.

A pool only claims the cores reading when it drives >= half the box:
an io-shaped pool on a host made busy by a NEIGHBOR falls through to
the share, whose fill is large and non-binding, exactly as io should
be. Regression tests: the queue-drained flap round holds at 4; the
io-neighbor pool is not CPU-capped; saturated cpuCeilingFor returns 4
where the poisoned share said 80.
@sylvesterdamgaard

Copy link
Copy Markdown
Contributor Author

Live proof, third and final run (2 CPUs / 1 GiB, 16→24-worker static pool, sub-3ms CPU flood via wrk -t2 -c32, embedded in cbox-init, 60s measurement windows, quiet host):

windows workers rps
1-30 (pre-trust plateau) 24 ~318
31 (trust lands, cut fires) 24 → 4 338
32-46 (steady state) 4 392-398, fifteen consecutive windows

One Pool resized event in the entire run. +25% throughput.

The three runs tell the whole story:

  1. Run 1 (first cut of this PR): the cut fired (+28%) but the share-based ceiling in the queue-drained rounds said 80 while the starved rounds said 4 → flap 4↔25 with a SIGUSR2 reload per flip, collapsing throughput to ~110 rps. Became the flap-round regression test.
  2. Run 2 (regime rule, 0.5× threshold): the cut never fired - measured cores DEFLATE under host contention (a co-located build left the container ~1.15 effective cores of its 2-CPU quota, a hair under the threshold). Became the 0.35× calibration + the agg_cores/agg_busy/saturation_measured log instrumentation, so this is never undiagnosable again.
  3. Run 3 (this revision): cut at the trust point, stable to the end.

Worker-sweep cross-check: the sweep measured 396-400 rps at 2-4 workers and ~305 at 20+ on this endpoint - the tuner now lands exactly on the measured optimum, from live evidence only.

… instrument the aggregate

Run 2 of the live proof never cut: a co-located build left the container
~1.15 effective cores of its 2-CPU quota, a hair under the 0.5x-cores
threshold, and nothing in the logs could say so. Three changes:

- Threshold 0.5x -> 0.35x of the box's cores. A genuinely CPU-bound
  pool's measured cores DEFLATE under host contention; an io pool's stay
  far below either number. Run 3 with this calibration: cut at the trust
  point, 24 -> 4 workers, +25% throughput, fifteen consecutive stable
  windows, ONE resize event in 46 minutes.
- The 'Pool bound by CPU' log line now carries agg_cores, agg_busy and
  saturation_measured, so the regime rule's choice is visible on a live
  system instead of undiagnosable (PoolCPU.AggCores/AggBusy).
- testing/mutations.py: the two ceiling-guard patterns follow the
  refactored code shape (they were NO MATCH, failing CI), and the
  saturated-cores branch gets its own mutation - removing it kills the
  flap-round regression test, which is exactly the guard-test relation
  the sweep exists to enforce.
@sylvesterdamgaard
sylvesterdamgaard merged commit 472e2e1 into main Sep 10, 2026
10 checks passed
@sylvesterdamgaard
sylvesterdamgaard deleted the feat/saturation-fill branch September 10, 2026 06:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant