test: verify all algorithms match their authors' proposals (3-layer fidelity)#12
Open
blankjul wants to merge 2 commits into
Open
test: verify all algorithms match their authors' proposals (3-layer fidelity)#12blankjul wants to merge 2 commits into
blankjul wants to merge 2 commits into
Conversation
…)EI acquisition Performance tests only show each algorithm beats a baseline; they cannot tell a faithful implementation from a lucky-but-wrong one. Four of the new algorithms had no fidelity check. Close the gap by exposing each one's paper-defining computation as a small static seam (mirroring EHVI.expected_hvi / CSEA.label_good) and asserting the defining property: - MOEA/D-EGO: lcb() is the optimistic mu - kappa*sigma; select_by_decomposition() picks the augmented-Tchebycheff minimizer per weight (Zhang et al., 2010). - TSEMO: thompson_sample() is mu + sigma*z (mean at zero variance, unbiased); greedy_hvi_select() is hypervolume-monotone (Bradford et al., 2018). - K-RVEA: select_infills() takes the most-uncertain points in the convergence regime and one-per-active-vector (topped up to a full batch) in the diversity regime, switching on the active-set change (Chugh et al., 2018). - SAASBO: the surrogate is an ARD Kriging carrying the shrinkage theta_prior (the sparse-axis mechanism; Eriksson & Jankowiak, 2021). - (Log)EI: EI.calc is the analytic closed form; LogEI equals log(EI) where EI is finite, shares its argmax, and stays finite where EI underflows to 0 -- the acquisition ParEGO/TuRBO/SAASBO all optimize (Ament et al., 2023). The seams are pure behavior-preserving extractions: all reproducibility tests stay bit-identical, so the seed-1 golden scores are unchanged. Fidelity suite grows from 8 to 20 tests; every new algorithm now has a defining-property check.
…se 3) Add tests/test_reproduction.py (slow-marked): each new algorithm is run over three seeds on the kind of problem its paper targets and asserted to reach a sensible metric level. This is the coarsest fidelity tier -- deliberately a ballpark check, not exact table reproduction, since pysamoo's versions are simplifications (MAP SAASBO, Monte-Carlo EHVI, small budgets). It catches a regression that leaves an algorithm running but no longer optimizing, which a single-seed beats-a-baseline test can miss. Robust statistic per algorithm, chosen from calibration: - the six low-variance algorithms are checked on the median over the seeds; - TuRBO-1 and the SAASBO scaffold are variance-prone across seeds (calibration showed TuRBO spanning 4.5..19.9 on Ackley), so they are checked on the best of the seeds -- asserting every seed converges would test noise, not correctness. Bands are ~2-3x the worst calibration value, robust to platform jitter but failing hard if convergence breaks. Validated: 8 passed in ~3.5 min.
blankjul
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
How we now check every algorithm is implemented as the authors proposed — a layered validation strategy, since "beats a baseline" (
test_performance.py) can't distinguish a faithful implementation from a lucky-but-wrong one.Layer 1 — paper-defining-property tests (ground truth)
Each algorithm's defining computation is exposed as a small static seam (like the existing
EHVI.expected_hvi/CSEA.label_good) and asserted directly:mu-kappa*sigmaoptimistic; decomposition picks the augmented-Tchebycheff minimizer per weightmu+sigma*z(unbiased); greedy selection hypervolume-monotonetheta_prior(the sparse-axis mechanism)EI=analytic closed form;LogEI=log(EI)where finite, same argmax, stays finite under underflow(EHVI, ParEGO, CSEA, TuRBO already had these.) Seams are behavior-preserving — reproducibility stays bit-identical, golden unchanged. Suite grew 8→20 tests.
Layer 2 — reference cross-check → delivered as ground truth instead
BoTorch was considered but rejected: EHVI is already checked against exact grid quadrature and LogEI against the exact analytic EI — stronger oracles than another implementation, and dependency-free (no torch in CI). TuRBO/SAASBO have no well-defined fixed-input comparison.
Layer 3 — convergence-sanity on each paper's problem (
test_reproduction.py, slow)Each algorithm runs over 3 seeds on its paper's problem class and must reach a sensible metric. Framed honestly as a ballpark check, not exact table reproduction (our versions are simplifications). Low-variance algorithms use the median; TuRBO-1 and the SAASBO scaffold are variance-prone (calibration showed TuRBO 4.5..19.9), so they use best-of-seeds. Bands ~2-3x calibration. 8 passed in ~3.5 min.
Every algorithm now has ground-truth + convergence coverage.
pyclawd checkgreen.🤖 Generated with Claude Code