Repository navigation
test: guard rails for changing the new-install default (T1-T4 of #1071), no default change - #1081
Merged
Merged
Conversation
…, no default change T1 pins that Candidate A (strategy index + the two-tool locate profile) is exactly what the study's locate arm writes, and what sigmap --init proposes today on both of its paths, so a flip shows up as a changed pin. T2 pins that four kinds of existing project resolve to the same effective config and tool profile as at v8.87.0 (recorded from the tag; develop already matches), and that --init never rewrites a config file. T3 pins the generated AGENTS.md and CLAUDE.md within 32 KiB and 200 lines under today's config and Candidate A; clones are skipped and said so, never counted as a pass. T4 is a cold-start timing tool whose verdict is an interval: within, slower, or inconclusive. Nothing here changes a default, a config or shipped behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Guard rails T1 to T4 of #1071, so the pull request that changes what a new install is proposed has to meet them or say why not. Nothing changes: no default, no config, no shipped behaviour. Refs #1071 (stays open: the maintainer's decision on Candidate A is still to be made).
test/integration/new-install-default-guard.test.js: T1 and T2 (withtest/fixtures/upgrade-invariance/).test/integration/instruction-footprint.test.js+scripts/check-instruction-footprint.mjs: T3.scripts/bench-cold-start.mjs+test/integration/bench-cold-start.test.js: T4, a measuring tool.version.json: tests 293 -> 296.What each one pins, and what it found
T1, config equivalence. Candidate A is
strategy: "index"+mcp.tools: "locate". Pinned: the study'slocatearm (prepared by the realprepareArm) writes exactly that and nothing else; the profile advertises exactlyquery_contextandget_lines; against an empty config file Candidate A differs in those two keys only. Also pinned: whatsigmap --initproposes today (full+ the 7-toolcoreprofile), on both its paths (the example file, and the fallback a lone bundle uses), so a flip shows up as a changed pin. The last check (T1c, a flipped--initequals the study's arm) is reported as a skip, not a pass, until--initproposeslocate.T2, upgrade invariance. Four kinds of existing project (no config, minimal, full custom, legacy
outputs) resolve to the same effective config and tool profile as at v8.87.0; the baseline was recorded by running the same fixture code against the tag.developalready equals the tag on all four, so the 14 commits since changed nothing for an existing install. Also pinned: a key a project sets keeps its value, and--initnever rewrites a config file ("already exists, skipping"). A premise test runsbuild-bundle.mjs --check, because these tests readsrc/while the CLI a user runs is the bundle.T3, footprint. Generated AGENTS.md at most 32 KiB and CLAUDE.md at most 200 lines, today's config and Candidate A (measured on a
git archivecopy, so no clone is written to). Run with--require-reposon this machine:All within the registered limits. Two things to read with it. The #995 caps do bind (without them gin writes 50 KB and 1,138 lines of CLAUDE.md), so the Codex and Claude files sit right under their limits. And the file a default install actually writes is the Copilot file, which has no cap and no registered limit: 32 to 101 KB today against 2 to 3 KB under Candidate A. In CI only SigMap's own repository is measured; gin, flask and django are reported as skips when their clones are absent, never as passes.
T4, cold start (a measuring tool, not a CI gate). Times
sigmapthensigmap askfrom a freshgit archivecopy, arms interleaved, 2 warm-up rounds discarded, verdict from a 95% interval of the paired ratios: within, slower, or inconclusive when the interval spans the 10% bar. It refuses a run that did not answer. Result on this laptop, 15 rounds, v8.87.0 againstdevelopand against the Candidate A config:developvs v8.87.0No arm was slower. Two of the four comparisons are within the bar and two are inconclusive (upper ends +10.5% and +14.6%), so T4 is not conclusively met: that takes more rounds on an idle machine. CPU time does not see waiting, only computing.
Not covered, said plainly
node(notnpx), on a warm file cache, with two repositories (the larger has 2,985 files).--initexample path and its lone-bundle fallback propose differentoutputs(four files against the copilot file only), although the fallback's comment says--initmeans the same wherever it runs. That is a divergence to settle in the flip.Test plan
new-install-default-guard: 10 pass, 1 explicit skip.instruction-footprint: 5 pass, 3 skip (clones absent), 8 pass with clones.bench-cold-start: 11 pass.outputsmapping, the locate profile and the study arm,--initoverwriting or flipping on either path, the baseline file, the footprint limits and the product's cap, the 10% bar, the median, and the "did not answer" refusals). The first round caught 18 of 21 and found one real gap in my own test (a failingaskwas never exercised, only a failing generate), one stale mutation pattern, and one mutant aimed at a source file the CLI never reads (it runs the bundle), which is why the premise test exists.node test/integration/all.jsand CI.🤖 Generated with Claude Code