Skip to content

test: guard rails for changing the new-install default (T1-T4 of #1071), no default change - #1081

Merged
manojmallick merged 1 commit into
developfrom
feat/default-guard-rails-1071
Nov 21, 2026
Merged

manojmallick merged 1 commit into
developfrom
feat/default-guard-rails-1071

Conversation

@manojmallick

Copy link
Copy Markdown
Owner

Summary

Guard rails T1 to T4 of #1071, so the pull request that changes what a new install is proposed has to meet them or say why not. Nothing changes: no default, no config, no shipped behaviour. Refs #1071 (stays open: the maintainer's decision on Candidate A is still to be made).

  • test/integration/new-install-default-guard.test.js: T1 and T2 (with test/fixtures/upgrade-invariance/).
  • test/integration/instruction-footprint.test.js + scripts/check-instruction-footprint.mjs: T3.
  • scripts/bench-cold-start.mjs + test/integration/bench-cold-start.test.js: T4, a measuring tool.
  • version.json: tests 293 -> 296.

What each one pins, and what it found

T1, config equivalence. Candidate A is strategy: "index" + mcp.tools: "locate". Pinned: the study's locate arm (prepared by the real prepareArm) writes exactly that and nothing else; the profile advertises exactly query_context and get_lines; against an empty config file Candidate A differs in those two keys only. Also pinned: what sigmap --init proposes today (full + the 7-tool core profile), on both its paths (the example file, and the fallback a lone bundle uses), so a flip shows up as a changed pin. The last check (T1c, a flipped --init equals the study's arm) is reported as a skip, not a pass, until --init proposes locate.

T2, upgrade invariance. Four kinds of existing project (no config, minimal, full custom, legacy outputs) resolve to the same effective config and tool profile as at v8.87.0; the baseline was recorded by running the same fixture code against the tag. develop already equals the tag on all four, so the 14 commits since changed nothing for an existing install. Also pinned: a key a project sets keeps its value, and --init never rewrites a config file ("already exists, skipping"). A premise test runs build-bundle.mjs --check, because these tests read src/ while the CLI a user runs is the bundle.

T3, footprint. Generated AGENTS.md at most 32 KiB and CLAUDE.md at most 200 lines, today's config and Candidate A (measured on a git archive copy, so no clone is written to). Run with --require-repos on this machine:

repo config AGENTS.md CLAUDE.md Copilot file
gin today 32,476 B 194 lines 50,492 B
gin Candidate A 2,647 B 82 2,842 B
flask today 32,354 B 197 32,570 B
flask Candidate A 2,405 B 89 2,621 B
django today 32,539 B 196 101,303 B
django Candidate A 2,070 B 67 2,286 B
SigMap today 32,264 B 197 96,935 B
SigMap Candidate A 2,233 B 62 2,449 B

All within the registered limits. Two things to read with it. The #995 caps do bind (without them gin writes 50 KB and 1,138 lines of CLAUDE.md), so the Codex and Claude files sit right under their limits. And the file a default install actually writes is the Copilot file, which has no cap and no registered limit: 32 to 101 KB today against 2 to 3 KB under Candidate A. In CI only SigMap's own repository is measured; gin, flask and django are reported as skips when their clones are absent, never as passes.

T4, cold start (a measuring tool, not a CI gate). Times sigmap then sigmap ask from a fresh git archive copy, arms interleaved, 2 warm-up rounds discarded, verdict from a 95% interval of the paired ratios: within, slower, or inconclusive when the interval spans the 10% bar. It refuses a run that did not answer. Result on this laptop, 15 rounds, v8.87.0 against develop and against the Candidate A config:

  • Wall-clock cannot resolve a 10% bar here: identical configurations ranged from 2.2 to 29.9 s on a 130-file repository, and every comparison was inconclusive. The tool says so rather than printing a number.
  • CPU time (user + system) is far tighter (gin 1.6 to 2.9 s):
develop vs v8.87.0 Candidate A vs v8.87.0
gin (130 files) +2.0% [-11.2, +10.5] inconclusive -19.6% [-23.5, -13.6] within the bar
fastapi (2,985 files) -1.0% [-19.3, +0.5] within the bar -13.2% [-19.6, +14.6] inconclusive

No arm was slower. Two of the four comparisons are within the bar and two are inconclusive (upper ends +10.5% and +14.6%), so T4 is not conclusively met: that takes more rounds on an idle machine. CPU time does not see waiting, only computing.

Not covered, said plainly

  • T1c waits for a flip. T3 on gin, flask and django does not run in CI. T4 is a manual measurement on one laptop, through node (not npx), on a warm file cache, with two repositories (the larger has 2,985 files).
  • Not changed, noticed: the --init example path and its lone-bundle fallback propose different outputs (four files against the copilot file only), although the fallback's comment says --init means the same wherever it runs. That is a divergence to settle in the flip.

Test plan

  • new-install-default-guard: 10 pass, 1 explicit skip. instruction-footprint: 5 pass, 3 skip (clones absent), 8 pass with clones. bench-cold-start: 11 pass.
  • Mutation sweep over a temp copy (each breakage must fail the test meant to catch it): 23 of 23 caught (existing-install defaults, legacy outputs mapping, the locate profile and the study arm, --init overwriting or flipping on either path, the baseline file, the footprint limits and the product's cap, the 10% bar, the median, and the "did not answer" refusals). The first round caught 18 of 21 and found one real gap in my own test (a failing ask was never exercised, only a failing generate), one stale mutation pattern, and one mutant aimed at a source file the CLI never reads (it runs the bundle), which is why the premise test exists.
  • node test/integration/all.js and CI.

🤖 Generated with Claude Code

…, no default change

T1 pins that Candidate A (strategy index + the two-tool locate profile) is exactly what the study's locate arm writes, and what sigmap --init proposes today on both of its paths, so a flip shows up as a changed pin. T2 pins that four kinds of existing project resolve to the same effective config and tool profile as at v8.87.0 (recorded from the tag; develop already matches), and that --init never rewrites a config file. T3 pins the generated AGENTS.md and CLAUDE.md within 32 KiB and 200 lines under today's config and Candidate A; clones are skipped and said so, never counted as a pass. T4 is a cold-start timing tool whose verdict is an interval: within, slower, or inconclusive.

Nothing here changes a default, a config or shipped behaviour.
@manojmallick
manojmallick merged commit a21e2cc into develop Nov 21, 2026
6 checks passed
@manojmallick
manojmallick deleted the feat/default-guard-rails-1071 branch November 21, 2026 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant