test(e2e): the standalone tool listing is asserted as a set, so the nightly gate stops going stale on a tool the platform always registered (#1644)

[cjimti]

What is true after this

TestAdminAPI_Standalone/list_tools_platform_level asserts the exact set of platform-level tools a standalone server registers — platform_info, list_connections, platform_find_tools — and reports by name any toolkit tool that leaked into a connectionless server, and any platform tool that went missing from the listing.

The nightly real-dependency E2E workflow passes.

Why the count had to go

The assertion read tools.Total != 2. #1586 (v1.129.0) fixed Platform.PlatformTools() (pkg/platform/platform.go:3262) to name every tool the platform registers rather than only two of them, and platform_find_tools has been registered unconditionally in Start (pkg/platform/platform.go:2887) since #504. A standalone server therefore listed three platform tools, and did before #1586 — what changed is that the admin listing (internal/httpserver/mounts.go:668) stopped under-reporting.

So the gate went red on 2026-09-02 and stayed red through v1.130.5: runs 33627629590, 33752789054, 33870775195, 33962678490, 34030656488, each failing on that one subtest and nothing else. The cost was not the failure. It was that the only gate running the platform against real PostgreSQL, Trino and SeaweedFS was unreadable for five days, so a genuine regression in that suite would have been indistinguishable from the noise.

The intent behind the assertion is worth keeping: a standalone server with no connections must not register a toolkit tool, and an exact assertion is what makes that detectable. A count is the wrong exact assertion. It goes stale the next time the platform reports a tool it always registered, and when it fails it names neither the tool that appeared nor the tool that left.

The shape of the assertion

helpers.DiffToolNames(got, want) returns what a listing lacks and what it carries beyond the expected set, both sorted so the message is the same on every run. helpers.AssertToolSet(t, got, want...) reports each by name:

tool listing carries [trino_execute] beyond the expected set [platform_info list_connections platform_find_tools]

The store-conditional tools stay out of the expected set. PlatformTools() adds manage_prompt, show_prompts, manage_script, run_script and show_scripts only where their stores exist, and the standalone fixture has no database — the sibling system_info subtest asserts Features.Database == false on the same server.

Total is still read, now against the number of tools the response body carries (pkg/admin/system.go:186 sets Total: len(tools)), so the field the old assertion was the sole reader of does not become unasserted.

Acceptance

Run against the real dependency stack from docker-compose.e2e.yml, the same suite the nightly workflow runs.

The E2E suite passes. go test -v -race -tags=integration ./test/e2e/... — ok test/e2e 11.857s, ok test/e2e/helpers 1.067s; 98 passes and 12 skips, the skips being the DataHub-dependent suites that CI also skips because DataHub is deliberately absent from the compose stack. The subtest that had been red every night:

--- PASS: TestAdminAPI_Standalone/list_tools_platform_level (0.00s)

A leaked toolkit tool fails the test, naming the tool. Proved through the real subtest, by appending trino_execute to the listing the assertion reads and reverting the edit afterward:

admin_api_test.go:175: tool listing carries [trino_execute] beyond the expected set [platform_info list_connections platform_find_tools]
--- FAIL: TestAdminAPI_Standalone/list_tools_platform_level (0.00s)

A server listing exactly the three passes. The suite run above, plus TestDiffToolNames over the exact set, an out-of-order set, a leaked toolkit tool, a missing platform tool, multiple leaks reported sorted, and an empty listing.

One local deviation, noted for reproducibility: host port 9000 was held by a non-Docker listener on 127.1.27.{1,3,4} that survived make e2e-down, so SeaweedFS was bound to 9100 and the suite run with E2E_S3_ENDPOINT=localhost:9100, which test/e2e/helpers/config.go:50 reads. Everything else matched make e2e-up and make e2e-test.

Scope

Test-only: test/e2e/admin_api_test.go, test/e2e/helpers/assertions.go, and the new test/e2e/helpers/assertions_test.go. No production Go changed, which make verify's acceptance-check reports directly, and codecov ignores **/*_test.go, so patch coverage finds no executable changed lines.

The other Total != assertions in admin_api_test.go count rows the test itself inserts — personas, API keys, audit events — rather than what the platform registers, so they do not rot the same way and are untouched.

Closes #1644

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.59%. Comparing base (3930ec1) to head (9514fc4).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1646      +/-   ##
==========================================
- Coverage   91.60%   91.59%   -0.02%     
==========================================
  Files         777      777              
  Lines       77877    77877              
==========================================
- Hits        71337    71328       -9     
- Misses       4271     4276       +5     
- Partials     2269     2273       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.