refactor(tokenless): drop the standalone toon binary

[ikunkun-sys]

Why

TOON encoding has run in-process through the toon-format library since 0.3.2 (7089ae0): every runtime path goes through tokenless compress-toon / tokenless decompress-toon, and nothing executes a standalone toon binary anymore. The binary survived only as an install-time dependency — built via cargo install toon-format (network access to crates.io at RPM build time, MSRV friction from its transitive deps) and shipped by every packaging channel without ever being invoked.

Verified before removal: zero exec/spawn/Command::new/resolve_binary call sites reference a toon binary across all adapters, hooks, crates, and the Python runtime; tool-ready-spec.json never lists it.

What

• Stop building/installing toon in the Makefile, justfile, RPM spec, raw packaging, the Anolisa component contract, and the npm packages (root bin entry, postinstall, packer).
• Remove the toon payload file and {bindir}/toon symlink entries from the anolisa release manifest (src/anolisa/manifests/components/tokenless/component.toml), which the anolisa installer stages as the real component contract at install time.
• Drop toon from helper-binary fallback path resolution in hook_utils.py, tool_ready_hook.sh, and env_check.rs.
• Clean up toon artifacts left behind by older releases on make install / make uninstall: the Tokenless-owned libexec helper is always removed, while $(BINDIR)/toon is removed only when it is a symlink pointing into a Tokenless helper layout, so an unrelated user-installed toon executable is preserved (regression-tested by the new tests/test-toon-cleanup.sh). RPM upgrades already remove the packaged files via %files, and the existing %post stale cleanup keeps covering user-local paths.
• Delete the standalone toon CLI test; port test-toon-full.sh and run-all-tests.sh to tokenless compress-toon / decompress-toon (the round-trip test now tolerates the no-savings JSON passthrough).
• Update packaging tests, the CI npm smoke check, build-all.sh, both READMEs, the npm README, the user manual, and the make build comment in AGENTS.md.
• Rewrite the stale NOTICE entry that still pointed at the removed third_party/toon/ submodule: toon-format is now listed under the Rust crate dependencies as a statically linked crates.io crate, keeping its MIT copyright attribution.

CLI users keep the same functionality through tokenless compress-toon / tokenless decompress-toon. The npm root package no longer exposes a toon bin and the RPM no longer provides /usr/bin/toon. Per specs/documentation-standard.md, the changelog entry for this user-visible change is deferred to the next release version-bump PR.

Testing

• cargo test --workspace: 12 suites, all green; clippy shows no new warnings against my changes.
• tests/test-package-raw.sh, tests/test-package-npm-prebuilt.sh, and the new tests/test-toon-cleanup.sh (install-helpers/uninstall × preserve-regular-file / remove-legacy-symlink / preserve-foreign-symlink) pass.
• Python hook suites pass (test_compress_response_hook, test_compress_toon_hook, codex/hermes thresholds, rewrite, resolve_agent_id); test-openclaw-toon-threshold.mjs 4/4.
• Rebased onto main after the 0.7.9 bump; scripts/check-component-versions.py and tests/test-check-component-versions.sh pass with the edited release manifest.
• End-to-end in a sandbox whose PATH contains no toon: tabular JSON encodes at 44% char savings, compress-toon → decompress-toon round-trips semantically identical JSON; compress_toon_hook.py (1845→825 chars) and the combined compress_response_hook.py pipeline (2437→745 chars, 69%) both produce correct hook output; stats recording works; make install into a DESTDIR succeeds and removes a pre-seeded stale toon from an older layout.

🤖 Generated with Claude Code

[CLAassistant]

All committers have signed the CLA.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9e67eeefb6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[kongche-jbw]

Review baseline: 52eeb520147b...9e67eeefb647

[P1] 仅清理 Tokenless 自己创建的 toon 链接

src/tokenless/Makefile:177 在全新执行 make install-helpers 时也会无条件删除
$(BINDIR)/toon。若用户此前独立安装了同名可执行文件，安装 Tokenless 会直接将其删除；
uninstall 的第 213 行也有同样风险。用普通文件作为现有 toon 可稳定复现删除。
Possible direction: 仅在该路径是指向旧版 Tokenless libexec helper 的符号链接时删除，
并补充“保留普通文件、删除旧版链接”的安装与卸载回归测试。

[P2] 把 CHANGELOG 条目留给发布提交

src/tokenless/CHANGELOG.md:14 和 CHANGELOG_zh.md:14 在非版本发布提交中写入
Unreleased，违反 specs/documentation-standard.md:272 的 release-only 规则。
请从本 PR 移除这两条，由下一次版本 bump 统一归档。

[Forrest-ly]

Overall evaluation

Clean, well-scoped removal of the standalone toon binary: TOON encoding has run in-process via the toon-format library since 0.3.2, and this PR strips the dead install-time dependency from every build/packaging channel, helper-binary fallback resolution, tests, and docs. I independently verified the removal is complete and that all upgrade paths shed the leftover executables (RPM drops the %files entries + keeps the %post stale cleanup; the anolisa CLI contract replay removes prior-manifest-owned files; Makefile install/uninstall clean up both paths).

Review conclusion

approve

Detailed findings

🔴 Must-fix (blocks merge)

None from my side. The two existing Codex review threads are not re-raised here; of those, I agree the unconditional rm -f cleanup one (Makefile install-helpers/uninstall) is worth resolving before merge — the suggested guard (only delete a symlink pointing at the legacy tokenless libexec path) is cheap and eliminates any chance of removing an unrelated user executable.

🟡 Suggested (non-blocking)

• [src/tokenless/docs/design/runtime-library.md:9 and runtime-library_zh.md:9] Stale reference to the removed binary: the doc still says Python applications do not require tokenless, rtk, or toon on PATH. The PR swept the equivalent wording from both READMEs, the npm README, and the user manual, but missed this design doc. Suggest dropping toon from the enumeration in both language versions.
• [src/tokenless/tests/test-toon-full.sh:197-205] Scenario 1.7 never checks subprocess exit codes: if tokenless compress-toon fails outright, p1.stdout is empty, json.loads("") raises, the fallback decompress-toon on empty input also fails, and the uncaught exception surfaces as a raw traceback plus the generic "往返转换存在数据不一致" failure, masking the real cause. Suggest failing fast with a clear message when p1.returncode != 0 or toon_out is empty (same for p2).

🟢 Worth noting

• The removal is genuinely complete: a repo-wide sweep for toon-binary usage (exec/spawn/Command::new, resolve_binary candidates, bin/toon layout paths, cargo install toon-format) finds zero leftovers across adapters, hooks, crates, the Python runtime, packaging, CI, and build scripts; tool-ready-spec.json never listed it.
• The ported tests match the real CLI surface: -f/--file exists on compress-toon/decompress-toon, and the round-trip test's json.loads heuristic correctly tolerates the new no-savings JSON passthrough branch in main.rs.
• make uninstall now also removes $(BINDIR)/toon, fixing a pre-existing gap where the old uninstall deleted the libexec binary but left a dangling symlink in the bin dir.

[Forrest-ly]

总体评价

新 head（509489d，squash 后的单 commit）完整落实了上一轮全部审查意见：clean-stale-toon 符号链接守卫 + 新增 tests/test-toon-cleanup.sh 回归测试解决了 Makefile 无条件删除 $(BINDIR)/toon 的 P1 问题；两处 CHANGELOG Unreleased 条目已从 PR 中移除（两文件的 Unreleased 段现为空）；runtime-library.md/runtime-library_zh.md 中遗留的 toon 表述与 test-toon-full.sh 缺少子进程退出码检查的问题也一并修复。我在本地对新 head 做了独立验证：cleanup 回归测试、raw/npm 打包测试、cargo test -p tokenless-cli（257 passed）全部通过。

审查结论

approve

详细意见

🔴 必须修改（阻塞合并）

无。

🟡 建议修改（不阻塞但推荐）

• [PR description] PR 正文仍写着 "noted in the Unreleased changelog entry"，但按上轮意见该 changelog 条目已从本 PR 移除（src/tokenless/CHANGELOG.md / CHANGELOG_zh.md 不再在 diff 中，Unreleased 段为空）。建议同步修订 PR 描述，避免发布时误导 release 流程；条目本身留给下一次版本 bump 归档即可。

🟢 值得肯定

• clean-stale-toon 的守卫实现正确：$(LIBEXECDIR) 为 Tokenless 自有目录故无条件清理；$(BINDIR)/toon 仅在 readlink 命中 */anolisa/tokenless/toon 或 */anolisa/libexec/tokenless/toon 时删除。我核对了全部历史安装布局（Makefile user/system、Anolisa CLI user/system 的 libexec 路径）均能命中这两个模式；普通文件与指向无关位置的符号链接被保留。本地运行 tests/test-toon-cleanup.sh 通过（install-helpers 与 uninstall 各 3 个场景），并手动验证了 Anolisa CLI user-mode 布局的匹配。
• test-toon-full.sh 场景 1.7（原 1.8）现在对 compress-toon/decompress-toon 的返回码与空输出 fail fast，且 json.loads(toon_out) 的 try/except 正确处理了 no-savings JSON passthrough 分支（与 main.rs 中 CompressToon 的实际行为一致）；-f/--file 参数与 CLI 定义吻合。
• 移除的完整性再次确认：全仓库 grep 无残留的 standalone toon 调用点（exec/spawn/Command::new/resolve_binary/cargo install toon-format），tool-ready-spec.json 从未包含 toon，env_check.rs 的 fallback 收敛为 binary == "rtk" 且无调用方再传 "toon"；spec 的 %post stale cleanup 对 user-local 路径的覆盖保持不变。
• 升级路径完备：RPM %files 移除后升级自动清理打包文件；component contract 移除对应 layout 条目；Makefile install/uninstall 由新守卫兜底。

[Forrest-ly]

Code review of new head 509489d (re-review after force-push; detailed findings in the review thread):

• ✅ All three first-round items are addressed: Makefile stale-toon cleanup is now scoped to Tokenless-owned symlinks (verified patterns against system/user/relative layouts), tests/test-toon-cleanup.sh adds the requested install/uninstall regression coverage (ran locally — passes), and the premature Unreleased CHANGELOG entries were removed.
• 💡 Non-blocking suggestions: sync src/anolisa/manifests/components/tokenless/component.toml (still declares two toon layout entries), update the stale make build note in root AGENTS.md:76, and wire test-toon-cleanup into the CI tokenless job.

Overall the removal is clean and well-tested; see the review for details.

[Forrest-ly]

Re-review of the squashed head 509489d (first round reviewed 9e67eee before the force-push).

First-round feedback — all three items verified addressed

1. [P1] Makefile cleanup scope (kongche-jbw / codex bot) — Fixed. clean-stale-toon (src/tokenless/Makefile:170-183) always removes the Tokenless-owned $(LIBEXECDIR)/toon helper but removes $(BINDIR)/toon only when it is a symlink whose target matches a Tokenless helper layout (*/anolisa/tokenless/toon or */anolisa/libexec/tokenless/toon). Both install-helpers (old line 177 risk) and uninstall (old line 213 risk) now use it. I verified the case patterns against system-mode (/usr/libexec/anolisa/tokenless/toon, /usr/local/libexec/...), user-mode (~/.local/lib/anolisa/libexec/tokenless/toon) and relative symlink targets; unrelated regular files and foreign symlinks are preserved.
2. [P1] Regression tests (kongche-jbw) — Added: tests/test-toon-cleanup.sh (wired into make test / .PHONY) covers preserve-regular-file / remove-legacy-symlink / preserve-foreign-symlink for both install-helpers and uninstall. I ran it locally on this head: passes.
3. [P1→P2] Unreleased CHANGELOG entries — Fixed: this head no longer touches CHANGELOG.md / CHANGELOG_zh.md at all; the release note should be folded into whichever release section ships this change (per specs/documentation-standard.md release-only rule).

Additional verification on 509489d

• Full-tree grep: no leftover standalone-toon binary references — env_check.rs fallback paths, tool-ready-spec.json (never listed it), user-guide docs, adapters, npm packer/postinstall, raw packaging, RPM spec and CI smoke check are all clean; remaining "toon" mentions are legitimate TOON-format/library references.
• All changed shell scripts pass bash -n; both npm scripts pass node --check; the ported test-toon-full.sh round-trip logic correctly tolerates compress-toon's no-savings JSON passthrough (matches tokenless-cli behavior), and compress-toon -f is a real flag.
• RPM %files removal + unchanged %post user-local cleanup cover the RPM upgrade path as described.

Suggestions (non-blocking)

1. P2 — anolisa catalog manifest now out of sync. src/anolisa/manifests/components/tokenless/component.toml:54-58,65-68 still declares the two toon layout entries (libexec executable + {bindir}/toon symlink). This PR updates the tokenless-side contract (component.toml.in) but not the bundled catalog copy that catalog.rs describes as "real component contracts extracted from the OSS release artifacts". Installs stay correct (the install runner prefers the artifact-embedded contract), but nothing else will resync this file — version-bump commits only touch the version = line. Recommend deleting those two entries in this PR.
2. P3 — root AGENTS.md:76: make build # tokenless + RTK + TOON + OpenClaw plugin is now stale; TOON is no longer a built artifact (linked via the toon-format library).
3. P3 — new cleanup test is not exercised by CI. The tokenless CI job runs cargo test, make test-integration, and the npm smoke test, but not the aggregate make test, so test-toon-cleanup.sh only runs locally. It is fast and self-contained (stubs rtk, DESTDIR-scoped) — consider adding make -C src/tokenless test-toon-cleanup to the CI job.

(FYI, pre-existing on main and out of this PR's scope: NOTICE:75 still lists src/tokenless/third_party/toon/ as a git submodule, which hasn't existed since the crates.io migration.)

Verdict: COMMENT — all first-round blockers are resolved; the removal itself is clean, complete, and well-tested. The catalog-manifest sync (suggestion 1) is the one item I'd recommend addressing, ideally in this PR.

[ikunkun-sys]

@Forrest-ly 已请求 re-review。相对你上次批准的 509489da，新 head 233617e0 的增量：

1. Rebase 到 0.7.9 bump 之后的 main（无冲突，无内容改动）。
2. 借主干合入后的全树复查，修复了 3 处此前遗漏的 toon 残留：
   • src/anolisa/manifests/components/tokenless/component.toml — 删除 toon 载荷文件和 {bindir}/toon symlink 两组布局条目（该清单是 anolisa 安装器实际 stage 的组件契约，独立于 .anolisa/component.toml.in，此前漏掉）；
   • NOTICE — toon 条目从已不存在的 third_party/toon/ 子模块描述改为 Rust 依赖小节里的 toon-format (v0.5) crates.io 静态链接条目，保留 MIT 版权归属；
   • AGENTS.md — make build 注释去掉 TOON（build: 已不含 build-toon）。

验证：manifest TOML 解析通过；scripts/check-component-versions.py 与 tests/test-check-component-versions.sh 通过；全树逐词扫描确认除清理逻辑和历史 changelog 外无 toon 二进制引用。

[Forrest-ly]

Re-review of the incremental delta 509489d → 233617e (new squashed head after rebase onto post-0.7.9 main).

Overall assessment

The increment is exactly three newly-touched files, each one resolving an open item from my last review; every other file's patch is byte-identical to the previously approved 509489d (verified by per-file patch-hash comparison of both commit trees), so the rebase introduced no content drift and all prior verification carries over.

Review conclusion

approve

Detailed findings

🔴 Must fix (blocking)

None.

🟡 Suggested (non-blocking)

None new. One carried-over note for traceability: the earlier P3 "consider wiring make test-toon-cleanup into the CI job" suggestion is still unaddressed in this head — remains a non-blocking consideration.

🟢 What was done well

1. [P2 resolved] anolisa catalog manifest synced. src/anolisa/manifests/components/tokenless/component.toml drops both toon layout entries (the libexec/anolisa/tokenless/toon executable and the {bindir}/toon symlink) and nothing else. Verified at 233617e: zero remaining toon references in the file, the remaining layout is untouched, and the entry set now matches the tokenless-side contract src/tokenless/.anolisa/component.toml.in line-for-line.
2. [P3 resolved] AGENTS.md:76 comment updated to make build # tokenless + RTK + OpenClaw plugin — accurate against the Makefile build: target (build-tokenless build-openclaw-plugin build-dsh-plugin, where build-dsh-plugin is only a bundle-validation seam and predates this PR).
3. [NOTICE FYI resolved] The stale third_party/toon/ submodule entry is replaced by a toon-format (v0.5) entry under Rust Dependencies. Cross-checked: version matches src/tokenless/Cargo.lock (toon-format 0.5.0), the MIT license plus both copyright attributions are preserved, and the rewrite also fixes the old description ("lossless binary JSON codec" → "lossless JSON re-encoding", which is accurate — TOON is text). Placement after clap is consistent with the section's existing append-style ordering.

Verification performed on this head

• Per-file patch-hash diff between 509489d and 233617e: all 26 previously-reviewed files unchanged; only AGENTS.md, NOTICE, and the catalog component.toml are new.
• Full-file greps confirm no leftover standalone-toon binary references in the two newly-edited contract/legal files; toon-format/TOON-format mentions that remain are the legitimate library references.
• kongche-jbw's earlier P1 items (guarded clean-stale-toon symlink cleanup + tests/test-toon-cleanup.sh regression coverage) and the CHANGELOG release-only rule remain intact since their patches are unchanged from the approved head.

[ikunkun-sys]

CI 的 Test tokenless 失败与本 PR 无关，是生态漂移：openai 3.3.0（昨晚发布）移除了 tqdm 依赖，而 AgentScope 1.x 在 agentscope.evaluate 里 import 时使用了未声明的 tqdm，导致 test-agentscope-integration 的 v1-minimum 环境 import 即挂。证据：昨天 main 上最后一次绿色 run（12:29）解析到 openai==3.2.0 + tqdm==4.70.0，本 PR 的 run 解析到 openai==3.3.0、无 tqdm。任何触碰 src/tokenless/** 的 PR 都会同样失败。

修复见 #2662（v1 冒烟环境显式安装 tqdm）。待其合入 main 后 re-run 本 PR 失败的 job 即可（pull_request 检出的是与 base 的 merge ref，会自动带上修复）。

[Forrest-ly]

增量审查（已批准 head 233617e → 新 head 17e7761）：本次推送为纯 rebase 到最新 main（main...17e7761 = ahead 1 / behind 0，吸收 20 个主干 commit 后重新 squash）。逐文件核对：全部 29 个改动文件的增删行与已批准的 233617e 完全一致，无任何新增未审查改动。

复核要点：

• 首轮 P1 修复仍然在位：src/tokenless/Makefile 的 clean-stale-toon 仅在 $(BINDIR)/toon 是指向 Tokenless helper 布局（/anolisa/tokenless/toon、/anolisa/libexec/tokenless/toon）的 symlink 时才删除，与 LIBEXECDIR=$(PREFIX)/libexec/anolisa/tokenless 的实际布局匹配；用户独立安装的 toon 可执行文件不受影响。tests/test-toon-cleanup.sh 覆盖 regular/legacy/foreign × install-helpers/uninstall 共 6 个场景。
• CHANGELOG 未被改动，符合 release-only 规则（首轮 P2）。
• toon 独立二进制的移除在 Makefile、justfile、RPM spec、raw 打包、component 契约（两份副本同步）、npm 包（package.json/package-npm.js/postinstall.js）、hooks（hook_utils.py/tool_ready_hook.sh）、env_check.rs、测试与双语文档中保持一致；NOTICE 将 toon-format 正确移到 crates.io 依赖区（MIT，静态链接）。
• 新 head CI：Test tokenless ✅ (7m53s)、Test anolisa ✅、Source Build (Ubuntu 22.04) ✅、Check component versions ✅，其余 job 为不相关组件 skipping。

Code review 通过：与已批准内容完全一致，无新增问题。