Skip to content

appstore: uninstall stops what an app left running; renamed apps point to their new id (io.pilot.smolmachines) - #473

Merged
TeoSlayer merged 7 commits into
mainfrom
fix/smolmachines-lifecycle
Oct 1, 2026
Merged

TeoSlayer merged 7 commits into
mainfrom
fix/smolmachines-lifecycle

Conversation

@TeoSlayer

Copy link
Copy Markdown
Collaborator

Problem

io.pilot.smolmachines was renamed to io.pilot.smol (#398). Its catalogue entry is now a hidden tombstone and cannot be installed. Copies of 1.2.0 are still installed (this includes a developer Mac). Two gaps in pilotctl leave those installs leaking:

  1. Uninstall leaves the app's processes running. A VM started through smolmachines.exec (smolvm-bin _boot-vm, about 311 MiB RSS) runs in its own session. So does a redis-server --daemonize, a postmaster, an exec child orphaned by the supervisor's pid-only SIGKILL, or an app instance orphaned by a daemon that died hard. All of these are reparented to init/launchd. appstore uninstall deleted the dir and left them running with nothing to manage them.

  2. Nothing tells installed copies about the rename. catalogue/README.md promises tombstone handling, but no released pilotctl implements it. On main:

    command result with the tombstone
    catalogue lists Smol Machines (renamed → io.pilot.smol)
    install io.pilot.smolmachines catalogue entry io.pilot.smolmachines has placeholder sha256 — the release pipeline hasn't filled this in yet
    outdated / upgrade --all (1.2.0 installed) all installed apps are up to date
    upgrade io.pilot.smolmachines already up to date (or not a catalogue app)

    So the old adapter stays in place, together with its leaks.

A related problem: 1.2.0 was also published for darwin/amd64, but its install.json has no smolvm build for that platform. Upstream ships none, from v1.2.0 through v1.18.0. On Intel Macs the adapter exited at every start (install assets: stage: no asset for darwin/amd64) until the supervisor suspended it.

Changes

  • uninstall stops what the app left running (appstore_procs*.go). It finds processes by the path of their executable: inside the app dir, or inside one of its kept backups. It does not use the process tree, group or session, because those processes are free to leave them. It sends SIGTERM, waits 5 s, then sends SIGKILL. This runs before the delete, and again after it to catch a supervisor respawn. The output, --json (stopped_processes) and the audit record list what was stopped.
    • macOS reads the path from kern.procargs2. The kernel keeps it after the file is deleted.
    • Linux reads /proc/<pid>/exe. Under binfmt emulation (Rosetta for Linux, qemu-user) that link points at the translator, so Linux also looks for an executable mapping in /proc/<pid>/maps.
    • Each pid is checked again right before it is signalled.
  • Rename tombstones.
    • catalogue hides them.
    • install <old> warns, then installs renamed_to. It follows one hop only.
    • install <old> --version X refuses and names the new id. The managed-fleet reconcile checks for the old id afterwards, so following the rename there would reinstall the new app every cycle.
    • outdated reports renamed.
    • upgrade --all skips renamed apps. The hourly updater runs it, and the new id has a different publisher key and different method names.
    • upgrade <old> exits 1 and prints the steps: install the new id, then uninstall the old one.
    • view prints a warning. call on an old id that is not installed says it was renamed.
  • install refuses a cli bundle whose install.json has no native tool for this os/arch. It fails with platform_mismatch.
  • Catalogue lint (catalogue/lint):
    • An entry with no bundle must be a well-formed tombstone. That means renamed_to names an installable entry, hidden is set, the publisher is kept, and there is no version and no metadata_url. Tombstones are checked on every run.
    • A cli bundle may only be published for platforms its install.json ships a tool for. Run against the historical smolmachines 1.2.0 entry, it reports: darwin/amd64: install.json ships smolvm only for darwin/arm64, linux/amd64, linux/arm64 ….
  • catalogue/README.md documents all of the above.

The catalogue data is unchanged. The tombstone stays as it is: it holds the publisher pin for installed copies. Moving a node over is pilotctl appstore install io.pilot.smol, then pilotctl appstore uninstall io.pilot.smolmachines --yes.

Verification

Real io.pilot.smolmachines 1.2.0 darwin/arm64 bundle on macOS. The VM was started through smolmachines.exec machine create/start. Everything was isolated in a temp root and HOME, and sandbox-exec denied ~/.pilot and /tmp/pilot.sock.

origin/main:  VM pid=58593 STILL RUNNING after uninstall ... 319712 KB smolvm-bin
              LEAK: VM pid=58593 still running with nothing left to manage it
this branch:  stopped 2 process(es) still running from the app's files:
              smolmachines-app (pid 58665), smolvm-bin (pid 58844)        (0.53 s)
orphaned VM + respawned app (dead-daemon case):
              stopped 2 process(es) ...: smolvm-bin (pid 59091), smolmachines-app (pid 59223)

Migration. After uninstall io.pilot.smolmachines, io.pilot.smol (left running) lists the machine as stopped. It restarts it with its own smolvm-bin, then stops and deletes it.

Linux, real bundle, orphaned in-flight exec child (smolvm-bin serve start, left behind by the pid-only SIGKILL):

  • linux/arm64 (native): LEFT on main, stopped on this branch.
  • linux/amd64 (Rosetta for Linux, where /proc/<pid>/exe is /run/rosetta/rosetta): same result. The maps fallback caught it.

Tombstone, linux/arm64 docker, committed signed catalogue:

  • Listing: 1 line on main, 0 on this branch (text and --json).
  • install io.pilot.smolmachines: warned, fetched io.pilot.smol-1.2.0-linux-arm64.tar.gz, sha256 OK (62cd9b71…), installed io.pilot.smol v1.2.0.
  • outdated: renamed: install io.pilot.smol, then uninstall io.pilot.smolmachines.
  • upgrade --all: skip io.pilot.smolmachines ….
  • darwin/amd64 (Rosetta): io.pilot.smol has no bundle for this platform (darwin/amd64); published platforms: darwin/arm64, linux/amd64, linux/arm64.

Tests:

  • New: appstore_procs_test.go, appstore_tombstone_test.go, appstore_assets_platform_test.go, lint TestTombstonesMustBeWellFormed, TestCLIBundleNeedsANativeToolForEachPublishedPlatform.
  • TestCmdAppStoreUninstallStopsLeftoverProcesses fails on main (stopped_processes = [], want pids …).
  • go test -short ./cmd/pilotctl and catalogue/lint pass on:
    • macOS arm64, under sandbox-exec with a temp HOME;
    • darwin/amd64 under Rosetta (new tests);
    • linux/arm64 and linux/amd64 in golang:1.25.13-bookworm.
  • go vet passes on every commit.
  • Online lint of the committed catalogue with every entry treated as new: the new checks add no findings. The only findings are the four already known ones (cosift, generallegal and wallet legacy native bundles; the slipstream pin).

Not in this PR

  • Stopping the app's process group when the supervisor stops an app is app-store Unit 3: Claim-based network mapping — auto-assign networks from token claims #39 (and a SIGTERM-first variant). With it, the in-flight exec child is never orphaned in the first place.
  • Teardown of io.pilot.smol VMs when the adapter stops is app-template work for io.pilot.smol. The 1.2.0 smolmachines adapter is signed with the old publisher key and is not rebuilt, so for old installs, uninstall is where the cleanup happens.
  • smolvm's disk images live in $HOME/Library/Caches/smolvm, outside $APP, and are shared with io.pilot.smol. Uninstall leaves them. That is what lets io.pilot.smol take over the machines.

🤖 Generated with Claude Code

teovl and others added 5 commits September 24, 2026 12:15
`appstore uninstall` deleted the app dir and left everything the app had
started that detached from it: a smolvm microVM (`smolvm-bin _boot-vm`, own
session, ~311 MiB RSS), a daemonized redis/postgres/mysql server, an
in-flight exec child orphaned by the supervisor's pid-only SIGKILL, an app
instance orphaned by a daemon that died hard. Reparented to init/launchd,
they ran on with nothing left to manage them.

Every one of them runs a binary that lives in the app dir (the app itself or
a native tool it staged under $APP), so uninstall now finds them by
executable path, not by process tree, group or session, which they are free
to leave: SIGTERM, 5 s grace, SIGKILL. It covers the app dir and its kept
backups (a process started before an upgrade runs from the retired dir),
runs once before the delete and once after (a supervisor that had not yet
seen the manifest go may respawn the app in between), and reports what it
stopped in the output, --json (`stopped_processes`) and the audit record.

- macOS reads the exec path from kern.procargs2 (kept after the file is
  deleted); Linux reads /proc/<pid>/exe (" (deleted)" dropped) and, for a
  binary run under binfmt emulation (Rosetta for Linux, qemu-user) where the
  kernel reports the translator, an executable mapping in /proc/<pid>/maps.
- Each pid is re-checked right before it is signalled, so a reused pid is
  never touched. pid 1, pilotctl and its parent are skipped.

io.pilot.smolmachines 1.2.0 (the real published darwin/arm64 bundle, VM
started through smolmachines.exec) against origin/main pilotctl:
  VM pid=58593 STILL RUNNING after uninstall ... 319712 KB smolvm-bin
  LEAK: VM pid=58593 still running with nothing left to manage it
with this change:
  stopped 2 process(es) still running from the app's files:
  smolmachines-app (pid 58665), smolvm-bin (pid 58844)
Linux arm64 and amd64 (Rosetta): the orphaned `smolvm-bin serve start`
child is LEFT on main and stopped here.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
catalogue/README.md ("Renaming an app") promised that a tombstone entry
(`renamed_to` + `hidden`, no bundle) is hidden from the listing and that
install/view/call of the old id warn and route to the new one. No released
pilotctl implements it: catalogueEntry had neither field (checked on main,
v1.13.9, v1.13.10-rc.1, v1.12.0, v1.11.0). With the io.pilot.smolmachines
tombstone (renamed to io.pilot.smol) on main today:

  catalogue       lists "Smol Machines (renamed → io.pilot.smol)"
  install         "catalogue entry io.pilot.smolmachines has placeholder
                  sha256 — the release pipeline hasn't filled this in yet"
  outdated        "all installed apps are up to date" (with 1.2.0 installed)
  upgrade <old>   "already up to date (or not a catalogue app)"
  call            "is the daemon running ...?"

so installed copies never learn about the rename and keep running an
adapter that leaves its VMs behind.

Now:
- `catalogue` (text and --json) omits tombstones.
- `install <old>` warns and installs `renamed_to` (one hop; a missing or
  chained target is an error). `install <old> --version X` refuses and names
  the new id: a pin names a release of one app, and following it would make
  the managed-fleet reconcile, which checks for the old id afterwards,
  reinstall the new app every cycle.
- `outdated` reports an installed old id as `renamed` (AVAILABLE = new id).
  `upgrade --all`, which the hourly updater runs, skips it: the new id has
  its own publisher key and method names. `upgrade <old>` exits 1 with the
  install-then-uninstall steps.
- `view <old>` warns; `call` of an old id that is not installed says it was
  renamed (the catalogue is only fetched when the app dir is absent).

Verified in docker linux/arm64 against the committed signed catalogue:
install io.pilot.smolmachines fetched io.pilot.smol-1.2.0-linux-arm64.tar.gz,
"sha256 OK (62cd9b71…)", installed io.pilot.smol v1.2.0. On darwin/amd64
(Rosetta) it fails with "io.pilot.smol has no bundle for this platform
(darwin/amd64); published platforms: darwin/arm64, linux/amd64, linux/arm64"
instead of the placeholder-sha error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A "cli" app's adapter is portable Go, so the binary platform check passes,
but at its first start it stages the fronted tool from install.json and
exits 1 when this host's os/arch is not listed. The supervisor restarts it
until the crash-loop limit suspends it: the install "succeeds" and the app
never runs. io.pilot.smolmachines 1.2.0 was published for darwin/amd64 like
that (upstream smolvm has no macOS x86_64 build):

  install assets: stage: no asset for darwin/amd64; available:
  darwin/arm64, linux/amd64, linux/arm64

install now refuses such a bundle with platform_mismatch before anything is
staged, naming the platforms the tool ships for. An install.json that does
not parse or lists no assets is left to the app, as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…atform

Two classes of broken entry passed the lint:

- An entry with no bundle was skipped as "a tombstone" whatever it held.
  Now it must be a rename tombstone as catalogue/README.md describes:
  renamed_to naming an installable entry (not itself a tombstone, one hop),
  hidden, the old publisher key kept (installed copies are pinned to it),
  no version (older pilotctl compares it against installed copies) and no
  metadata_url. Tombstones are checked on every run, touched or not, since
  a PR can remove or retire the target without editing the tombstone.
- A cli bundle published for a platform its install.json ships no native
  tool for. Against the historical io.pilot.smolmachines 1.2.0 entry
  (catalogue 386a77d) the lint now reports:
    darwin/amd64: install.json ships smolvm only for darwin/arm64,
    linux/amd64, linux/arm64, so the app exits at every start on
    darwin/amd64 ... Drop darwin/amd64 from `bundles`, or add its asset

The committed catalogue passes both checks: offline in the unit test, and
online with every entry treated as new (the only findings are the four
already known: cosift, generallegal and wallet legacy native bundles, and
the slipstream linux/amd64 pin).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Document what pilotctl now does with a rename tombstone (listing, install,
--version, outdated, upgrade, view, call), how to move a node to the new id,
that the lint checks tombstones, and to keep the old id's R2 prefix while the
new id's install.json still downloads from it (io.pilot.smol 1.2.0 stages
smolvm from io.pilot.smolmachines/1.2.0/). Add uninstall's process stop and
install's native-tool platform check to the install/upgrade notes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment thread cmd/pilotctl/appstore.go Fixed
Comment thread cmd/pilotctl/appstore_platform.go Fixed
teovl and others added 2 commits September 24, 2026 12:40
Code scanning flagged the stat of <root>/<appID> in `appstore call` (only
reached when app.sock is missing, and only stat'ed) and the read of the
bundle's install.json in the install platform check. Neither writes, and
both use the same paths the surrounding code already touches.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@TeoSlayer
TeoSlayer merged commit 2e93d13 into main Oct 1, 2026
16 checks passed
TeoSlayer added a commit that referenced this pull request Oct 1, 2026
)

`pilotctl appstore uninstall . --yes` resolved "." to the install root
itself: resolveUnder only checks containment, and the root contains
itself. The whole root was removed, and since #473 every app's leftover
processes were stopped too.

App ids are single directory names, so uninstall, caps and the
next-steps loaders now go through resolveAppDir, which rejects ".",
".." and anything with a path separator before the containment check.
install keeps resolveUnder: its argument may be a bundle directory.

Also drop two references to the removed managed-fleet reconcile from the
rename comments added in #473, and name the release in the catalogue
README.

Co-authored-by: Teo Calin <calinteodor@Teos-MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants