Skip to content

fix(api): rewrite the pubspec inside the archive too, not only the metadata - #7

Merged
greymag merged 1 commit into
masterfrom
fix/https-migration-archive-rewrite
Sep 8, 2026
Merged

greymag merged 1 commit into
masterfrom
fix/https-migration-archive-rewrite

Conversation

@greymag

@greymag greymag commented Sep 8, 2026

Copy link
Copy Markdown

Fixes the migration reported after 3.6.0: pub get works once after a cache clean, then fails again with the source conflict.

The mechanism

dart pub reads a hosted package's dependencies from the version listing only while that package is not yet in the local cache. Once it has been extracted into $PUB_CACHE/hosted/<host>/<package>-<version>/, every later solve reads that copy instead. Rewriting the metadata therefore fixes exactly one resolution per cleared cache.

Proved, not reasoned about — one PUB_CACHE, three runs:

RUN 1  exit=0     ← clean cache, rewritten listing
RUN 2  exit=1     ← the reported failure, verbatim
RUN 3  exit=0     ← after patching ONLY the extracted pubspec.yaml in the cache

The acceptance test that shipped with 3.6.0 ran pub get once and asserted the extracted archive still named the old address — reading the cause as evidence of harmlessness. It now runs twice.

The fix

The archive is rewritten on its way out as well. The stored file is never touched; the bytes are transformed per request and only the one url differs. The pubspec text is edited at the source spans package:yaml reports rather than re-serialised from a parsed document, so comments, quoting and key order survive; file modes and timestamps are carried through. Nothing is unpacked unless the stored metadata says a rewrite applies, so a repository where nothing was published under an old address pays nothing.

Deterministic gzip. GZipEncoder stamps DateTime.now() into the header's MTIME field, so the same archive encoded two seconds apart hashes differently — and pub records that hash in pubspec.lock and checks the cached copy against it. Every clean-cache resolve would have reported the lockfile hash as stale and rewritten it, and --enforce-lockfile would simply have failed. The field is zeroed, which is what RFC 1952 reserves for "no timestamp" and what gzip -n writes. Caught in review; the first determinism test passed only because both encodes landed in the same second, so there is now one asserting the field itself.

What consumers pay, once

The archive is no longer byte-identical to the one downloaded before, so a cache already holding the old copy is not repaired — pub does not re-download what it already has. Each consumer needs one dart pub cache clean; the next pub get reports the hash in pubspec.lock is out of date, updates it, and succeeds. pubspec.lock does not need deleting, and it does not have to be done twice. Verified end to end.

A package store that redirects to its own download urls (supportsDownloadUrl) cannot be rewritten this way, the bytes never passing through the server. That would silently reproduce this bug, so the server warns once per package instead. The built-in FileStore streams through and is unaffected.

Tests

574 pass. New coverage: pub get twice on a warm cache (the run that would have caught this), the archive byte-identical across requests, the gzip timestamp field zeroed, a package published since the move served untouched without being unpacked, a stale cache proven not self-healing and then repaired by clearing it, and the text-level rewrite preserving comments, quoting, structured hosted: maps, and leaving git/path/sdk, lookalike hostnames and unparseable yaml alone.

🤖 Generated with Claude Code

…tadata

3.6.0 rewrote the repository url in the metadata and stopped there, which
fixed exactly one `pub get` per cleared cache. `dart pub` reads a hosted
package's dependencies from the version listing only while that package is
not yet in the local cache; once it has been extracted into
`$PUB_CACHE/hosted/<host>/<package>-<version>/`, every later solve reads
that copy instead. So the first resolution succeeded on the rewritten
listing and the next one failed with the same source conflict, off the
`pubspec.yaml` pub had kept.

Proved rather than reasoned about: with one PUB_CACHE, run 1 exits 0 and run
2 exits 1 with the conflict; patching only the extracted pubspec makes run 3
exit 0. The acceptance test that shipped with 3.6.0 ran `pub get` once, and
asserted the extracted archive still named the old address — reading the
cause as evidence of harmlessness. It now runs twice, which is the run that
could have caught this.

So the archive is rewritten on its way out as well. The stored file is never
touched; the bytes are transformed per request, and only the one url differs
— comments, quoting, key order, file modes and timestamps are all carried
through, because the pubspec text is edited at the source spans `package:yaml`
reports rather than re-serialised from a parsed document. Nothing is unpacked
unless the stored metadata says a rewrite applies, so a repository where
nothing was published under an old address pays nothing.

The gzip output is made deterministic by hand. `GZipEncoder` stamps
`DateTime.now()` into the header's MTIME field, so the same archive encoded
two seconds apart hashes differently — and pub records that hash in
`pubspec.lock` and checks the cached copy against it. Every clean-cache
resolve would have reported the lockfile hash as stale and rewritten it, and
`--enforce-lockfile` would simply have failed. The field is zeroed, which is
what RFC 1952 reserves for "no timestamp" and what `gzip -n` writes. The
first determinism test passed only because both encodes landed in the same
second; there is now one asserting the field itself.

What consumers pay, once: the archive they receive is no longer byte-identical
to the one they received before, so a cache already holding the old copy is
not repaired — pub does not re-download what it already has. One
`dart pub cache clean` each, after which `pub get` reports the hash in
`pubspec.lock` is out of date, updates it, and succeeds. `pubspec.lock` does
not need deleting and it does not have to be done twice.

A package store that redirects to its own download urls cannot be rewritten
this way, the bytes never passing through here. That would silently reproduce
the bug this fixes, so the server says so once per package instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@greymag
greymag merged commit 8af3f0e into master Sep 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant