fix(api): rewrite the pubspec inside the archive too, not only the metadata - #7
Merged
Merged
Conversation
…tadata 3.6.0 rewrote the repository url in the metadata and stopped there, which fixed exactly one `pub get` per cleared cache. `dart pub` reads a hosted package's dependencies from the version listing only while that package is not yet in the local cache; once it has been extracted into `$PUB_CACHE/hosted/<host>/<package>-<version>/`, every later solve reads that copy instead. So the first resolution succeeded on the rewritten listing and the next one failed with the same source conflict, off the `pubspec.yaml` pub had kept. Proved rather than reasoned about: with one PUB_CACHE, run 1 exits 0 and run 2 exits 1 with the conflict; patching only the extracted pubspec makes run 3 exit 0. The acceptance test that shipped with 3.6.0 ran `pub get` once, and asserted the extracted archive still named the old address — reading the cause as evidence of harmlessness. It now runs twice, which is the run that could have caught this. So the archive is rewritten on its way out as well. The stored file is never touched; the bytes are transformed per request, and only the one url differs — comments, quoting, key order, file modes and timestamps are all carried through, because the pubspec text is edited at the source spans `package:yaml` reports rather than re-serialised from a parsed document. Nothing is unpacked unless the stored metadata says a rewrite applies, so a repository where nothing was published under an old address pays nothing. The gzip output is made deterministic by hand. `GZipEncoder` stamps `DateTime.now()` into the header's MTIME field, so the same archive encoded two seconds apart hashes differently — and pub records that hash in `pubspec.lock` and checks the cached copy against it. Every clean-cache resolve would have reported the lockfile hash as stale and rewritten it, and `--enforce-lockfile` would simply have failed. The field is zeroed, which is what RFC 1952 reserves for "no timestamp" and what `gzip -n` writes. The first determinism test passed only because both encodes landed in the same second; there is now one asserting the field itself. What consumers pay, once: the archive they receive is no longer byte-identical to the one they received before, so a cache already holding the old copy is not repaired — pub does not re-download what it already has. One `dart pub cache clean` each, after which `pub get` reports the hash in `pubspec.lock` is out of date, updates it, and succeeds. `pubspec.lock` does not need deleting and it does not have to be done twice. A package store that redirects to its own download urls cannot be rewritten this way, the bytes never passing through here. That would silently reproduce the bug this fixes, so the server says so once per package instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the migration reported after 3.6.0:
pub getworks once after a cache clean, then fails again with the source conflict.The mechanism
dart pubreads a hosted package's dependencies from the version listing only while that package is not yet in the local cache. Once it has been extracted into$PUB_CACHE/hosted/<host>/<package>-<version>/, every later solve reads that copy instead. Rewriting the metadata therefore fixes exactly one resolution per cleared cache.Proved, not reasoned about — one
PUB_CACHE, three runs:The acceptance test that shipped with 3.6.0 ran
pub getonce and asserted the extracted archive still named the old address — reading the cause as evidence of harmlessness. It now runs twice.The fix
The archive is rewritten on its way out as well. The stored file is never touched; the bytes are transformed per request and only the one url differs. The pubspec text is edited at the source spans
package:yamlreports rather than re-serialised from a parsed document, so comments, quoting and key order survive; file modes and timestamps are carried through. Nothing is unpacked unless the stored metadata says a rewrite applies, so a repository where nothing was published under an old address pays nothing.Deterministic gzip.
GZipEncoderstampsDateTime.now()into the header's MTIME field, so the same archive encoded two seconds apart hashes differently — and pub records that hash inpubspec.lockand checks the cached copy against it. Every clean-cache resolve would have reported the lockfile hash as stale and rewritten it, and--enforce-lockfilewould simply have failed. The field is zeroed, which is what RFC 1952 reserves for "no timestamp" and whatgzip -nwrites. Caught in review; the first determinism test passed only because both encodes landed in the same second, so there is now one asserting the field itself.What consumers pay, once
The archive is no longer byte-identical to the one downloaded before, so a cache already holding the old copy is not repaired — pub does not re-download what it already has. Each consumer needs one
dart pub cache clean; the nextpub getreports the hash inpubspec.lockis out of date, updates it, and succeeds.pubspec.lockdoes not need deleting, and it does not have to be done twice. Verified end to end.A package store that redirects to its own download urls (
supportsDownloadUrl) cannot be rewritten this way, the bytes never passing through the server. That would silently reproduce this bug, so the server warns once per package instead. The built-inFileStorestreams through and is unaffected.Tests
574 pass. New coverage:
pub gettwice on a warm cache (the run that would have caught this), the archive byte-identical across requests, the gzip timestamp field zeroed, a package published since the move served untouched without being unpacked, a stale cache proven not self-healing and then repaired by clearing it, and the text-level rewrite preserving comments, quoting, structuredhosted:maps, and leavinggit/path/sdk, lookalike hostnames and unparseable yaml alone.🤖 Generated with Claude Code