Repository navigation
fix(pack): keep blobs larger than 512 MiB readable - #163
Merged
wesm merged 2 commits intoOct 8, 2026
Merged
Conversation
encodeFrame compresses every blob with WithSingleSegment(true). A
single-segment frame carries no window descriptor, so a decoder derives its
window from the frame content size, while normalizeReaderLimits and the shared
zstdDec both cap windows at zstd.MaxWindowSize (512 MiB). Blobs above that are
therefore written in a form this package refuses to read:
pack: corrupt: zstd decode: window size exceeded
The data is intact; it decodes correctly once a larger window is allowed.
Reading is what unblocks an existing repository: backup walks the hash-map
chain through such a blob before writing, so once one exists no further
snapshots can be created at all.
- reader.go: default WindowBytes ceiling raised from zstd.MaxWindowSize to
MaxRawLen. Decoder memory stays bounded by WithDecoderMaxMemory in openBlob.
- frame.go: the shared whole-blob decoder gets the same allowance.
- frame.go: blobs over zstd.MaxWindowSize are encoded without single-segment
framing, so they carry an explicit window descriptor and stay within what a
stock decoder accepts.
Reaching the limit needs an unusually large blob. The case that surfaced this
was 681.7 MiB, produced when VACUUM rewrote a SQLite archive into one
contiguous run; neighbouring blobs in the same pack were 1.6 and 2.2 MiB.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
roborev: Combined Review (
|
Member
|
Thanks for catching, reviewing now |
Member
|
i have fixes prpepared |
Existing repositories need coverage for frames written before the encoder fix. A round trip through the new encoder cannot detect regressions in those reads, and four huge payloads imposed unnecessary memory cost on CI. Allowing 4 GiB windows also lets valid legacy frames overflow the codec's integer buffer calculations on 32-bit systems. Preserve the previous 512 MiB ceiling there. On 64-bit systems, reading existing packs requires the larger allowance and its corresponding memory cost. Generated with Codex Co-authored-by: Codex <noreply@openai.com>
roborev: Combined Review (
|
This was referenced Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Existing blobs larger than 512 MiB become readable again. The previous writer emitted single-segment zstd frames whose effective window equals the full blob size, exceeding both the buffered decoder and default streaming reader's 512 MiB window limits.
On 64-bit systems, both readers now allow windows up to the format's 4 GiB raw-length ceiling. The 512 MiB window ceiling remains on 32-bit systems, where larger windows can overflow the codec's integer buffer calculations. This raises the default streaming memory allowance: reading an existing single-segment frame can require a decoder window as large as the blob. Callers can set a lower streaming window limit explicitly; packstore's bounded read APIs retain their policy limits, while Store.Open uses the raised default.
New blobs over 512 MiB use an explicit window within the klauspost decoder's default limit. Smaller blobs retain their existing framing, and existing packs need no rewrite.