Skip to content

feat(aea,download): add random-access AEA and HTTP range readers - #1408

Merged
blacktop merged 4 commits into
blacktop:masterfrom
KishanBagaria:codex/remote-ranges-readers
Nov 17, 2026
Merged

blacktop merged 4 commits into
blacktop:masterfrom
KishanBagaria:codex/remote-ranges-readers

Conversation

@KishanBagaria

@KishanBagaria KishanBagaria commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Adds two random-access readers and one atomic-write helper. Nothing user-visible changes here; this is the groundwork for reading single files out of a remote filesystem image (discussed in #1375), which follows in #1409.

Three commits:

  • refactor(atomicfile): WriteRoot writes atomically inside an os.Root, sharing one implementation with Write, whose behaviour is unchanged.
  • feat(aea): aea.Reader, a random-access io.ReaderAt over an AEA1 symmetric archive. It authenticates the root header and the cluster header chain, then decrypts only the segments that are read, with a plaintext cache capped at the existing 32 MiB decrypt budget. The prologue parser, root validation and cluster walk are factored out of decrypt (readPrologue, validateRoot, walkSegments) and shared by both paths. Also adds ReadMetadata(io.ReaderAt, size), which Info now wraps, and Metadata.SymmetricKey, which Decrypt now uses.
  • feat(download): RangeReader, an io.ReaderAt over HTTP with 64 KiB pages in an 8 MiB LRU. Every response must be a 206 whose Content-Range, length and ETag match the request; a full-body 200 is rejected unread. Transport failures and 5xx, 429 and 408 responses are retried twice.

Notes for review:

  • aea.decrypt behaves as before but now runs through the shared helpers and reads with ReadAt instead of Seek.
  • aea.Reader does not verify unread segments or the archive padding. Everything it returns is authenticated.
  • RangeReader sits beside the ranger-based ZIP reader and does not replace it: that one never evicts and does not validate responses, and the ZIP path wants large sequential blocks.
  • aea.Reader, RangeReader and WriteRoot have no callers outside tests until feat(extract): read remote APFS files through HTTP ranges #1409.

Testing

  • go build ./... and go test for internal/atomicfile, internal/download, pkg/aea, internal/commands/extract and cmd/ipsw/cmd/... pass on this branch.
  • New tests: range reader response validation, cache bound and retries; AEA reader random access across segment boundaries, cache hits, and rejection of a modified root header, cluster header and segment.
  • Fuzz tests (test(aea) commit): FuzzNewReader over arbitrary archives, and FuzzWalkSegments, which authenticates a cluster header built from the fuzz input so the segment bounds checks are reached. Ran about 30 s and 40 s (85 thousand and 2.9 million executions) with no failures.
  • The existing pkg/aea decrypt tests pass unchanged apart from one call site whose signature changed.

Not run: make fmt in full (gofmt and go vet are clean on the changed files).

AI Assistance

Planned and written by GPT 6.1 Sol + Claude Fable 5.2. Reviewed manually.

🤖 Generated with Claude Code

KishanBagaria and others added 3 commits November 16, 2026 16:25
Write and WriteRoot share one temp-file, sync and rename sequence. Write keeps operating on plain paths; WriteRoot runs the same steps confined to an os.Root the caller already holds.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Reader authenticates the root header and cluster header chain, then decrypts only the segments a ReadAt touches, with a bounded plaintext cache.

The prologue parser, root validation and cluster walk are factored out of decrypt (readPrologue, validateRoot, walkSegments) and shared by both paths. ReadMetadata reads AEA metadata from an io.ReaderAt, and Metadata.SymmetricKey resolves an embedded or FCS-wrapped key.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
An io.ReaderAt for sparse remote access: 64 KiB pages in an 8 MiB LRU, large reads sent as direct ranges of up to 4 MiB. Every response must be a 206 whose Content-Range, length and ETag match the request; a full-body 200 is rejected unread. Transport failures and 5xx, 429 and 408 responses are retried twice.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
FuzzNewReader feeds arbitrary archives to the prologue parser. FuzzWalkSegments authenticates a cluster header built from the fuzz input, so the segment bounds checks are exercised rather than stopped at the HMAC.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

type osFS struct{}

func (osFS) Lstat(name string) (os.FileInfo, error) { return os.Lstat(name) }

func (osFS) Lstat(name string) (os.FileInfo, error) { return os.Lstat(name) }
func (osFS) OpenFile(name string, flag int, perm os.FileMode) (*os.File, error) {
return os.OpenFile(name, flag, perm)
func (osFS) OpenFile(name string, flag int, perm os.FileMode) (*os.File, error) {
return os.OpenFile(name, flag, perm)
}
func (osFS) Remove(name string) error { return os.Remove(name) }
return os.OpenFile(name, flag, perm)
}
func (osFS) Remove(name string) error { return os.Remove(name) }
func (osFS) Rename(oldname, newname string) error { return os.Rename(oldname, newname) }
return os.OpenFile(name, flag, perm)
}
func (osFS) Remove(name string) error { return os.Remove(name) }
func (osFS) Rename(oldname, newname string) error { return os.Rename(oldname, newname) }
Comment thread pkg/aea/decrypt.go
}

func decrypt(in, out string, symmetricKey []byte) (string, error) {
f, err := os.Open(in)
@blacktop
blacktop merged commit b6b732e into blacktop:master Nov 17, 2026
5 of 9 checks passed
@KishanBagaria
KishanBagaria deleted the codex/remote-ranges-readers branch November 17, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants