-
Notifications
You must be signed in to change notification settings - Fork 3
fix(protocol): enforce max chunk size on manifests and chunk data #70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
89168ef
37ddf3f
d8a50da
a5fff0a
680ebdd
3f797a6
d440031
3fa81f8
a881fcd
6891205
137d0eb
366bc0d
9880048
835cb4d
5cad078
71bafca
d8b4e4f
981f9ff
c699249
94c355c
2dc8e1b
327648c
a51ec07
69b1d29
8e3393e
286a8cf
fb08a2b
86a8591
c61f9c0
da625b1
fe94540
5e6d9c1
299b30f
8d3b554
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -390,11 +390,32 @@ export class Downloader { | |
| // Phase 1: fetch manifest from a peer if needed | ||
| if (this.state === 'awaiting-manifest') { | ||
| if (this.peerManager.size() === 0) return; | ||
| for (const [, client] of this.peerManager.entries()) { | ||
| // LIVE iteration on purpose (Map iterators tolerate deletes and visit entries | ||
| // added mid-loop): a peer joining via HAVE while we await another's manifest | ||
| // gets its turn in THIS pass — its doWork() trigger no-ops on the locked mutex. | ||
| for (const [peerID, client] of this.peerManager.entries()) { | ||
| let manifest: import('@shared').IStoredLISH | null = null; | ||
| try { | ||
| manifest = await client.requestManifest(this.lishID); | ||
| } catch (error: any) { | ||
| if (error instanceof CodedError && error.code === ErrorCodes.LISH_CHUNK_SIZE_TOO_LARGE) { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When an Useful? React with 👍 / 👎.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 5cad078 — the awaiting-manifest loop now drops a peer whose manifest exceeds the limit (a forged/stale one from a single peer) and keeps trying the rest; another peer can still serve a valid manifest. The terminal LISH_CHUNK_SIZE_TOO_LARGE is surfaced only once every peer has been dropped this way (the LISH itself is over-limit), so one bad peer can no longer block a valid import, and a genuinely oversized LISH no longer stalls silently in awaiting-manifest. The probe path just drops such peers too and leaves the terminal decision to this loop. |
||
| // The peer delivered a well-formed manifest for the LISH we asked for and it | ||
| // declares a chunk size above our limit. Chunk size is a property of the LISH | ||
| // itself, so every honest peer serves the same value — asking the rest only | ||
| // makes the user watch each peer fail in turn before the same error appears. | ||
| // Surface it now and stop — unless the download was torn down while we | ||
| // awaited the manifest, in which case there is no state left to fail. | ||
| this.peerManager.remove(peerID, 'drop'); | ||
| if (!this.destroyed) this.setError(error.code, error.detail); | ||
| return; | ||
| } | ||
| // A structurally malformed manifest (mapped to PEER_INVALID_REQUEST) is this | ||
| // peer's fault — keeping it would leave the download stuck asking the same | ||
| // bad peer forever while discovery skips it as "connected". | ||
| if (error instanceof CodedError && error.code === ErrorCodes.PEER_INVALID_REQUEST) { | ||
| this.peerManager.remove(peerID, 'drop'); | ||
| continue; | ||
| } | ||
| console.warn(`[DL] Manifest request failed: ${error.message?.slice(0, 120) ?? error}`); | ||
| } | ||
| if (manifest && manifest.files && manifest.files.length > 0) { | ||
|
|
@@ -571,6 +592,22 @@ export class Downloader { | |
| try { | ||
| manifest = await probeClient.requestManifest(this.lishID); | ||
| } catch (error: any) { | ||
| // Any manifest error (unreachable, malformed) → drop this peer and let another | ||
| // serve it, except over-limit which is terminal for the whole LISH — but only | ||
| // while we are still looking for a manifest. Probing also runs mid-download | ||
| // purely to find more peers, and there requestManifest is just a "do you have | ||
| // this LISH?" test whose answer is discarded; failing the transfer on it would | ||
| // hand any peer on the topic a way to kill a healthy download. | ||
| if (this.needsManifest && error instanceof CodedError && error.code === ErrorCodes.LISH_CHUNK_SIZE_TOO_LARGE) { | ||
| // Same reasoning as the connected-peer loop: a delivered manifest that is over | ||
| // the limit answers the question for the whole LISH, so stop probing the rest. | ||
| // close() must not throw past this point — the outer catch would swallow the | ||
| // verdict, log it as an unreachable peer and let the probe loop carry on. | ||
| this.peerManager.remove(peerID, 'drop'); | ||
| await probeClient.close().catch(() => {}); | ||
| if (!this.destroyed) this.setError(error.code, error.detail); | ||
| return; | ||
| } | ||
| console.debug(`[DL] probe ${peerID.slice(0, 12)}: manifest error ${error.code ?? error.message?.slice(0, 60) ?? error}`); | ||
| this.peerManager.remove(peerID, 'drop'); | ||
| } | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a checksum appears in multiple slots,
writeChunkToAllSlotslater writes this same payload to every slot sharing the chunk ID, but this new check only compares the length for the queued slot. With a malformed manifest that lists the same checksum for a full chunk before a shorter last-chunk slot, a full-length payload passes here and is then written past the end of the shorter file while all matching slots are marked downloaded. Reject duplicate checksum groups with mismatched expected lengths, or validate the payload against every target before accepting it.Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in 3fa81f8 at the root cause — validateLISHStructure now rejects manifests where the same checksum is claimed by slots with different expected lengths (unsatisfiable by any single payload), so such a manifest never reaches the duplicate-slot write path. Tracking only short last chunks keeps the check O(#files) in memory.