Repository navigation
Verify LaneProposal signer and parent hash chain in PushBlock - #2925
Conversation
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2925 +/- ##
=======================================
Coverage 68.42% 68.42%
=======================================
Files 5 5
Lines 456 456
=======================================
Hits 312 312
Misses 114 114
Partials 30 30
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
| // Verify parent hash chain to prevent a malicious proposer from | ||
| // breaking the block chain, which would deadlock header reconstruction. | ||
| // A mismatch is not an error: it can happen legitimately when a | ||
| // producer equivocates (restarts and produces a different chain). |
There was a problem hiding this comment.
equivocation of producer is as malicious behavior as equivocation of proposer. It does not endanger the security, but imo it would be worth escalating as an error, so that it gets logged. It will help debugging stalled lanes, if nothing more.
There was a problem hiding this comment.
Hmm, if we return an error, and the producer keep pushing the same equivocating block, then every receiver will repeatedly reconnect?
I've changed it to log an error but return nil for now
PushBlock accepted blocks without checking that the signer matches the lane or that the block's parentHash chains to the previous block. A malicious proposer could impersonate another lane or break the parent hash chain, which would deadlock header reconstruction in headers(). Added signer == lane check (before expensive crypto verification) and parent hash chain verification when the previous block is available. Co-authored-by: Cursor <cursoragent@cursor.com>
A parentHash mismatch can happen legitimately when a producer equivocates (e.g. restarts and produces a different chain). The block is still prevented from entering the queue, but the caller is not treated as malicious. Co-authored-by: Cursor <cursoragent@cursor.com>
Parent hash mismatch indicates a producer equivocated, which is malicious behavior worth logging to aid debugging stalled lanes. Co-authored-by: Cursor <cursoragent@cursor.com>
Parent hash mismatch is logged at error level to aid debugging stalled lanes, but not returned as an error to avoid tearing down the peer connection in the p2p layer. Co-authored-by: Cursor <cursoragent@cursor.com>
8836922 to
8df3583
Compare
## Summary - Fixes flaky `TestAvailClientServer` (`sei-tendermint/internal/p2p/giga`), which has been hanging Race Detection CI (including on unrelated PRs) with `parent hash mismatch` logs and a stuck `GlobalBlock` wait. - Replaces the broken “fake node0” setup on **shared** `nodes[0].data` with a same-key equivocating peer that has its **own** store and feeds corrupt blocks into node2 via `PushBlock` (admission API), after an honest tip exists. ## History **sei-v3** (`TestClientServer` in `stream/consensus/avail/server_test.go`) intentionally stressed the mesh by spawning a task that wrote **corrupt node0-lane blocks directly into node2’s avail** (`ProduceBlock` on node2’s state for lane 0) **concurrently** with honest production. There was no second node0, no shared data dual-`Run`, and no wait for an honest tip. That worked because sei-v3 `PushBlock` did **not** verify parent hashes—later honest blocks could still append at `next` even after a bad tip. **sei-chain migration (sei-protocol#2791)** rewrote this into giga `TestAvailClientServer` as a “fake unconnected node0” that: 1. Built a second `consensus.State` on **shared** `nodes[0].data` 2. Called `fakeNode0.Run` while `env.Run` already ran `nodes[0].consensus.Run` 3. Seeded node2 from that fake producer **before** honest production That mistranslated the sei-v3 inject and introduced a data race under `-race`. **sei-protocol#2925** then added parent-hash verification in `PushBlock`/`PushHeader`: on mismatch it logs and returns `nil` (drop, no rewind). A divergent tip on node2 can no longer be extended by honest gossip, so `GlobalBlock` can wait forever. The test was already probed as flaky on clean main in Feb (`2fbb0c6e3`); under Race Detection the hang shows up as a ~30m package timeout. ## What we test now 1. **3-of-4 committee mesh** via `env.Run` (giga avail client/server) — same as before. 2. **Equivocating peer**: same key as node0, **own** `data`/avail (`newTestNode`), **not** joined to the mesh (`testEnv` keys nodes by `PublicKey`, so a second node0 cannot be `AddNode`’d). The test calls `PushBlock` on node2 directly—same admission path gossip would use, without a real peer connection. The peer is never `Run`, so inject is bounded to `avail.BlocksPerLane` (initial lane capacity); `ProduceLocalBlock` fails if that bound is raised. 3. **Honest production** on each active node’s own lane, concurrent with the corrupt sender (`SpawnBg`). 4. **Agreement**: all nodes see the same `GlobalBlock` sequence and accept `PushAppHash`. No second `consensus.Run` on shared `nodes[0].data`. ## Why we wait for a non-empty honest tip If the corrupt peer’s genesis block lands on node2 while lane 0 is still empty: 1. Corrupt block 0 becomes the tip 2. Honest block 0 is ignored as stale 3. Honest block 1 fails parent-hash check and is dropped 4. The lane never advances → `GlobalBlock` hangs So the corrupt task blocks on `a2.Block(ctx, lane0, 0)` (tolerating `ErrPruned`) until node2 has an honest tip, then pushes its alternate chain. Those `PushBlock`s are then stale or parent-mismatch drops; the honest tip is preserved. This wait did **not** exist in sei-v3; it is required under today’s `PushBlock` semantics (sei-protocol#2925). ## Why main is flaky On main, dual `Run` + shared data + early corrupt/fake seeding races under `-race`. When node2 adopts a fork that honest gossip cannot extend, parent-hash drops leave sequencing stuck until the test package times out. Without `-race` / low `-count` it often still passes by luck (historically known-flaky). Verified locally: ```bash GOWORK=off go test ./internal/p2p/giga/ -race -count=30 -timeout 300s -run '^TestAvailClientServer$' ``` (30/30 after this change.) ## Test plan - [x] `GOWORK=off go test ./internal/p2p/giga/ -race -count=30 -timeout 300s -run '^TestAvailClientServer$'` - [x] CI Race Detection on this PR --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Summary
PushBlockdid not verify that the signer of aLaneProposalmatches the block's lane. A malicious validator could craft blocks on another validator's lane, bypassingVerifySig(which only checks the signer is a committee replica, not the lane producer). Added a cheap signer-lane comparison before expensive crypto verification.PushBlockdid not verify that the incoming block'sparentHashmatches the previous block's header hash. A malicious lane proposer could send a block with a fakeparentHash, which honest validators would accept and vote on. Whenheaders()later tries to reconstruct the block chain by walkingparentHashlinks backwards, it cannot find a matching vote and waits forever — deadlocking the node. Added parent hash verification when the previous block is available (after pruning, the check is safely skipped sinceheaders()never follows the first block'sparentHashin aLaneRange). A mismatch silently drops the block rather than returning an error, since equivocation (same key producing different chains) is a legitimate scenario.