feat: implement WebSocket streaming protocol for large payloads #2256

[netsirius]

Problem

The WebSocket interface between clients and the Freenet node used a single monolithic message frame for all payloads, requiring a 100 MB max_message_size limit to handle large WASM contract uploads. This caused excessive memory pressure on both client and server, since the entire payload had to be buffered in a single WebSocket frame before processing could begin.

Solution

Introduce a streaming protocol for WebSocket connections, using the chunking primitives from freenet-stdlib 0.2.0. Clients opt in via a ?streaming=true query parameter; non-streaming clients are completely unaffected.

How it works

All message types are chunked when large enough. The chunking decision is based purely on payload size — any serialized response or request that exceeds CHUNK_THRESHOLD (512 KiB) is automatically split into 256 KiB StreamChunk frames, regardless of the HostResponse or ClientRequest variant.

Server → Client (responses):
When streaming is enabled and a serialized response exceeds CHUNK_THRESHOLD, the server:

1. Optionally sends a StreamHeader (Native encoding only) with stream_id, total_bytes, and content metadata so the client can pre-allocate or show progress. Currently only GetResponse includes this header.
2. Splits the payload into StreamChunk messages via chunk_response() from freenet-stdlib.
3. Sends each chunk as a separate WebSocket binary frame.
4. Yields to the tokio runtime every MAX_CHUNKS_PER_BATCH chunks so pings and other control messages can flow through the select loop.

Client → Server (requests):
freenet-stdlib 0.2.2+ automatically chunks large serialized requests (>512 KiB) into StreamChunk frames on the client side, regardless of the streaming query parameter. The server always reassembles them using ReassemblyBuffer before decoding the full ClientRequest.

Chunking vs StreamHeader — two layers

Layer	Applies to	Purpose
Chunking (StreamChunk)	All message types over CHUNK_THRESHOLD	Fragment and transparently reassemble any large payload
StreamHeader (optional metadata)	Only GetResponse + Native encoding	Pre-announce total size and content type for incremental client consumption

The StreamHeader is an optimization that enables recv_stream() on the client — incremental chunk-by-chunk processing via an async Stream<Item = Bytes>. Without a header, the client's recv() transparently reassembles chunks and returns the complete HostResponse. Both paths deliver correct data; the header just enables richer client UX (progress bars, pre-allocation).

This two-layer design is intentional: GetResponse is the primary large-payload case (contract WASM up to 50 MiB + state up to 50 MiB) and the one where incremental consumption matters most. Other potentially large responses like UpdateNotification with full state are chunked and reassembled transparently. The StreamContent::Raw variant exists as a placeholder for future extension.

Key design decisions

• Opt-in streaming — backward compatible; existing clients work unchanged without the query parameter.
• max_message_size kept at 100 MB — non-streaming clients still send large payloads (WASM contracts) as single frames. The ?streaming=true parameter controls chunked responses only; the client-side stdlib always chunks large requests automatically.
• StreamHeader only for Native encoding — Flatbuffers clients (browser) use transparent reassembly via StreamChunk alone; the header is an optimization for bincode-based clients.
• Cooperative yielding — after every MAX_CHUNKS_PER_BATCH chunks sent, the sender yields to the tokio runtime to prevent starving pings, subscriptions, and other control flow in the connection's select loop.
• Wrapping stream IDs — per-connection u32 counter with wrapping_add; collisions would require 4B streams on a single connection.

Code changes

• ConnectionState struct — consolidates per-connection state (encoding protocol, streaming flag, reassembly buffer, stream ID counter) instead of passing individual parameters.
• send_response_message() — new helper that abstracts the "send as single frame or chunk" decision, used by all four response paths: host responses, subscription notifications, auth errors, and node-unavailable errors.
• decode_client_request() — extracted as an inner function to avoid duplicating deserialization logic between normal requests and post-reassembly decoded requests.
• EncodingProtocol now derives PartialEq, Eq — needed for conditional StreamHeader logic.

Bonus bugfix

The NodeUnavailable error path in process_host_response previously used bincode::serialize unconditionally, which would send malformed data to Flatbuffers clients. Now correctly serializes per the connection's encoding protocol.

Testing

• Streaming protocol unit tests (chunking, reassembly, edge cases) are in freenet-stdlib (freenet-stdlib#58).
• cargo test -p freenet passes (once freenet-stdlib 0.2.0 is published).

Dependencies

This PR must be merged after freenet-stdlib#58, which adds the streaming types (StreamChunk, StreamHeader, ReassemblyBuffer, chunk_response, etc.) in freenet-stdlib 0.2.0. CI will fail until that crate is published.

Files changed

File	Change
Cargo.toml	Bump freenet-stdlib to 0.2.0
Cargo.lock, apps/freenet-ping/Cargo.lock	Lockfile updates
crates/core/src/client_events/websocket.rs	Streaming protocol integration, ConnectionState, reassembly, chunked send across all response paths
crates/core/src/util/mod.rs	Add PartialEq, Eq to EncodingProtocol

Fixes

Closes #2256

[github-actions]

🔍 Rule Review: WebSocket streaming protocol (PR #3346)

Rules checked: git-workflow.md, code-style.md, testing.md
Files reviewed: 7 (apps/freenet-ping/Cargo.{lock,toml}, apps/freenet-ping/app/tests/common/mod.rs, apps/freenet-ping/app/tests/run_app.rs, apps/freenet-ping/types/src/lib.rs, crates/core/src/client_events/websocket.rs, crates/core/src/util.rs)

Critical

No critical findings.

Warnings

No warning-level findings.

Info

• crates/core/src/client_events/websocket.rs:691 — extract_stream_content uses a catch-all _ => None arm over HostResponse variants. If a future variant should produce StreamContent metadata it will silently return None with no compiler prompt. (rule: code-style.md — "Catch-all _ => in exhaustive matches")

---

Notes (not rule violations):

• send_response_message yields via tokio::task::yield_now() every MAX_CHUNKS_PER_BATCH chunks. The inline comment says this lets "pings, subscriptions, and other control flow in the connection's select loop" proceed, but yield_now only yields to other tasks in the executor, not to other arms of the same select!. The ping arm of the surrounding loop cannot run until send_response_message returns. The yield is still useful (it helps other spawned tasks) but the comment overstates what it accomplishes. Worth a wording fix, not a blocking issue.

• rand::rngs::StdRng::seed_from_u64(...) and tokio::time::sleep in the new integration test at apps/freenet-ping/app/tests/run_app.rs:168,310 are not in crates/core/, so the DST/GlobalRng/TimeSource rules do not apply there. ✓

• ConnectionState::reassembly (ReassemblyBuffer) is bounded: MAX_CONCURRENT_STREAMS and MAX_TOTAL_CHUNKS are enforced and tested in the new unit tests. ✓

• No tests were deleted or commented out; no bare .unwrap() in new production paths; no fire-and-forget spawns; no biased; added. ✓

To acknowledge findings: Check each box above once addressed (or a tracking issue opened),
or post /ack to dismiss all findings for this revision.
Merge is not blocked — the single INFO item is advisory only.

---

Automated review against .claude/rules/. Critical and Warning findings block merge — check boxes or post /ack to acknowledge.