Densepose Server Fusion error & Live Observatory issue

[itstayyabniazi]

My Workstation:

OS: Parrot-7 [7.0.9+parrot7-amd64 (64-bit)]
ESP Nodes: ESP32-WROOM-32 with 4 MB flash.
No. of Nodes: 3

When I open. http://localhost:3000/health .. it shows..
clients 3
source "esp32"
status "ok"
tick 16589

For Debugging i have added below line of code in,
RuView/v2/crates/wifi-densepose-sensing-server/src/main.rs
to check about the packets arriving from diff nodes..
// ADR-110: recover mesh-aligned timestamp for this CSI frame.

                let mesh_timestamp_us = ns.mesh_aligned_us_for_csi_frame(frame.sequence);
                
                ns.last_mesh_timestamp_us = mesh_timestamp_us;
                
                tracing::info!(
                    "node={} seq={} mesh_ts={:?}",
                    node_id,
                    frame.sequence,
                    ns.last_mesh_timestamp_us
                );

Densepose Server Logs:
server_log.txt
Here, the logs confirm all 3 nodes are active & sending CSI data to the server..

On the server terminal.. it keep on showing me when more then 1 ESP Nodes connect,
fusion error: Timestamp spread 130920 us exceeds guard interval 60000 us

Here are the individual ESP Nodes logs:
ESP Node 1: ESP_Node_1.txt

ESP Node 2: ESP_Node_2.txt

ESP Node 3: ESP_Node_3.txt

But the Body localization on observaotry.html are not correct.. while this test, we were 2 person in the room.

Here, below are the fix/changes that I have made.. but it also didn't removed that issue.

Multistatic Timestamp Issue: :
RuView_Multistatic_Timestamp_Issue.md

Multistatic Guard Configuration:
RuView_Multistatic_Guard_Configuration_Update.md

[ruvnet]

Thanks for the detailed report with logs.

Root cause: unsupported hardware

The ESP32-WROOM-32 (original ESP32, single-core) is not supported by this project. From the hardware compatibility table:

Not supported: ESP32 (original), ESP32-C3 — single-core, can't run CSI DSP pipeline.

Supported devices are:

• ESP32-S3 (8 MB flash, ~$9) — recommended
• ESP32-S3 SuperMini (4 MB, ~$6)
• ESP32-C6 + Seeed MR60BHA2 (for mmWave HR/BR)

Why the fusion error occurs

The error Timestamp spread 130920 us exceeds guard interval 60000 us means the per-node mesh-aligned timestamps differ by ~131 ms across your 3 nodes — more than twice the 60 ms guard. Multistatic fusion requires all nodes to report CSI frames within the same ~60 ms window so the server can treat them as a coherent snapshot of the same instant.

The original ESP32 doesn't implement the TDM mesh-clock protocol that the ESP32-S3 firmware uses to synchronise frames across nodes. Without that synchronisation, timestamps arrive with arbitrary offsets and fusion is rejected. This also explains the incorrect observatory localisation — the server falls back to single-node data rather than a fused multi-node view.

Path forward

Switching to ESP32-S3 boards and flashing the official firmware (see Tutorial #34) should resolve the timestamp spread and enable proper multistatic fusion.

---

Generated by Claude Code

[itstayyabniazi]

Now, I have shifted to ESP32-S3..
ESP Nodes: ESP32-S3-N16R8 with 8 MB flash.
No. of Nodes: 3

Densepose Server Logs:

Here, the logs confirm all 3 nodes are active & sending CSI data to the server..
updated_server_log.txt

On the server terminal.. Again it keep on showing me when more then 1 ESP Nodes connect,
fusion error: Timestamp spread 130920 us exceeds guard interval 60000 us

Here are the individual ESP Nodes logs:
ESP Node 1:
ESP32_S3_Node_1.txt

ESP Node 2:
ESP32_S3_Node_2.txt

ESP Node 3:
ESP32_S3_Node_3.txt

Also again the Body localization on observaotry.html are not correct...

What will be the possible root cause??

[ruvnet]

The ESP32-WROOM-32 (original ESP32) is not on the supported hardware list for this project — only ESP32-S3 and ESP32-C6 are supported. The original ESP32 is single-core and cannot reliably run the full CSI DSP pipeline, which includes concurrent Wi-Fi CSI capture and mesh timestamp alignment on separate cores.

The Timestamp spread 130920 us exceeds guard interval 60000 us error is a direct symptom of this: on a single-core device the scheduler interleaves the CSI capture task with everything else, producing large and unpredictable inter-frame gaps that exceed the multistatic fusion guard window.

Recommended path: Switch to ESP32-S3 (e.g., the N16R8 or DevKitC-1) or ESP32-C6-DevKitC-1 and flash the matching pre-built binary from the latest release. Either board runs the dual-core firmware correctly and should eliminate the timestamp spread error.

See the hardware table in the README and firmware/esp32-csi-node/README.md for supported boards and flash instructions.

---

Generated by Claude Code

[ruvnet]

Context note: the July 21 update above confirms the reporter has already switched from ESP32-WROOM-32 to ESP32-S3-N16R8 (dual-core — supported hardware). The timestamp spread error (130920 µs > 60 000 µs guard interval) persists on three S3-N16R8 nodes.

This may be worth connecting to #1401, which also reports ESP32-S3-N16R8 boards (5×) with CSI pipeline issues on the pre-built release binaries. The N16R8 uses 8 MB Octal PSRAM — a different PSRAM configuration than the N8R2/N8R8 variants the firmware was primarily tested on. Worth confirming whether the release binaries handle N16R8 PSRAM initialisation correctly before investigating the multi-node timestamp sync path further.

To help narrow the root cause: could you share whether the timestamp spread is consistent across sessions, or varies? And which release version of the firmware you flashed?

---

Generated by Claude Code

[ruvnet]

Thanks for the detailed debug steps. The root cause here is the hardware: ESP32-WROOM-32 uses the original ESP32 chip, which is not supported by RuView. The original ESP32 is single-core and cannot run the CSI DSP pipeline — this is why you're seeing 0 bytes in the fusion path and no skeleton in the Observatory even though UDP frames arrive.

Supported hardware:

Board	Flash	Notes
ESP32-S3 DevKit	8 MB	Recommended
ESP32-S3 SuperMini	4 MB	Compact; needs airflow (runs hot under continuous WiFi load)
ESP32-C6 + Seeed MR60BHA2	—	For mmWave HR/BR integration

Replacing your three nodes with ESP32-S3 boards and flashing the current release firmware should resolve both the fusion error and the missing skeleton. The rest of your setup (Parrot Linux, Docker, 3-node topology) looks correct.

---

Generated by Claude Code

[ruvnet]

The fusion errors and missing Observatory skeleton are causally linked — when fusion fails continuously, no persons array is published over WebSocket, so the Observatory shows connected nodes but no pose.

From docs/trust-and-engine-errors.md:

engine_error_count increments when StreamingEngine::process_cycle returns Err(EngineError::Fusion(..)). The most common cause in multi-node setups is TimestampMismatch — node timestamp spread exceeds the 60 ms hard guard. The log will print "governed trust cycle failed" (rate-limited to once per 10 s) with the actual spread value.

Quickest fix for persistent timestamp drift:

WDP_GUARD_INTERVAL_US=200000 wifi-densepose-sensing-server ...

This raises the hard guard to 200 ms without a code change (documented escape hatch for ESP32 nodes drifting 10–150 ms — multistatic_guard_config_from_env at main.rs:6791).

Other variants to rule out:

• InsufficientNodes — fewer than 2 nodes contributing frames. Fix: power cycle any node that shows associated but sends 0 CSI (TROUBLESHOOTING.md §1).
• DimensionMismatch — subcarrier count mismatch between nodes. Now rare since Multistatic fusion DimensionMismatch flood: live bridge feeds raw (un-canonicalized) per-node CSI to the fuser #1170 canonicalized to a 56-tone grid, but can still occur on mixed HT/HE captures.

Check GET /health/ready for engine_error_count and restart the server to reset the counter before re-testing after any change (it's monotonically increasing with no clear path short of a restart).

---

Generated by Claude Code