Skip to content

feat(scene): add AiBOX NPU support and split Webots deployment - #324

Open
li041 wants to merge 9 commits into
syswonder:devfrom
li041:scene-npu
Open

li041 wants to merge 9 commits into
syswonder:devfrom
li041:scene-npu

Conversation

@li041

@li041 li041 commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Add RKNPU3 support for Scene’s DualMap perception on AIBOX and provide a split Webots/AIBOX deployment example for TIAGo.

Changes

  • Add tools to prepare the AIBOX model-conversion environment, export and build RKNN models, assemble the model bundle, and test model outputs on the board.
  • Add an NPU mode to DualMap that runs supported model graphs on RKNPU3 and keeps the CLIP text tower on the CPU. Check required model files before starting native Scene.
  • Add examples/webots-aibox with a deployment manifest, remote TIAGo primitive wrappers, configuration templates, and setup instructions.
  • Increase the rbnx provider registration timeout from 60 to 180 seconds to accommodate slower startup.
  • Document the Scene NPU settings.

Validation

  • Add scripts for RKNPU3 model smoke tests on AiBOX.
  • Ran the Scene tests in the Webots office world.

Based on #311

@github-actions github-actions Bot added type:feature New feature (feat:) comp:scene system/scene comp:rbnx tools/rbnx comp:docs docs/ and READMEs labels Nov 15, 2026

@enkerewpo enkerewpo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding support for the new AIBOX hardware!

Tested 2b21d57 merged onto dev 39ab370 locally: bash -n on the 7 changed shell scripts, py_compile on the 7 Python files and the Scene unit tests (24 for AIBOX conversion and the DualMap detector, 61 for prepare_*) all pass; cargo fmt --check passes, rbnx clippy -D warnings is clean and the rbnx tests pass (39 + 37, 1 ignored each). I could not run the RKNPU3 path or the split Webots/AIBOX deployment.

A few things to fix are noted inline. This PR includes everything in #311, so #311 can be closed once this lands.

Comment thread system/scene/README.md Resolved
Comment thread tools/rbnx/src/cmd/deploy.rs Resolved
Deploy the four TIAGo primitives (camera, chassis, lidar, health) from
the board while their drivers keep running on the Ubuntu sim host: each
package manifest forwards build/start/stop over SSH to the sim host's
robonix checkout (remote_build.sh runs codegen there, start.sh
docker-execs the driver into the running sim container and re-exports
RBNX_DEPLOY_MANAGED so boot still owns the lifecycle, stop.sh sweeps
remote leftovers). Machine-specific facts (ssh alias, sim-host IP,
Atlas endpoint) go into remote.env, a local copy of
remote.env.example; the wrapper scripts fail with a pointer to the
example when it is missing.
One `rbnx boot` on the board brings up the full split stack: the system
components (atlas, soma, executor, pilot, liaison, vitals), mapping and
nav2 native on the board, the explore skill, and the four SSH-wrapped
TIAGo primitives. Includes boot.env (board-side Zenoh/VLM wiring sourced
from run/*.env), the Nav2 parameter file, and mapping/explore configs
tuned for the TIAGo office world; the README ties the two machines
together.
start_scene_npu_aibox.sh exports the DualMap RKNPU3 model paths and
ROBONIX_SCENE_FORCE before delegating to start.sh; start_native.sh
validates those SCENE_DUALMAP_* variables and skips the ONNX model
check when NPU mode is requested.
DualMap runs its perception graphs on the AIBOX NPU (RK1828). This adds
the tooling to provision both ends, convert the graphs and verify them
on the board.

prepare_aibox_npu_env.sh provisions both ends from scratch. The toolkit
release follows the board firmware's rknn3 API version as reported by
rknn-smi; wheels are md5-checked against the release manifest and a
stale version on either end is reinstalled with a warning.

convert_aibox_models.py exports the yoloworld detector (vocabulary
baked via --classes-file), the MobileSAM encoder and the SamOnnxModel
decoder from the pinned li041/MobileSAM checkout (--mobilesam-repo;
the build drops the orig-size resize and the iou output, keeping the
low-res logits whose token 0 the runtime reads), compiles fp16 rk1820
bundles with a simulator check, and assembles the bundle directory —
npu_meta.json carries the classes list and the pre_nms decode-path
selector; --dualmap-repo clones the pinned runtime checkout alongside.
fetch_zoo_aibox_models.py pulls the zoo CLIP graphs and drives the zoo
convert.py for the same target; the mobileclip self-export route is
dropped — the zoo clip graph is the deployed one.

test_aibox_rknn_lite.py runs one graph on the board NPU and reports
cosine against a reference npz (--inputs replays reference inputs for
the decoder, which has no image input). test_aibox_models.sh deploys
every converted graph under the workdir and runs that test per model.
Setting SCENE_DUALMAP_NPU_DIR switches the detector to the RKNPU3
bundle and keeps only the CLIP text tower on torch, after checking
npu_meta.json, the MobileSAM decoder pair and the ViT-B-32 weights.
Without the variable the CPU path is unchanged.
wait_for_registration waits the 60 s default for a freshly spawned
package to register its driver capability with atlas;
ROBONIX_DRIVER_REGISTER_TIMEOUT_S overrides it for deployments whose
providers register slowly — same pattern as ROBONIX_DRIVER_INIT_TIMEOUT_S.
The AIBOX boot env sets 180: the four TIAGo primitives register through
SSH wrappers on the sim host, which can exceed the default on a cold start.
Add the scene service entry to robonix_manifest.yaml, the
scene-aibox-npu.yaml config, the SCENE_* model paths in boot.env, and
update the README (boot layering, known trade-offs, sim GUI flag, the
Scene NPU models section, the li041/DualMap and li041/MobileSAM pins,
and the prebuilt aibox-npu-bundle release pointer).

Re-copy the aibox nav2 params from the current webots example so the
recovery BT XML (passed via bt_xml_file), footprint clearing and the
0.1 m planner tolerance return; only default_server_timeout (200 ms
for the cross-machine rmw_zenoh goal acks) stays deployment-specific.
detect() required the Jetson BSP files unconditionally, so rbnx build
with the jetson-native manifest failed on the Rockchip AIBOX. l4t and
model become optional host facts — only the jetson-docker target
consumes l4t, and select_profile falls through to its unsupported-host
error when they are absent. The AIBOX boot env now sets
SCENE_CG_FORCE_CPU=1 so the native build installs CPU-only torch; the
NPU graphs are unaffected.
@li041
li041 requested a review from Chenyantt as a code owner November 15, 2026 16:13
@github-actions github-actions Bot added comp:soma system/soma comp:api pylib/robonix-api labels Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:api pylib/robonix-api comp:docs docs/ and READMEs comp:rbnx tools/rbnx comp:scene system/scene comp:soma system/soma type:feature New feature (feat:)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants