Repository navigation
Conversation
enkerewpo
reviewed
Nov 15, 2026
enkerewpo
left a comment
Member
There was a problem hiding this comment.
Thanks for adding support for the new AIBOX hardware!
Tested 2b21d57 merged onto dev 39ab370 locally: bash -n on the 7 changed shell scripts, py_compile on the 7 Python files and the Scene unit tests (24 for AIBOX conversion and the DualMap detector, 61 for prepare_*) all pass; cargo fmt --check passes, rbnx clippy -D warnings is clean and the rbnx tests pass (39 + 37, 1 ignored each). I could not run the RKNPU3 path or the split Webots/AIBOX deployment.
A few things to fix are noted inline. This PR includes everything in #311, so #311 can be closed once this lands.
Deploy the four TIAGo primitives (camera, chassis, lidar, health) from the board while their drivers keep running on the Ubuntu sim host: each package manifest forwards build/start/stop over SSH to the sim host's robonix checkout (remote_build.sh runs codegen there, start.sh docker-execs the driver into the running sim container and re-exports RBNX_DEPLOY_MANAGED so boot still owns the lifecycle, stop.sh sweeps remote leftovers). Machine-specific facts (ssh alias, sim-host IP, Atlas endpoint) go into remote.env, a local copy of remote.env.example; the wrapper scripts fail with a pointer to the example when it is missing.
One `rbnx boot` on the board brings up the full split stack: the system components (atlas, soma, executor, pilot, liaison, vitals), mapping and nav2 native on the board, the explore skill, and the four SSH-wrapped TIAGo primitives. Includes boot.env (board-side Zenoh/VLM wiring sourced from run/*.env), the Nav2 parameter file, and mapping/explore configs tuned for the TIAGo office world; the README ties the two machines together.
start_scene_npu_aibox.sh exports the DualMap RKNPU3 model paths and ROBONIX_SCENE_FORCE before delegating to start.sh; start_native.sh validates those SCENE_DUALMAP_* variables and skips the ONNX model check when NPU mode is requested.
DualMap runs its perception graphs on the AIBOX NPU (RK1828). This adds the tooling to provision both ends, convert the graphs and verify them on the board. prepare_aibox_npu_env.sh provisions both ends from scratch. The toolkit release follows the board firmware's rknn3 API version as reported by rknn-smi; wheels are md5-checked against the release manifest and a stale version on either end is reinstalled with a warning. convert_aibox_models.py exports the yoloworld detector (vocabulary baked via --classes-file), the MobileSAM encoder and the SamOnnxModel decoder from the pinned li041/MobileSAM checkout (--mobilesam-repo; the build drops the orig-size resize and the iou output, keeping the low-res logits whose token 0 the runtime reads), compiles fp16 rk1820 bundles with a simulator check, and assembles the bundle directory — npu_meta.json carries the classes list and the pre_nms decode-path selector; --dualmap-repo clones the pinned runtime checkout alongside. fetch_zoo_aibox_models.py pulls the zoo CLIP graphs and drives the zoo convert.py for the same target; the mobileclip self-export route is dropped — the zoo clip graph is the deployed one. test_aibox_rknn_lite.py runs one graph on the board NPU and reports cosine against a reference npz (--inputs replays reference inputs for the decoder, which has no image input). test_aibox_models.sh deploys every converted graph under the workdir and runs that test per model.
Setting SCENE_DUALMAP_NPU_DIR switches the detector to the RKNPU3 bundle and keeps only the CLIP text tower on torch, after checking npu_meta.json, the MobileSAM decoder pair and the ViT-B-32 weights. Without the variable the CPU path is unchanged.
wait_for_registration waits the 60 s default for a freshly spawned package to register its driver capability with atlas; ROBONIX_DRIVER_REGISTER_TIMEOUT_S overrides it for deployments whose providers register slowly — same pattern as ROBONIX_DRIVER_INIT_TIMEOUT_S. The AIBOX boot env sets 180: the four TIAGo primitives register through SSH wrappers on the sim host, which can exceed the default on a cold start.
Add the scene service entry to robonix_manifest.yaml, the scene-aibox-npu.yaml config, the SCENE_* model paths in boot.env, and update the README (boot layering, known trade-offs, sim GUI flag, the Scene NPU models section, the li041/DualMap and li041/MobileSAM pins, and the prebuilt aibox-npu-bundle release pointer). Re-copy the aibox nav2 params from the current webots example so the recovery BT XML (passed via bt_xml_file), footprint clearing and the 0.1 m planner tolerance return; only default_server_timeout (200 ms for the cross-machine rmw_zenoh goal acks) stays deployment-specific.
detect() required the Jetson BSP files unconditionally, so rbnx build with the jetson-native manifest failed on the Rockchip AIBOX. l4t and model become optional host facts — only the jetson-docker target consumes l4t, and select_profile falls through to its unsupported-host error when they are absent. The AIBOX boot env now sets SCENE_CG_FORCE_CPU=1 so the native build installs CPU-only torch; the NPU graphs are unaffected.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add RKNPU3 support for Scene’s DualMap perception on AIBOX and provide a split Webots/AIBOX deployment example for TIAGo.
Changes
examples/webots-aiboxwith a deployment manifest, remote TIAGo primitive wrappers, configuration templates, and setup instructions.rbnxprovider registration timeout from 60 to 180 seconds to accommodate slower startup.Validation
officeworld.Based on #311