feat(eval): integrate LIBERO-PRO code-policy benchmark

[TomCC7]

Contribution path

• Linked PR: feat(eval): add live-agent VLN-CE benchmark #3471
• Final PR in the evaluation stack; merge feat(eval): add live-agent VLN-CE benchmark #3471 first.

Problem

DimOS needs a LIBERO-PRO evaluation whose score remains comparable with the original benchmark while exercising the same unified, realtime policy interface intended for physical robots. LIBERO and its simulator dependencies must stay out of the host Python environment, and submitted policies must not access benchmark lifecycle or scoring state.

Solution

Integrate one end-to-end LIBERO-PRO single-trial evaluation on top of #3471:

• run every debug submission and the scored policy in a fresh DimOS blueprint and rootless Podman container
• expose normal robot observations and joint targets over a policy-only gRPC/protobuf interface
• expose initialization, trial lifecycle, and native scoring through a separately authenticated privileged interface
• mount revision-pinned, checksum-verified BDDL and initial-state assets into each container
• preserve task selection, initial state, settling steps, 20 Hz realtime execution, horizon, reward, and binary native score
• record public camera streams and joint state to Memory2 and generate a side-by-side MP4 for every trial
• keep LIBERO environment creation, stepping, rendering, and teardown on one simulator thread so EGL frames remain valid
• register the code-policy-v1 runtime, strict smoke case, CLI coverage, generated blueprints, documentation, and architecture decisions

The first integration evaluates one initialization of one task. Parallel and batched execution remain out of scope.

This PR remains draft until a smoke policy produces nonzero arm-joint motion. Trial completion, a score, and gripper-only motion do not satisfy that readiness gate.

How to Test

uv sync --extra agents
npm --prefix packages/pi-code-policy-extension install
npm --prefix packages/pi-code-policy-extension run build

dimos eval run \
  dimos/benchmark/libero_pro/cases/goal-task-0-single-trial/evaluation.json \
  --output /tmp/dimos-libero-pro-smoke \
  --json --quiet

Focused checks:

uv run pytest \
  dimos/benchmark/libero_pro \
  dimos/benchmark/evaluation \
  dimos/cli/test_eval.py \
  dimos/robot/test_all_blueprints_generation.py -q
uv run --group lint mypy \
  dimos/benchmark/evaluation \
  dimos/benchmark/libero_pro \
  dimos/cli/eval.py

Validated after restacking: 46 focused tests pass, the generated blueprint registry is current, and mypy reports no issues across 21 source files. The prior Podman smoke completed 300 policy ticks, emitted a native 0.0 score, and produced a valid 301-frame, 20 FPS MP4; it did not move the arm and therefore does not clear the readiness gate.

AI assistance

Codex with GPT-5.6 assisted with design discussion, implementation, diagnosis, stack restructuring, tests, documentation, and PR drafting. The author reviewed the benchmark-parity and runtime decisions interactively.

Checklist

• I have read and approved the CLA.

[codecov]

❌ 3 Tests Failed:

Tests completed	Failed	Passed	Skipped
4274	3	4271	69

View the top 2 failed test(s) by shortest run time

dimos.e2e_tests.test_dimsim_path_replaning::test_path_replanning

Stack Traces | 108s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f6b18d79f70>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7f6b186607c0>
dim_sim = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7f6be3a001d0>
direct_cmd_vel_explorer = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f6b1903f980>
spawn_wall_on_pose = <function spawn_wall_on_pose.<locals>.spawn at 0x7f6b18660ea0>

    @pytest.mark.self_hosted_large
    def test_path_replanning(
        lcm_spy, start_blueprint, dim_sim, direct_cmd_vel_explorer, spawn_wall_on_pose
    ) -> None:
        start_blueprint(
            "--dimsim-scene=empty",
            "run",
            "unitree-go2-agentic",
            simulator="dimsim",
        )
        lcm_spy.save_topic(".../McpClient/on_system_modules/res")
        lcm_spy.wait_for_saved_topic(".../McpClient/on_system_modules/res", timeout=1200.0)
    
        # robot spawns at (3, 2)
    
        # side wall
        dim_sim.add_wall(2, -2.5, 12, -2.5)
        # other side wall
        dim_sim.add_wall(2, 3.5, 12, 3.5)
        # back wall (behind robot)
        dim_sim.add_wall(2, -2.5, 2, 3.5)
        # forward wall (far end)
        dim_sim.add_wall(12, -2.5, 12, 3.5)
        # dividing wall at x=7 with doors at y=[-1.5,-0.5] and y=[1.5,2.5]
        dim_sim.add_wall(7, -2.5, 7, -1.5)
        dim_sim.add_wall(7, -0.5, 7, 1.5)
        dim_sim.add_wall(7, 2.5, 7, 3.5)
    
        direct_cmd_vel_explorer.linear_speed = 0.8
>       direct_cmd_vel_explorer.follow_points([(10, 2), (2.5, 2), (3, 2)])

dim_sim    = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7f6be3a001d0>
direct_cmd_vel_explorer = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f6b1903f980>
lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f6b18d79f70>
spawn_wall_on_pose = <function spawn_wall_on_pose.<locals>.spawn at 0x7f6b18660ea0>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7f6b186607c0>

dimos/e2e_tests/test_dimsim_path_replaning.py:47: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../simulation/mujoco/direct_cmd_vel_explorer.py:105: in follow_points
    self._wait_for_pose()
        self       = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f6b1903f980>
        waypoints  = [(10, 2), (2.5, 2), (3, 2)]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f6b1903f980>

    def _wait_for_pose(self) -> PoseStamped:
        self._new_pose.clear()
        self._new_pose.wait(timeout=5.0)
>       assert self._pose is not None, "No odom received"
E       AssertionError: No odom received

self       = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f6b1903f980>

.../simulation/mujoco/direct_cmd_vel_explorer.py:65: AssertionError

dimos.e2e_tests.test_dimsim_walk_forward::test_walk_forward

Stack Traces | 194s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f6b19630470>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7f6b18a72a20>
human_input = <function human_input.<locals>.send_human_input at 0x7f6b18a72ac0>
dim_sim = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7f6b71f4ff80>

    @pytest.mark.self_hosted_large
    def test_walk_forward(lcm_spy, start_blueprint, human_input, dim_sim) -> None:
        start_blueprint(
            "run",
            "--disable",
            "spatial-memory",
            "--disable",
            "security-module",
            "unitree-go2-agentic",
            simulator="dimsim",
        )
        lcm_spy.save_topic(".../McpClient/on_system_modules/res")
        lcm_spy.wait_for_saved_topic(".../McpClient/on_system_modules/res", timeout=1200.0)
    
        origin_x, origin_y = 1, 2
        dim_sim.set_agent_position(origin_x, origin_y)
    
        human_input("move forward 3 meter")
    
>       lcm_spy.wait_until_odom_position(origin_x + 3, origin_y, threshold=0.4, timeout=120)

dim_sim    = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7f6b71f4ff80>
human_input = <function human_input.<locals>.send_human_input at 0x7f6b18a72ac0>
lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f6b19630470>
origin_x   = 1
origin_y   = 2
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7f6b18a72a20>

dimos/e2e_tests/test_dimsim_walk_forward.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/e2e_tests/lcm_spy.py:167: in wait_until_odom_position
    self.wait_for_message_result(
        predicate  = <function LcmSpy.wait_until_odom_position.<locals>.predicate at 0x7f6b18a737e0>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f6b19630470>
        threshold  = 0.4
        timeout    = 120
        x          = 4
        y          = 2
dimos/e2e_tests/lcm_spy.py:153: in wait_for_message_result
    wait_until(
        event      = <threading.Event at 0x7f6c50edec60: unset>
        fail_message = 'Failed to get to position x=4, y=2'
        listener   = <function LcmSpy.wait_for_message_result.<locals>.listener at 0x7f6b18a73ba0>
        predicate  = <function LcmSpy.wait_until_odom_position.<locals>.predicate at 0x7f6b18a737e0>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f6b19630470>
        timeout    = 120
        topic      = '/odom#geometry_msgs.PoseStamped'
        type       = <class 'dimos.msgs.geometry_msgs.PoseStamped.PoseStamped'>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

predicate = <bound method Event.is_set of <threading.Event at 0x7f6c50edec60: unset>>

    def wait_until(
        predicate: Callable[[], bool],
        *,
        timeout: float,
        interval: float = 0.1,
        message: str | None = None,
    ) -> None:
        """Poll ``predicate`` until it returns truthy or ``timeout`` elapses."""
        deadline = time.monotonic() + timeout
        while time.monotonic() < deadline:
            if predicate():
                return
            time.sleep(interval)
>       raise TimeoutError(message or f"Timed out after {timeout}s waiting for condition")
E       TimeoutError: Failed to get to position x=4, y=2

deadline   = 3211845.539660169
interval   = 0.1
message    = 'Failed to get to position x=4, y=2'
predicate  = <bound method Event.is_set of <threading.Event at 0x7f6c50edec60: unset>>
timeout    = 120

.../utils/testing/waiting.py:35: TimeoutError

View the full list of 1 ❄️ flaky test(s)

dimos.e2e_tests.test_dimsim_spatial_memory::test_go_to_the_bed

Flake rate in main: 16.11% (Passed 125 times, Failed 24 times)

Stack Traces | 99.1s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f6b19113080>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7f6b18a725c0>
human_input = <function human_input.<locals>.send_human_input at 0x7f6b18a727a0>
dim_sim = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7f6b19666330>
explore_house = <function explore_house.<locals>.explore at 0x7f6b18a72e80>

    @pytest.mark.self_hosted_large
    def test_go_to_the_bed(lcm_spy, start_blueprint, human_input, dim_sim, explore_house) -> None:
        start_blueprint(
            "run",
            "unitree-go2-agentic",
            simulator="dimsim",
        )
        lcm_spy.save_topic(".../McpClient/on_system_modules/res")
        lcm_spy.wait_for_saved_topic(".../McpClient/on_system_modules/res", timeout=1200.0)
    
>       explore_house()

dim_sim    = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7f6b19666330>
explore_house = <function explore_house.<locals>.explore at 0x7f6b18a72e80>
human_input = <function human_input.<locals>.send_human_input at 0x7f6b18a727a0>
lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f6b19113080>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7f6b18a725c0>

dimos/e2e_tests/test_dimsim_spatial_memory.py:28: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/e2e_tests/conftest.py:249: in explore
    direct_cmd_vel_explorer.follow_points(points)
        direct_cmd_vel_explorer = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f6b198145f0>
        points     = [(3.881, 4.803), (4.16, 1.615), (1.596, 1.505), (1.649, 0.137), (-3.644, -0.064), (-3.759, -2.661), ...]
.../simulation/mujoco/direct_cmd_vel_explorer.py:105: in follow_points
    self._wait_for_pose()
        self       = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f6b198145f0>
        waypoints  = [(3.881, 4.803), (4.16, 1.615), (1.596, 1.505), (1.649, 0.137), (-3.644, -0.064), (-3.759, -2.661), ...]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f6b198145f0>

    def _wait_for_pose(self) -> PoseStamped:
        self._new_pose.clear()
        self._new_pose.wait(timeout=5.0)
>       assert self._pose is not None, "No odom received"
E       AssertionError: No odom received

self       = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f6b198145f0>

.../simulation/mujoco/direct_cmd_vel_explorer.py:65: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}