diff --git a/CONTEXT.md b/CONTEXT.md
index 8dcceec696..6e7b7bbda4 100644
--- a/CONTEXT.md
+++ b/CONTEXT.md
@@ -8,6 +8,26 @@ The language used to describe how DimOS measures task-performing behavior across
A complete benchmark integration that owns its environment lifecycle, cases, scoring, aggregation, and native result semantics. Third-party evaluations retain their original harness rather than translating it into DimOS scoring primitives.
_Avoid_: Runtime, universal scorer
+**Comparable LIBERO-PRO Evaluation**:
+An Evaluation that preserves LIBERO-PRO's tasks, observations, action semantics, initial states, native success, and reporting while declaring any policy-runtime deviations that affect direct score comparison.
+_Avoid_: Official LIBERO-PRO evaluation, protocol-identical score
+
+**CaP-X-Aligned LIBERO Policy Surface**:
+The non-privileged observations and robot capabilities available to code policies in CaP-X's reduced LIBERO environment, expressed through ordinary DimOS modules and messages. It includes calibrated RGB-D cameras, camera-to-robot transforms, robot state, perception, inverse kinematics, and executable arm and gripper motion, but excludes simulator object poses and native success state.
+_Avoid_: CaP-X privileged API, simulator oracle, identical CaP-X implementation
+
+**Unified Real-Time Policy**:
+A Policy Artifact that uses the same ordinary DimOS interface in simulation and on a real robot while the environment continues advancing independently of policy computation latency.
+_Avoid_: Benchmark-stepped policy, simulator-specific policy
+
+**Real-Time Horizon**:
+The fixed number of simulator control ticks available to a Unified Real-Time Policy after startup and settling complete. Every tick consumes the horizon whether or not the policy produces a new command.
+_Avoid_: Policy-call budget, wall-clock timeout
+
+**LIBERO-PRO Trial**:
+One scored policy episode against a selected LIBERO-PRO task and fresh initial state, producing the benchmark's native result and a side-by-side MP4 of the public camera observations. A trial validates the integration but is not by itself a comparable aggregate benchmark score.
+_Avoid_: LIBERO-PRO benchmark result, aggregate score
+
**Evaluation Case**:
A fixed task, environment, budget, and scoring definition. It does not select the behavior being evaluated.
_Avoid_: Policy configuration, run
@@ -25,21 +45,45 @@ The callable produced during exploration and replayed without an agent during ev
_Avoid_: Agent, exploration transcript, execution mode
**Policy Artifact**:
-The serialized callable and human-readable source captured when `submit_policy(policy)` is called. One artifact is produced per benchmark task and reused across its held-out evaluation cases or seeds. REPL outputs and the agent transcript are separate exploration evidence.
+The serialized callable and human-readable source captured when `submit_policy(policy)` is called. One artifact is produced per benchmark task and reused across fresh evaluation episodes or seeds of that task. REPL outputs and the agent transcript are separate exploration evidence.
_Avoid_: Exploration transcript, policy source
**Exploration Stage**:
-The unscored stage in which an agent uses a persistent Python REPL and calls `submit_policy(policy)` to run complete debug trials in fresh environments and blueprints. Model latency does not consume the evaluation horizon.
+The unscored training stage in which an agent receives the selected benchmark task's exact instruction, uses a persistent Python REPL, and calls `submit_policy(policy)` to run complete debug trials of that task in fresh environments and blueprints. It is part of the evaluated code-as-policy system, while model latency remains outside the evaluation horizon.
_Avoid_: Evaluation rollout, scoring
+**Task-Aligned Exploration**:
+An Exploration Stage whose debug trials and scored Evaluation Stage use the same benchmark suite, task identity, and task instruction. Episodes and initial states are fresh, but the policy is not asked to transfer to an undisclosed task.
+_Avoid_: Hidden-task evaluation, unperturbed training suite
+
+**Debug Trial**:
+One unscored policy attempt created by `submit_policy()`, owning a fresh complete DimOS blueprint and a fresh episode of the selected benchmark task. It returns a stopped `TrialRun` snapshot containing the outcome, logs, Memory2 recording, and artifacts.
+_Avoid_: Shared simulator reset, partial run, evaluation rollout
+
**Evaluation Stage**:
-The measured stage in which the Policy Artifact executes without an agent against a reset or held-out environment. The native benchmark owns its real-time or step horizon and privileged scoring.
+The measured stage in which the Policy Artifact executes without an agent in a fresh episode of the same suite and task disclosed during Task-Aligned Exploration. The task instruction is unchanged, the episode initial state is fresh, and the evaluation owns the declared real-time horizon while native privileged scoring remains benchmark-owned and unavailable to the policy.
_Avoid_: Agent session, policy generation
**Policy Environment**:
-The capabilities exposed by the fresh policy-only DimOS blueprint while a policy runs. Simulated, live, and replay-backed evaluations all pass the policy a connected `Dimos` application; completed trials additionally expose their Memory2 recording read-only through `TrialRun`.
+The capabilities exposed by the fresh policy-only DimOS blueprint while a policy runs. Simulated, live, and replay-backed evaluations all pass the policy a connected `Dimos` application whose `app.memory` property is the active read-only Memory2 store. Completed trials expose the same recording through `TrialRun`.
_Avoid_: Agent tools, scorer context
+**RGB-D Observation**:
+A timestamp-aligned bundle of one public color image, metric depth image, exact camera calibration, and world-to-optical transform selected from Memory2. It is sensor evidence chosen by the policy, not a persistent scene model or simulator snapshot.
+_Avoid_: Scene façade, latest-camera RPC, simulator observation
+
+**Grounded Segmentation**:
+An ordinary on-demand DimOS perception capability that accepts an explicit timestamped image and text descriptions and returns typed image masks. It has no camera stream subscription and can operate identically on live, simulated, replayed, or post-trial images.
+_Avoid_: LIBERO perception, implicit latest frame, privileged object mask
+
+**Policy Interface**:
+The non-privileged robot observations, state, commands, and operational health that a simulator or real robot connection contributes to the complete Policy Environment.
+_Avoid_: Normal API, evaluator interface
+
+**Evaluation Control Interface**:
+The evaluator-only capabilities for environment configuration, reset, initial-state selection, clock control, native terminal state, and scoring. It is not part of the Policy Environment.
+_Avoid_: Privileged API, robot interface
+
**Evaluation Oracle**:
The evaluator-only source of truth used to score a policy attempt. In simulation it contains privileged state, such as true poses and object identities, that the Policy Environment cannot access.
_Avoid_: Runtime memory, perception output
diff --git a/dimos/agents/code_policy_core.py b/dimos/agents/code_policy_core.py
index e1df3aa538..8365387558 100644
--- a/dimos/agents/code_policy_core.py
+++ b/dimos/agents/code_policy_core.py
@@ -144,9 +144,10 @@ def _bootstrap_source(environment: SubmissionEnvironment | LiveDimosEnvironment)
from dimos.memory2.store.sqlite import SqliteStore
from dimos.porcelain.dimos import Dimos
-memory = SqliteStore(path={environment.recording_path!r}, must_exist=True, read_only=True)
-memory.start()
-app = Dimos.connect()
+app = Dimos.connect(
+ memory=SqliteStore(path={environment.recording_path!r}, must_exist=True, read_only=True)
+)
+app.memory.start()
"""
diff --git a/dimos/agents/test_code_policy_core.py b/dimos/agents/test_code_policy_core.py
index bee50ba06f..79ccc7ed94 100644
--- a/dimos/agents/test_code_policy_core.py
+++ b/dimos/agents/test_code_policy_core.py
@@ -111,8 +111,10 @@ def test_live_repl_bootstraps_public_runtime_without_credentials(
source = _bootstrap_source(environment)
kernel_environment = _kernel_environment(config)
- assert "memory = SqliteStore" in source
- assert "app = Dimos.connect()" in source
+ assert "app = Dimos.connect(" in source
+ assert "memory=SqliteStore" in source
+ assert "\nmemory =" not in source
+ assert "app.memory.start()" in source
assert "submit_policy" not in source
assert "OPENAI_API_KEY" not in kernel_environment
assert kernel_environment["ORDINARY_SETTING"] == "retained"
diff --git a/dimos/benchmark/evaluation/protocol.py b/dimos/benchmark/evaluation/protocol.py
index ace25581b0..aab4161c1c 100644
--- a/dimos/benchmark/evaluation/protocol.py
+++ b/dimos/benchmark/evaluation/protocol.py
@@ -46,7 +46,7 @@ class PolicyArtifact:
@dataclass(frozen=True)
class PolicyExecution:
- status: Literal["completed", "policy_error", "timed_out", "infrastructure_error"]
+ status: Literal["completed", "policy_error", "stopped", "infrastructure_error"]
duration_seconds: float
error: str | None = None
@@ -165,12 +165,22 @@ def explore(
max_submissions: int = 5,
) -> ExplorationOutcome: ...
- def execute(
+ def prepare(
self,
policy: PolicyArtifact,
*,
- timeout_s: float,
- ) -> PolicyExecution: ...
+ memory_path: Path,
+ startup_timeout_s: float,
+ ) -> PolicyExecutionHandle: ...
+
+
+@runtime_checkable
+class PolicyExecutionHandle(Protocol):
+ """A loaded policy waiting behind an evaluator-owned start gate."""
+
+ def start(self) -> None: ...
+
+ def finish(self, *, grace_s: float = 1.0) -> PolicyExecution: ...
@dataclass(frozen=True)
diff --git a/dimos/benchmark/evaluation/registry.py b/dimos/benchmark/evaluation/registry.py
index 716917d294..c7dc2a2e4a 100644
--- a/dimos/benchmark/evaluation/registry.py
+++ b/dimos/benchmark/evaluation/registry.py
@@ -30,6 +30,7 @@
ENTRY_POINT_GROUP = "dimos.evaluations"
LOCAL_NAME_PATTERN = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$")
BUILTIN_EVALUATIONS: dict[str, str] = {
+ "libero-pro": "dimos.benchmark.libero_pro.evaluation:libero_pro",
"vlnce-r2r": "dimos.benchmark.vlnce_r2r.evaluation:vlnce_r2r",
}
diff --git a/dimos/benchmark/evaluation/runtime.py b/dimos/benchmark/evaluation/runtime.py
index 2b39705420..1fc6784c5d 100644
--- a/dimos/benchmark/evaluation/runtime.py
+++ b/dimos/benchmark/evaluation/runtime.py
@@ -20,7 +20,6 @@
import json
import multiprocessing
from pathlib import Path
-import queue
import shutil
import time
from typing import Any, Literal
@@ -34,6 +33,7 @@
ExplorationOutcome,
PolicyArtifact,
PolicyExecution,
+ PolicyExecutionHandle,
TrialRun,
)
@@ -195,52 +195,41 @@ def explore(
server.stop()
shutil.rmtree(path / "working", ignore_errors=True)
- def execute(self, policy: PolicyArtifact, *, timeout_s: float) -> PolicyExecution:
- if timeout_s <= 0:
- raise ValueError("timeout_s must be positive")
- started = time.monotonic()
+ def prepare(
+ self,
+ policy: PolicyArtifact,
+ *,
+ memory_path: Path,
+ startup_timeout_s: float,
+ ) -> PolicyExecutionHandle:
+ if startup_timeout_s <= 0:
+ raise ValueError("startup_timeout_s must be positive")
context = multiprocessing.get_context("spawn")
- result_queue = context.Queue(maxsize=1)
+ messages, worker_messages = context.Pipe(duplex=False)
+ start_event = context.Event()
process = context.Process(
target=_execute_policy_worker,
- args=(str(policy.serialized_path), result_queue),
+ args=(str(policy.serialized_path), str(memory_path), worker_messages, start_event),
daemon=True,
)
try:
process.start()
- process.join(timeout_s)
+ worker_messages.close()
+ if not messages.poll(startup_timeout_s):
+ _stop_process(process)
+ raise TimeoutError(f"policy startup timed out after {startup_timeout_s:g}s")
+ message, error = messages.recv()
+ if message != "ready":
+ _stop_process(process)
+ messages.close()
+ raise RuntimeError(error or "policy worker failed before readiness")
+ return _PolicyExecutionProcess(process, messages, start_event)
+ except BaseException:
if process.is_alive():
- process.terminate()
- process.join(5)
- if process.is_alive():
- process.kill()
- process.join()
- return PolicyExecution(
- status="timed_out",
- duration_seconds=time.monotonic() - started,
- error=f"policy timed out after {timeout_s:g}s",
- )
- try:
- status, error = result_queue.get_nowait()
- except queue.Empty:
- return PolicyExecution(
- status="infrastructure_error",
- duration_seconds=time.monotonic() - started,
- error=f"policy worker exited with code {process.exitcode} without a result",
- )
- return PolicyExecution(
- status=status,
- duration_seconds=time.monotonic() - started,
- error=error,
- )
- except Exception as exc:
- return PolicyExecution(
- status="infrastructure_error",
- duration_seconds=time.monotonic() - started,
- error=f"{type(exc).__name__}: {exc}",
- )
- finally:
- result_queue.close()
+ _stop_process(process)
+ messages.close()
+ worker_messages.close()
+ raise
def _record_prompt(
self,
@@ -401,36 +390,108 @@ def submit(self, source: str, serialized: bytes) -> TrialRun:
return trial
-def _execute_policy_worker(serialized_path: str, result_queue: Any) -> None:
+def _execute_policy_worker(
+ serialized_path: str,
+ memory_path: str,
+ messages: Any,
+ start_event: Any,
+) -> None:
try:
- import cloudpickle
+ try:
+ import cloudpickle
- with Path(serialized_path).open("rb") as handle:
- policy = cloudpickle.load(handle)
- except Exception as exc:
- result_queue.put(
- ("infrastructure_error", f"policy load failed: {type(exc).__name__}: {exc}")
- )
- return
- try:
- from dimos.porcelain.dimos import Dimos
+ with Path(serialized_path).open("rb") as handle:
+ policy = cloudpickle.load(handle)
+ except Exception as exc:
+ messages.send(
+ ("infrastructure_error", f"policy load failed: {type(exc).__name__}: {exc}")
+ )
+ return
+ try:
+ from dimos.memory2.store.sqlite import SqliteStore
+ from dimos.porcelain.dimos import Dimos
- app = Dimos.connect()
- except Exception as exc:
- result_queue.put(
- ("infrastructure_error", f"DimOS connection failed: {type(exc).__name__}: {exc}")
+ memory = SqliteStore(path=memory_path, must_exist=True, read_only=True)
+ memory.start()
+ app = Dimos.connect(memory=memory)
+ except Exception as exc:
+ messages.send(
+ ("infrastructure_error", f"DimOS connection failed: {type(exc).__name__}: {exc}")
+ )
+ return
+ messages.send(("ready", None))
+ start_event.wait()
+ try:
+ result = policy(app)
+ if result is not None:
+ raise TypeError("policy(app) must return None")
+ except Exception as exc:
+ messages.send(("policy_error", f"{type(exc).__name__}: {exc}"))
+ else:
+ messages.send(("completed", None))
+ finally:
+ app.stop()
+ finally:
+ messages.close()
+
+
+class _PolicyExecutionProcess:
+ def __init__(self, process: Any, messages: Any, start_event: Any) -> None:
+ self._process = process
+ self._messages = messages
+ self._start_event = start_event
+ self._started_at: float | None = None
+ self._finished: PolicyExecution | None = None
+
+ def start(self) -> None:
+ if self._started_at is not None:
+ raise RuntimeError("policy execution already started")
+ self._started_at = time.monotonic()
+ self._start_event.set()
+
+ def finish(self, *, grace_s: float = 1.0) -> PolicyExecution:
+ if grace_s < 0:
+ raise ValueError("grace_s must be non-negative")
+ if self._finished is not None:
+ return self._finished
+ if self._started_at is None:
+ raise RuntimeError("policy execution has not started")
+ if self._messages.poll(grace_s):
+ try:
+ status, error = self._messages.recv()
+ except EOFError:
+ status = "infrastructure_error"
+ error = "policy worker closed its result channel without a result"
+ self._process.join(grace_s)
+ if self._process.is_alive():
+ _stop_process(self._process)
+ else:
+ stopped = self._process.is_alive()
+ if stopped:
+ _stop_process(self._process)
+ status, error = "stopped", None
+ else:
+ self._process.join()
+ status = "infrastructure_error"
+ error = f"policy worker exited with code {self._process.exitcode} without a result"
+ self._messages.close()
+ self._finished = PolicyExecution(
+ status=status,
+ duration_seconds=time.monotonic() - self._started_at,
+ error=error,
)
+ return self._finished
+
+
+def _stop_process(process: Any) -> None:
+ if not process.is_alive():
+ process.join()
return
- try:
- result = policy(app)
- if result is not None:
- raise TypeError("policy(app) must return None")
- except Exception as exc:
- result_queue.put(("policy_error", f"{type(exc).__name__}: {exc}"))
- else:
- result_queue.put(("completed", None))
- finally:
- app.stop()
+ process.terminate()
+ process.join(5)
+ if process.is_alive():
+ process.kill()
+ process.join()
def _assemble_user_message(evaluation_protocol: str, task_input: str) -> str:
diff --git a/dimos/benchmark/evaluation/test_policy_runtime.py b/dimos/benchmark/evaluation/test_policy_runtime.py
index 45d1c777c7..9686a5fcfe 100644
--- a/dimos/benchmark/evaluation/test_policy_runtime.py
+++ b/dimos/benchmark/evaluation/test_policy_runtime.py
@@ -15,8 +15,9 @@
from __future__ import annotations
import json
+import multiprocessing
from pathlib import Path
-import queue
+import threading
import cloudpickle
import pytest
@@ -28,6 +29,7 @@
from dimos.benchmark.evaluation.runtime import (
CodePolicyRuntimeFactory,
_execute_policy_worker,
+ _PolicyExecutionProcess,
_SubmissionManager,
)
from dimos.memory2.store.sqlite import SqliteStore
@@ -53,6 +55,13 @@ def stop(self) -> None:
pass
+def _send_large_policy_error(messages, start_event, result_sending) -> None:
+ start_event.wait()
+ result_sending.set()
+ messages.send(("policy_error", "x" * 1_000_000))
+ messages.close()
+
+
def _trial(path: Path, number: int) -> TrialRun:
path.mkdir(exist_ok=True)
log_path = path / "main.jsonl"
@@ -125,15 +134,73 @@ def test_policy_artifact_records_serialized_callable(tmp_path: Path) -> None:
def test_policy_worker_connects_and_invokes_callable(mocker, tmp_path: Path) -> None:
serialized_path = tmp_path / "policy.pkl"
serialized_path.write_bytes(cloudpickle.dumps(policy))
+ memory_path = tmp_path / "recording.db"
+ with SqliteStore(path=str(memory_path)) as memory:
+ memory.stream("events", int).append(1)
app = mocker.Mock(spec=Dimos)
connect = mocker.patch.object(Dimos, "connect", return_value=app)
- results: queue.Queue[tuple[str, str | None]] = queue.Queue()
-
- _execute_policy_worker(str(serialized_path), results)
+ results, worker_results = multiprocessing.Pipe(duplex=False)
+ start_event = threading.Event()
- assert results.get_nowait() == ("completed", None)
- connect.assert_called_once_with()
+ worker = threading.Thread(
+ target=_execute_policy_worker,
+ args=(str(serialized_path), str(memory_path), worker_results, start_event),
+ )
+ worker.start()
+
+ assert results.poll(1)
+ assert results.recv() == ("ready", None)
+ connect.assert_called_once()
+ attached = connect.call_args.kwargs["memory"]
+ assert attached.config.read_only is True
+ start_event.set()
+ assert results.poll(1)
+ assert results.recv() == ("completed", None)
+ worker.join(timeout=1)
+ assert not worker.is_alive()
app.stop.assert_called_once_with()
+ results.close()
+ worker_results.close()
+
+
+def test_prepared_policy_waits_for_explicit_start_and_stops_at_trial_end(mocker) -> None:
+ process = mocker.Mock()
+ process.is_alive.return_value = True
+ messages = mocker.Mock()
+ messages.poll.return_value = False
+ start_event = threading.Event()
+ execution = _PolicyExecutionProcess(process, messages, start_event)
+
+ assert not start_event.is_set()
+ execution.start()
+ assert start_event.is_set()
+
+ result = execution.finish(grace_s=0.0)
+
+ assert result.status == "stopped"
+ process.terminate.assert_called_once_with()
+
+
+def test_policy_result_larger_than_pipe_buffer_does_not_deadlock() -> None:
+ context = multiprocessing.get_context("spawn")
+ messages, worker_messages = context.Pipe(duplex=False)
+ start_event = context.Event()
+ result_sending = context.Event()
+ process = context.Process(
+ target=_send_large_policy_error,
+ args=(worker_messages, start_event, result_sending),
+ daemon=True,
+ )
+ process.start()
+ worker_messages.close()
+ execution = _PolicyExecutionProcess(process, messages, start_event)
+ execution.start()
+ assert result_sending.wait(timeout=2)
+
+ result = execution.finish(grace_s=0.01)
+
+ assert result.status == "policy_error"
+ assert result.error == "x" * 1_000_000
def test_explore_freezes_last_of_five_debug_submissions(mocker, tmp_path: Path) -> None:
diff --git a/dimos/benchmark/evaluation/test_registry.py b/dimos/benchmark/evaluation/test_registry.py
index 027dae0f3d..d47b475161 100644
--- a/dimos/benchmark/evaluation/test_registry.py
+++ b/dimos/benchmark/evaluation/test_registry.py
@@ -76,6 +76,13 @@ def test_unknown_evaluation_reports_name(monkeypatch) -> None:
registry.resolve_evaluation("missing")
+def test_builtin_libero_pro_evaluation_resolves_in_repo() -> None:
+ resolved = registry.resolve_evaluation("libero-pro")
+
+ assert resolved.provider == "dimos"
+ assert resolved.evaluation.name == "libero-pro"
+
+
def test_builtin_vlnce_r2r_evaluation_resolves_with_live_agent() -> None:
resolved = registry.resolve_evaluation("vlnce-r2r")
diff --git a/dimos/benchmark/libero_pro/assets.py b/dimos/benchmark/libero_pro/assets.py
new file mode 100644
index 0000000000..cae7f42ba5
--- /dev/null
+++ b/dimos/benchmark/libero_pro/assets.py
@@ -0,0 +1,72 @@
+"""Download and verify task assets before a LIBERO-PRO trial starts."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+import hashlib
+from pathlib import Path, PurePosixPath
+
+import requests
+
+from dimos.benchmark.libero_pro.models import AssetReference, LiberoTaskManifest
+from dimos.constants import CACHE_DIR
+
+
+class LiberoAssetError(RuntimeError):
+ """A pinned LIBERO-PRO task asset is missing or invalid."""
+
+
+@dataclass(frozen=True)
+class PreparedAssets:
+ bddl: Path
+ init_states: Path
+
+
+def prepare_assets(
+ manifest: LiberoTaskManifest,
+ *,
+ cache_root: Path = CACHE_DIR / "evaluation" / "libero_pro",
+) -> PreparedAssets:
+ root = cache_root / manifest.source.dataset_revision
+ return PreparedAssets(
+ bddl=_materialize(manifest, manifest.assets.bddl, root),
+ init_states=_materialize(manifest, manifest.assets.init_states, root),
+ )
+
+
+def _materialize(
+ manifest: LiberoTaskManifest,
+ reference: AssetReference,
+ root: Path,
+) -> Path:
+ relative = PurePosixPath(reference.repository_path)
+ if relative.is_absolute() or ".." in relative.parts:
+ raise LiberoAssetError(f"unsafe asset path: {reference.repository_path}")
+ target = root.joinpath(*relative.parts)
+ if target.is_file():
+ _verify(target, reference)
+ return target
+ target.parent.mkdir(parents=True, exist_ok=True)
+ url = (
+ f"https://huggingface.co/datasets/{manifest.source.dataset_repository}/resolve/"
+ f"{manifest.source.dataset_revision}/{reference.repository_path}"
+ )
+ response = requests.get(url, timeout=120)
+ response.raise_for_status()
+ temporary = target.with_suffix(target.suffix + ".partial")
+ temporary.write_bytes(response.content)
+ try:
+ _verify(temporary, reference)
+ temporary.replace(target)
+ except BaseException:
+ temporary.unlink(missing_ok=True)
+ raise
+ return target
+
+
+def _verify(path: Path, reference: AssetReference) -> None:
+ if path.stat().st_size != reference.size_bytes:
+ raise LiberoAssetError(f"asset size mismatch: {path}")
+ digest = hashlib.sha256(path.read_bytes()).hexdigest()
+ if digest != reference.sha256:
+ raise LiberoAssetError(f"asset digest mismatch: {path}")
diff --git a/dimos/benchmark/libero_pro/blueprint.py b/dimos/benchmark/libero_pro/blueprint.py
new file mode 100644
index 0000000000..cf09f4fa58
--- /dev/null
+++ b/dimos/benchmark/libero_pro/blueprint.py
@@ -0,0 +1,119 @@
+"""Complete policy-only DimOS blueprint for one LIBERO-PRO trial."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from dimos.benchmark.libero_pro.connection import LiberoConnection, LiberoRecorder
+from dimos.benchmark.libero_pro.video import LiberoVideoRecorder
+from dimos.control.components import HardwareComponent, HardwareType, make_joints
+from dimos.control.coordinator import ControlCoordinator
+from dimos.core.coordination.blueprints import Blueprint, autoconnect
+from dimos.manipulation.grasping.grasp_gen_x import GraspGenXModule
+from dimos.manipulation.manipulation_module import ManipulationModule
+from dimos.manipulation.planning.groups.models import PlanningGroupDefinition
+from dimos.manipulation.planning.spec.config import RobotModelConfig
+from dimos.memory2.module import OnExisting
+from dimos.perception.grounded_segmentation import GroundedSegmentationModule
+from dimos.robot.manipulators._modeling import base_pose, coordinator_joint_mapping, joint_names
+from dimos.robot.manipulators.common.blueprints import trajectory_task
+
+PANDA_MODEL_PATH = Path(__file__).with_name("panda.urdf")
+
+
+def _panda_model() -> RobotModelConfig:
+ local_joints = joint_names(7)
+ return RobotModelConfig(
+ name="panda",
+ model_path=PANDA_MODEL_PATH,
+ base_pose=base_pose(),
+ joint_names=local_joints,
+ base_link="base",
+ planning_groups=[
+ PlanningGroupDefinition(
+ name="manipulator",
+ joint_names=tuple(local_joints),
+ base_link="base",
+ tip_link="tcp",
+ )
+ ],
+ joint_limits_lower=[-2.8973, -1.7628, -2.8973, -3.0718, -2.8973, -0.0175, -2.8973],
+ joint_limits_upper=[2.8973, 1.7628, 2.8973, -0.0698, 2.8973, 3.7525, 2.8973],
+ # The connection turns each target into LIBERO's native OSC pose action.
+ # Keep planner timing effectively instantaneous so real-time policies
+ # control motion through measured setpoint updates rather than a second
+ # host-side rate limiter.
+ velocity_limits=[100.0] * 7,
+ max_velocity=100.0,
+ max_acceleration=200.0,
+ joint_name_mapping=coordinator_joint_mapping("panda", 7),
+ gripper_hardware_id="panda",
+ )
+
+
+def libero_trial_blueprint(
+ *,
+ policy_endpoint: str,
+ discovery_address: str,
+ memory_path: Path,
+ video_path: Path,
+) -> Blueprint:
+ panda = HardwareComponent(
+ hardware_id="panda",
+ hardware_type=HardwareType.MANIPULATOR,
+ joints=make_joints("panda", 7),
+ gripper_joints=["panda/gripper"],
+ adapter_type="sim_mujoco",
+ address=discovery_address,
+ gripper_open_position=0.04,
+ gripper_closed_position=0.0,
+ )
+ return autoconnect(
+ LiberoConnection.blueprint(
+ endpoint=policy_endpoint,
+ discovery_address=discovery_address,
+ ),
+ ControlCoordinator.blueprint(
+ tick_rate=20.0,
+ hardware=[panda],
+ tasks=[
+ trajectory_task(
+ panda,
+ start_position_tolerance=0.2,
+ goal_position_tolerance=0.1,
+ )
+ ],
+ ),
+ ManipulationModule.blueprint(robots=[_panda_model()]),
+ GroundedSegmentationModule.blueprint(),
+ GraspGenXModule.blueprint(
+ gripper={
+ "extents_open": (0.08, 0.04, 0.10),
+ "offset_open": (0.0, 0.0, 0.05),
+ "extents_half_open": (0.04, 0.04, 0.10),
+ "offset_half_open": (0.0, 0.0, 0.05),
+ "fingertip_depth": 0.10,
+ "family": "parallel_2f",
+ },
+ max_candidates=25,
+ ),
+ LiberoRecorder.blueprint(
+ db_path=memory_path,
+ on_existing=OnExisting.OVERWRITE,
+ record_tf=True,
+ stream_codecs={
+ "agentview_depth_image": "pickle",
+ "eye_in_hand_depth_image": "pickle",
+ },
+ poseless_streams=[
+ "joint_state",
+ "agentview_color_image",
+ "agentview_depth_image",
+ "agentview_camera_info",
+ "eye_in_hand_color_image",
+ "eye_in_hand_depth_image",
+ "eye_in_hand_camera_info",
+ ],
+ ),
+ LiberoVideoRecorder.blueprint(output_path=video_path),
+ )
diff --git a/dimos/benchmark/libero_pro/cases/goal-task-0-single-trial/evaluation.json b/dimos/benchmark/libero_pro/cases/goal-task-0-single-trial/evaluation.json
new file mode 100644
index 0000000000..b658b4e55f
--- /dev/null
+++ b/dimos/benchmark/libero_pro/cases/goal-task-0-single-trial/evaluation.json
@@ -0,0 +1,13 @@
+{
+ "schema_version": "2.0",
+ "runtime": {
+ "model": "gpt-5.6-luna",
+ "thinking_level": "medium"
+ },
+ "evaluation": {
+ "name": "libero-pro",
+ "config": {
+ "task_manifest": "task.json"
+ }
+ }
+}
diff --git a/dimos/benchmark/libero_pro/cases/goal-task-0-single-trial/task.json b/dimos/benchmark/libero_pro/cases/goal-task-0-single-trial/task.json
new file mode 100644
index 0000000000..d3531faef4
--- /dev/null
+++ b/dimos/benchmark/libero_pro/cases/goal-task-0-single-trial/task.json
@@ -0,0 +1,46 @@
+{
+ "schema_version": "1.0",
+ "case_id": "libero-pro-goal-task-0-single-trial",
+ "source": {
+ "repository": "Zxy-MLlab/LIBERO-PRO",
+ "revision": "eafdb809426b13153aa1e4c42d6601844217dfec",
+ "dataset_repository": "zhouxueyang/LIBERO-Pro",
+ "dataset_revision": "e9f3a5356efed7a23b06c28cfe880a22c3306074"
+ },
+ "task": {
+ "suite": "libero_goal_task",
+ "task_order_index": 0,
+ "task_index": 0,
+ "task_name": "open_the_middle_drawer_of_the_cabinet",
+ "instruction": "open the bottom drawer of the cabinet"
+ },
+ "assets": {
+ "bddl": {
+ "repository_path": "bddl_files/libero_goal_task/open_the_middle_drawer_of_the_cabinet.bddl",
+ "sha256": "79069933e60bdeb0a7ba4c414b3e2db3ac44ca02408dfc00a73607e201c3b334",
+ "size_bytes": 2937
+ },
+ "init_states": {
+ "repository_path": "init_files/libero_goal_task/open_the_middle_drawer_of_the_cabinet.pruned_init",
+ "sha256": "aca0dd09ccc97b099a6fbd3b3c0f6700e438234cc1b0272b47fb53b88290c6cf",
+ "size_bytes": 4111
+ }
+ },
+ "episodes": {
+ "debug_init_state_indices": [1, 2, 3, 4, 5],
+ "scored_init_state_index": 0
+ },
+ "contract": {
+ "robot": "Panda",
+ "controller": "OSC_POSE",
+ "cameras": [
+ {"name": "agentview", "width": 128, "height": 128},
+ {"name": "robot0_eye_in_hand", "width": 128, "height": 128}
+ ],
+ "control_frequency_hz": 20,
+ "settling_ticks": 5,
+ "horizon_ticks": 300,
+ "clock": "continuous_real_time",
+ "success": "native_bddl_goal_predicates"
+ }
+}
diff --git a/dimos/benchmark/libero_pro/connection.py b/dimos/benchmark/libero_pro/connection.py
new file mode 100644
index 0000000000..883e8e7a1a
--- /dev/null
+++ b/dimos/benchmark/libero_pro/connection.py
@@ -0,0 +1,238 @@
+"""Ordinary DimOS Module adapting the LIBERO policy gRPC interface."""
+
+from __future__ import annotations
+
+import threading
+import time
+from typing import Any
+
+import grpc # type: ignore[import-untyped]
+import numpy as np
+from numpy.typing import NDArray
+
+from dimos.benchmark.libero_pro.proto import libero_pro_pb2 as pb2, libero_pro_pb2_grpc as pb2_grpc
+from dimos.core.core import rpc
+from dimos.core.module import Module, ModuleConfig
+from dimos.core.stream import In, Out
+from dimos.memory2.module import Recorder
+from dimos.msgs.geometry_msgs.Transform import Transform
+from dimos.msgs.sensor_msgs.CameraInfo import CameraInfo
+from dimos.msgs.sensor_msgs.Image import Image, ImageFormat
+from dimos.msgs.sensor_msgs.JointState import JointState
+from dimos.msgs.tf2_msgs.TFMessage import TFMessage
+from dimos.simulation.engines.mujoco_shm import ManipShmWriter, shm_key_from_path
+
+CAMERA_OPTICAL_FRAMES = {
+ "agentview": "agentview_optical",
+ "robot0_eye_in_hand": "eye_in_hand_optical",
+}
+
+
+class LiberoConnectionConfig(ModuleConfig):
+ endpoint: str
+ discovery_address: str
+ startup_timeout_s: float = 30.0
+
+
+class LiberoConnection(Module):
+ """Publish permitted Panda observations and forward ordinary joint commands."""
+
+ config: LiberoConnectionConfig
+ joint_state: Out[JointState]
+ tf: Out[TFMessage]
+ agentview_color_image: Out[Image]
+ agentview_depth_image: Out[Image]
+ agentview_camera_info: Out[CameraInfo]
+ eye_in_hand_color_image: Out[Image]
+ eye_in_hand_depth_image: Out[Image]
+ eye_in_hand_camera_info: Out[CameraInfo]
+
+ def __init__(self, endpoint: str, discovery_address: str, **kwargs: Any) -> None:
+ super().__init__(endpoint=endpoint, discovery_address=discovery_address, **kwargs)
+ self._channel: grpc.Channel | None = None
+ self._stub: pb2_grpc.PolicyInterfaceStub | None = None
+ self._shm: ManipShmWriter | None = None
+ self._stop = threading.Event()
+ self._watch_thread: threading.Thread | None = None
+ self._command_thread: threading.Thread | None = None
+ self._sequence = 0
+ self._latest_gripper = 0.04
+ self._arm_target: NDArray[np.float64] | None = None
+
+ @rpc
+ def start(self) -> None:
+ super().start()
+ self._channel = grpc.insecure_channel(self.config.endpoint)
+ self._stub = pb2_grpc.PolicyInterfaceStub(self._channel)
+ grpc.channel_ready_future(self._channel).result(timeout=self.config.startup_timeout_s)
+ self._stub.GetHealth(pb2.Empty(), timeout=self.config.startup_timeout_s)
+ self._shm = ManipShmWriter(shm_key_from_path(self.config.discovery_address))
+ self._watch_thread = threading.Thread(target=self._watch, daemon=True)
+ self._command_thread = threading.Thread(target=self._pump_commands, daemon=True)
+ self._watch_thread.start()
+ self._command_thread.start()
+
+ @rpc
+ def stop(self) -> None:
+ self._stop.set()
+ if self._channel is not None:
+ self._channel.close()
+ for thread in (self._watch_thread, self._command_thread):
+ if thread is not None:
+ thread.join(timeout=2)
+ if self._shm is not None:
+ self._shm.cleanup()
+ self._shm = None
+ super().stop()
+
+ def _watch(self) -> None:
+ assert self._stub is not None
+ try:
+ for snapshot in self._stub.WatchState(pb2.WatchRequest()):
+ if self._stop.is_set():
+ return
+ names = [f"panda/joint{i + 1}" for i in range(7)] + ["panda/gripper"]
+ positions = [*snapshot.joint_position, snapshot.gripper_position]
+ if self._arm_target is None:
+ self._arm_target = np.asarray(snapshot.joint_position, dtype=np.float64)
+ velocities = [*snapshot.joint_velocity, 0.0]
+ self.joint_state.publish(
+ JointState(
+ ts=snapshot.timestamp_s,
+ frame_id="panda",
+ name=names,
+ position=positions,
+ velocity=velocities,
+ effort=[0.0] * 8,
+ )
+ )
+ if self._shm is not None:
+ self._shm.write_joint_state(
+ list(snapshot.joint_position), list(snapshot.joint_velocity), [0.0] * 7
+ )
+ self._shm.write_gripper_state(snapshot.gripper_position)
+ self._shm.signal_ready(num_joints=8)
+ transforms = []
+ for frame in snapshot.cameras:
+ image, depth, info, transform = _decode_camera(frame, snapshot.timestamp_s)
+ transforms.append(transform)
+ if frame.camera == "agentview":
+ self.agentview_color_image.publish(image)
+ self.agentview_depth_image.publish(depth)
+ self.agentview_camera_info.publish(info)
+ elif frame.camera == "robot0_eye_in_hand":
+ self.eye_in_hand_color_image.publish(image)
+ self.eye_in_hand_depth_image.publish(depth)
+ self.eye_in_hand_camera_info.publish(info)
+ if transforms:
+ self.tf.publish(TFMessage(*transforms))
+ except grpc.RpcError:
+ if self._stop.is_set():
+ return
+ raise
+
+ def _pump_commands(self) -> None:
+ while not self._stop.wait(0.005):
+ if self._shm is None or self._stub is None:
+ continue
+ arm = self._shm.read_position_command(7)
+ gripper = self._shm.read_gripper_command()
+ if arm is None and gripper is None:
+ continue
+ if gripper is not None:
+ self._latest_gripper = float(gripper)
+ arm = self._resolve_arm_target(arm)
+ if arm is None:
+ continue
+ self._sequence += 1
+ self._stub.SetJointTargets(
+ pb2.JointTargets(
+ joint_position=list(arm),
+ gripper_position=self._latest_gripper,
+ sequence=self._sequence,
+ ),
+ timeout=2,
+ )
+ time.sleep(0)
+
+ def _resolve_arm_target(
+ self,
+ arm: NDArray[np.float64] | None,
+ ) -> NDArray[np.float64] | None:
+ """Retain the commanded arm target across gripper-only updates."""
+ if arm is not None:
+ self._arm_target = np.asarray(arm, dtype=np.float64).copy()
+ if self._arm_target is None:
+ return None
+ return self._arm_target.copy()
+
+
+class LiberoRecorder(Recorder):
+ joint_state: In[JointState]
+ agentview_color_image: In[Image]
+ agentview_depth_image: In[Image]
+ agentview_camera_info: In[CameraInfo]
+ eye_in_hand_color_image: In[Image]
+ eye_in_hand_depth_image: In[Image]
+ eye_in_hand_camera_info: In[CameraInfo]
+
+
+def _decode_camera(
+ frame: pb2.CameraFrame,
+ timestamp_s: float,
+) -> tuple[Image, Image, CameraInfo, Transform]:
+ if frame.camera not in CAMERA_OPTICAL_FRAMES:
+ raise ValueError(f"Unknown LIBERO camera: {frame.camera}")
+ if len(frame.intrinsic) != 9:
+ raise ValueError("camera intrinsic must contain a 3x3 matrix")
+ if len(frame.camera_to_robot_base) != 16:
+ raise ValueError("camera pose must contain a 4x4 matrix")
+ frame_id = CAMERA_OPTICAL_FRAMES[frame.camera]
+ color = np.frombuffer(frame.rgb, dtype=np.uint8).reshape(frame.height, frame.width, 3)
+ depth = np.frombuffer(frame.depth_meters, dtype=np.float32).reshape(frame.height, frame.width)
+ intrinsic = list(frame.intrinsic)
+ info = CameraInfo(
+ height=frame.height,
+ width=frame.width,
+ distortion_model="plumb_bob",
+ D=[0.0] * 5,
+ K=intrinsic,
+ P=[
+ intrinsic[0],
+ intrinsic[1],
+ intrinsic[2],
+ 0.0,
+ intrinsic[3],
+ intrinsic[4],
+ intrinsic[5],
+ 0.0,
+ intrinsic[6],
+ intrinsic[7],
+ intrinsic[8],
+ 0.0,
+ ],
+ frame_id=frame_id,
+ ts=timestamp_s,
+ )
+ transform = Transform.from_matrix(
+ np.asarray(frame.camera_to_robot_base, dtype=np.float64).reshape(4, 4),
+ ts=timestamp_s,
+ frame_id="world",
+ child_frame_id=frame_id,
+ )
+ return (
+ Image(
+ data=color,
+ format=ImageFormat.RGB,
+ frame_id=frame_id,
+ ts=timestamp_s,
+ ),
+ Image(
+ data=depth,
+ format=ImageFormat.DEPTH,
+ frame_id=frame_id,
+ ts=timestamp_s,
+ ),
+ info,
+ transform,
+ )
diff --git a/dimos/benchmark/libero_pro/control.py b/dimos/benchmark/libero_pro/control.py
new file mode 100644
index 0000000000..e154dd106d
--- /dev/null
+++ b/dimos/benchmark/libero_pro/control.py
@@ -0,0 +1,64 @@
+"""Evaluator-only client for the privileged LIBERO-PRO control listener."""
+
+from __future__ import annotations
+
+from typing import cast
+
+import grpc # type: ignore[import-untyped]
+
+from dimos.benchmark.libero_pro.models import LiberoTaskManifest
+from dimos.benchmark.libero_pro.proto import libero_pro_pb2 as pb2, libero_pro_pb2_grpc as pb2_grpc
+
+
+class EvaluationControlClient:
+ def __init__(self, endpoint: str, token: str) -> None:
+ self._channel = grpc.insecure_channel(endpoint)
+ self._stub = pb2_grpc.EvaluationControlStub(self._channel)
+ self._metadata = (("authorization", f"Bearer {token}"),)
+
+ def wait_ready(self, timeout_s: float = 30.0) -> None:
+ grpc.channel_ready_future(self._channel).result(timeout=timeout_s)
+ self._stub.GetHealth(pb2.Empty(), metadata=self._metadata, timeout=timeout_s)
+
+ def initialize(
+ self,
+ manifest: LiberoTaskManifest,
+ init_state_index: int,
+ *,
+ timeout_s: float = 120.0,
+ ) -> None:
+ ready = self._stub.InitializeTrial(
+ pb2.InitializeTrialRequest(
+ suite=manifest.task.suite,
+ task_order_index=manifest.task.task_order_index,
+ task_index=manifest.task.task_index,
+ init_state_index=init_state_index,
+ horizon_ticks=manifest.contract.horizon_ticks,
+ control_frequency_hz=manifest.contract.control_frequency_hz,
+ settling_ticks=manifest.contract.settling_ticks,
+ ),
+ metadata=self._metadata,
+ timeout=timeout_s,
+ )
+ if ready.task_name != manifest.task.task_name:
+ raise RuntimeError("container task name does not match task manifest")
+ if ready.instruction != manifest.task.instruction:
+ raise RuntimeError("container instruction does not match task manifest")
+
+ def start(self) -> None:
+ self._stub.StartTrial(pb2.Empty(), metadata=self._metadata, timeout=5)
+
+ def wait_terminal(self, timeout_s: float) -> pb2.TerminalResult:
+ return cast(
+ "pb2.TerminalResult",
+ self._stub.WaitForTerminal(pb2.Empty(), metadata=self._metadata, timeout=timeout_s),
+ )
+
+ def cancel(self) -> pb2.TerminalResult:
+ return cast(
+ "pb2.TerminalResult",
+ self._stub.CancelTrial(pb2.Empty(), metadata=self._metadata, timeout=5),
+ )
+
+ def close(self) -> None:
+ self._channel.close()
diff --git a/dimos/benchmark/libero_pro/evaluation.py b/dimos/benchmark/libero_pro/evaluation.py
new file mode 100644
index 0000000000..dfd1629c7a
--- /dev/null
+++ b/dimos/benchmark/libero_pro/evaluation.py
@@ -0,0 +1,356 @@
+"""One complete built-in LIBERO-PRO code-policy Evaluation."""
+
+from __future__ import annotations
+
+from datetime import datetime, timezone
+import json
+import os
+from pathlib import Path
+import time
+from typing import Literal, TypedDict
+
+from pydantic import BaseModel
+
+from dimos.benchmark.evaluation.models import (
+ ArtifactReference,
+ EvaluationReport,
+ InlineNativeResult,
+ SummaryItem,
+)
+from dimos.benchmark.evaluation.progress import StatusProgress, emit_progress
+from dimos.benchmark.evaluation.protocol import (
+ CodePolicyRuntime,
+ EvaluationContext,
+ PolicyArtifact,
+ TrialOutcome,
+ TrialRun,
+)
+from dimos.benchmark.libero_pro.assets import PreparedAssets, prepare_assets
+from dimos.benchmark.libero_pro.blueprint import libero_trial_blueprint
+from dimos.benchmark.libero_pro.control import EvaluationControlClient
+from dimos.benchmark.libero_pro.models import LiberoProConfig, LiberoTaskManifest
+from dimos.benchmark.libero_pro.podman import LiberoPodmanContainer
+from dimos.core.coordination.module_coordinator import ModuleCoordinator
+from dimos.core.coordination.process_lifecycle import DIMOS_RUN_ID_ENV
+
+EVALUATION_PROTOCOL = """Develop a task-specific real-time DimOS policy.
+Use submit_policy to test each revision. Every submission runs a fresh episode of
+the exact task below. The final submitted callable is frozen and evaluated on a
+fresh initial state. The simulator advances continuously at 20 Hz while the
+callable runs, so policies must tolerate real execution latency.
+
+Inspect the actual recorded RGB images after every debug submission. `python_exec`
+supports rich image output, just as it does in the navigation benchmark:
+
+ trial = submit_policy(policy)
+ from IPython.display import display
+ from PIL import Image as PILImage
+ with trial.open_memory() as memory:
+ frame = memory.stream("agentview_color_image").last().data
+ display(PILImage.fromarray(frame.to_rgb().data))
+
+Displayed images are delivered to you visually. Inspect both `agentview_color_image`
+and `eye_in_hand_color_image` when the wrist view can disambiguate contact. Do not
+substitute ASCII art, pixel statistics, database internals, or textual image
+representations for viewing the RGB frames directly.
+
+Inside policy(app), `app.memory` is the live read-only Memory2 recording and normal
+modules are available by class name. Use the calibrated RGB-D helper rather than
+guessing camera geometry:
+
+ from dimos.perception.rgbd import latest_rgbd, project_depth
+ observation = latest_rgbd(
+ app.memory,
+ color_stream="agentview_color_image",
+ depth_stream="agentview_depth_image",
+ camera_info_stream="agentview_camera_info",
+ optical_frame="agentview_optical",
+ )
+ masks = app.GroundedSegmentationModule.segment(observation.color, ["object description"])
+ objects = project_depth(masks, observation)
+
+`objects` contains world-frame point clouds and ordinary shape helpers. Request ranked
+grasps with `app.GraspGenXModule.propose_grasps(objects[0].pointcloud)`. A proposal's
+header supplies its timestamp and frame. Turn each ranked candidate into a planning
+target with `PoseStamped(ts=grasps.header.timestamp,
+frame_id=grasps.header.frame_id, position=candidate.pose.position,
+orientation=candidate.pose.orientation)`. Inspect
+`app.ManipulationModule.list_planning_groups()` and use a returned `.id` instead of
+guessing group IDs. Try targets in rank order with `plan_to_poses({group.id: target})`,
+then call `execute()` only for a successful PlanResult. Every motion, gripper command,
+and execution call returns a typed result; inspect `.succeeded` and `.message` rather
+than assuming that a command moved the robot. Re-observe after motion and keep the
+policy closed-loop. The Panda gripper positions are 0.04 m open and 0.0 m closed.
+Use collision-checked planning for the free-space approach. Once that route is clear,
+`move_to_pose(..., check_collision=False)` provides the low-latency absolute Cartesian
+updates needed for contact-rich motion; keep those updates small and verify each result.
+
+Plan with the complete hand volume, not only the TCP. Infer the grasp and motion axes
+from observed geometry. For articulated handles, compare a force-closure pinch with a
+geometric hook: approach through free space, place closed fingers behind the handle,
+engage it orthogonally, then withdraw along the articulation axis. Test equivalent
+wrist-roll candidates because the fingers can reach while the wrist or forearm still
+collides with nearby objects. If IK fails at a joint boundary, retry nearby poses a few
+millimeters away instead of abandoning the strategy. Prefer short, smooth, measured
+waypoint updates over one large target jump, but do not waste the real-time horizon on
+long sleeps. A held final setpoint continues executing after policy(app) returns, so
+finish the callable once the commanded motion is safely engaged.
+"""
+
+
+class LiberoProEvaluation:
+ name = "libero-pro"
+ runtime_profile = "code-policy-v1"
+ config_model: type[BaseModel] = LiberoProConfig
+
+ def run(self, config: BaseModel, context: EvaluationContext) -> EvaluationReport:
+ if not isinstance(config, LiberoProConfig):
+ raise TypeError("libero-pro received the wrong configuration type")
+ if not isinstance(context.runtime, CodePolicyRuntime):
+ raise TypeError("libero-pro requires the code-policy-v1 runtime")
+ runtime = context.runtime
+ manifest_path = Path(config.task_manifest)
+ if not manifest_path.is_absolute():
+ manifest_path = context.spec_dir / manifest_path
+ manifest = LiberoTaskManifest.model_validate_json(manifest_path.read_bytes())
+ emit_progress(
+ context.progress,
+ StatusProgress(channel="eval", message="LIBERO-PRO preflight"),
+ )
+ assets = prepare_assets(manifest)
+ LiberoPodmanContainer.ensure_image()
+
+ def submit(policy: PolicyArtifact, number: int, path: Path) -> TrialRun:
+ init_index = manifest.episodes.debug_init_state_indices[number - 1]
+ return _run_trial(
+ manifest,
+ assets,
+ policy,
+ init_index=init_index,
+ path=path,
+ runtime=runtime,
+ run_id=f"{context.run_id}-debug-{number}",
+ )[0]
+
+ exploration = runtime.explore(
+ evaluation_protocol=EVALUATION_PROTOCOL,
+ task_input=manifest.task.instruction,
+ submit_debug_trial=submit,
+ )
+ if exploration.policy is None:
+ raise RuntimeError(exploration.error or "exploration produced no policy")
+ scored_path = context.workspace / "scored-trial"
+ trial, native = _run_trial(
+ manifest,
+ assets,
+ exploration.policy,
+ init_index=manifest.episodes.scored_init_state_index,
+ path=scored_path,
+ runtime=runtime,
+ run_id=f"{context.run_id}-scored",
+ )
+ if trial.outcome.status != "completed":
+ raise RuntimeError(trial.outcome.error or "scored LIBERO-PRO trial failed")
+ native_value = {
+ "result_type": "single_trial",
+ "case_id": manifest.case_id,
+ "suite": manifest.task.suite,
+ "task_name": manifest.task.task_name,
+ "instruction": manifest.task.instruction,
+ "init_state_index": manifest.episodes.scored_init_state_index,
+ "success": native["success"],
+ "score": native["score"],
+ "reward": native["reward"],
+ "terminal_reason": native["terminal_reason"],
+ "policy_ticks": native["policy_ticks"],
+ "backend_ticks": native["backend_ticks"],
+ "debug_submissions": len(exploration.trials),
+ "policy_sha256": exploration.policy.sha256,
+ "policy_execution_status": native["policy_execution_status"],
+ "source_revision": manifest.source.revision,
+ "dataset_revision": manifest.source.dataset_revision,
+ "deviations": ["continuous_real_time", "joint_target_to_native_osc_adapter"],
+ }
+ (scored_path / "score.json").write_text(
+ json.dumps(native_value, indent=2, sort_keys=True) + "\n"
+ )
+ return EvaluationReport(
+ summary=(
+ SummaryItem(key="result_type", label="Result type", value="single_trial"),
+ SummaryItem(key="suite", label="Suite", value=manifest.task.suite),
+ SummaryItem(key="task", label="Task", value=manifest.task.instruction),
+ SummaryItem(key="success", label="Native success", value=trial.outcome.success),
+ SummaryItem(key="score", label="Native score", value=native["score"]),
+ SummaryItem(
+ key="terminal_reason",
+ label="Terminal reason",
+ value=native["terminal_reason"],
+ ),
+ SummaryItem(key="policy_ticks", label="Policy ticks", value=native["policy_ticks"]),
+ SummaryItem(
+ key="debug_trials", label="Debug trials", value=len(exploration.trials)
+ ),
+ ),
+ native_result=InlineNativeResult(value=native_value),
+ artifacts=(
+ _artifact("scored-trial/task.json", "Task manifest", "application/json"),
+ _artifact("scored-trial/score.json", "Native score", "application/json"),
+ _artifact("scored-trial/trial.jsonl", "Trial log", "application/x-ndjson"),
+ _artifact("scored-trial/container.log", "Container log", "text/plain"),
+ _artifact("scored-trial/trial.mp4", "Rendered trial", "video/mp4"),
+ _artifact(
+ "scored-trial/recording.db", "Memory2 recording", "application/x-sqlite3"
+ ),
+ ),
+ )
+
+
+class NativeTrialResult(TypedDict):
+ success: bool
+ score: float
+ reward: float
+ terminal_reason: str
+ policy_ticks: int
+ backend_ticks: int
+ policy_execution_status: str
+
+
+def _run_trial(
+ manifest: LiberoTaskManifest,
+ assets: PreparedAssets,
+ policy: PolicyArtifact,
+ *,
+ init_index: int,
+ path: Path,
+ runtime: CodePolicyRuntime,
+ run_id: str,
+) -> tuple[TrialRun, NativeTrialResult]:
+ path.mkdir(parents=True, exist_ok=True)
+ log_path = path / "trial.jsonl"
+ memory_path = path / "recording.db"
+ video_path = path / "trial.mp4"
+ discovery = str(path / "panda-shm")
+ container = LiberoPodmanContainer(manifest, assets, artifact_dir=path)
+ control = None
+ coordinator = None
+ execution = None
+ started = time.monotonic()
+ native: NativeTrialResult
+ previous_run_id = os.environ.get(DIMOS_RUN_ID_ENV)
+ try:
+ endpoints = container.start()
+ _log(log_path, "container_started", run_id=run_id)
+ control = EvaluationControlClient(endpoints.control, endpoints.control_token)
+ control.wait_ready()
+ control.initialize(manifest, init_index)
+ blueprint = libero_trial_blueprint(
+ policy_endpoint=endpoints.policy,
+ discovery_address=discovery,
+ memory_path=memory_path,
+ video_path=video_path,
+ )
+ os.environ[DIMOS_RUN_ID_ENV] = run_id
+ coordinator = ModuleCoordinator.build(blueprint)
+ coordinator.start_rpc_service()
+ execution = runtime.prepare(
+ policy,
+ memory_path=memory_path,
+ startup_timeout_s=30.0,
+ )
+ control.start()
+ execution.start()
+ timeout = manifest.contract.horizon_ticks / manifest.contract.control_frequency_hz + 15.0
+ terminal = control.wait_terminal(timeout)
+ policy_result = execution.finish()
+ policy_execution_status = "completed" if terminal.success else policy_result.status
+ native = {
+ "success": bool(terminal.success),
+ "score": float(terminal.score),
+ "reward": float(terminal.reward),
+ "terminal_reason": terminal.terminal_reason,
+ "policy_ticks": int(terminal.policy_ticks),
+ "backend_ticks": int(terminal.backend_ticks),
+ "policy_execution_status": policy_execution_status,
+ }
+ _log(log_path, "trial_terminal", **native)
+ status: Literal["completed", "policy_error", "infrastructure_error"]
+ if terminal.terminal_reason == "failure" or policy_result.status == "infrastructure_error":
+ status = "infrastructure_error"
+ elif policy_execution_status == "policy_error":
+ status = "policy_error"
+ else:
+ status = "completed"
+ error = terminal.error or ("" if terminal.success else policy_result.error)
+ except BaseException as exc:
+ if execution is not None:
+ try:
+ execution.finish()
+ except Exception:
+ pass
+ if control is not None:
+ try:
+ control.cancel()
+ except Exception:
+ pass
+ status = "infrastructure_error"
+ error = f"{type(exc).__name__}: {exc}"
+ native = {
+ "success": False,
+ "score": 0.0,
+ "reward": 0.0,
+ "terminal_reason": "failure",
+ "policy_ticks": 0,
+ "backend_ticks": 0,
+ "policy_execution_status": "infrastructure_error",
+ }
+ _log(log_path, "trial_error", error=error)
+ if isinstance(exc, KeyboardInterrupt):
+ raise
+ finally:
+ if coordinator is not None:
+ coordinator.stop()
+ if control is not None:
+ control.close()
+ container.stop()
+ if previous_run_id is None:
+ os.environ.pop(DIMOS_RUN_ID_ENV, None)
+ else:
+ os.environ[DIMOS_RUN_ID_ENV] = previous_run_id
+ if status == "completed" and (not video_path.is_file() or video_path.stat().st_size == 0):
+ status = "infrastructure_error"
+ error = "LIBERO trial did not produce a rendered video artifact"
+ outcome = TrialOutcome(
+ success=bool(native["success"]),
+ reward=float(native["reward"]),
+ status=status,
+ error=error,
+ duration_seconds=time.monotonic() - started,
+ )
+ return (
+ TrialRun(
+ run_id=run_id,
+ outcome=outcome,
+ artifacts=path,
+ log_path=log_path,
+ memory_path=memory_path,
+ ),
+ native,
+ )
+
+
+def _log(path: Path, event: str, **values: object) -> None:
+ record = {
+ "timestamp": datetime.now(timezone.utc).isoformat(),
+ "module": "libero-pro-evaluation",
+ "event": event,
+ **values,
+ }
+ with path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(record, sort_keys=True) + "\n")
+
+
+def _artifact(path: str, label: str, media_type: str) -> ArtifactReference:
+ return ArtifactReference(path=path, label=label, media_type=media_type)
+
+
+libero_pro = LiberoProEvaluation()
diff --git a/dimos/benchmark/libero_pro/models.py b/dimos/benchmark/libero_pro/models.py
new file mode 100644
index 0000000000..8d3fddc296
--- /dev/null
+++ b/dimos/benchmark/libero_pro/models.py
@@ -0,0 +1,94 @@
+"""Strict persisted contracts for one LIBERO-PRO task evaluation."""
+
+from __future__ import annotations
+
+from typing import Literal
+
+from pydantic import BaseModel, ConfigDict, Field, model_validator
+
+
+class LiberoModel(BaseModel):
+ model_config = ConfigDict(extra="forbid", frozen=True, strict=True)
+
+
+class SourceIdentity(LiberoModel):
+ repository: Literal["Zxy-MLlab/LIBERO-PRO"]
+ revision: str = Field(min_length=40, max_length=40)
+ dataset_repository: Literal["zhouxueyang/LIBERO-Pro"]
+ dataset_revision: str = Field(min_length=40, max_length=40)
+
+
+class TaskIdentity(LiberoModel):
+ suite: str = Field(min_length=1)
+ task_order_index: int = Field(ge=0)
+ task_index: int = Field(ge=0)
+ task_name: str = Field(min_length=1)
+ instruction: str = Field(min_length=1)
+
+
+class AssetReference(LiberoModel):
+ repository_path: str = Field(min_length=1)
+ sha256: str = Field(pattern=r"^[0-9a-f]{64}$")
+ size_bytes: int = Field(gt=0)
+
+
+class TaskAssets(LiberoModel):
+ bddl: AssetReference
+ init_states: AssetReference
+
+
+class EpisodeSelection(LiberoModel):
+ debug_init_state_indices: tuple[int, ...]
+ scored_init_state_index: int = Field(ge=0)
+
+ @model_validator(mode="after")
+ def rows_are_distinct(self) -> EpisodeSelection:
+ if len(self.debug_init_state_indices) != 5:
+ raise ValueError("exactly five debug init-state indices are required")
+ if any(index < 0 for index in self.debug_init_state_indices):
+ raise ValueError("debug init-state indices must be non-negative")
+ if len(set(self.debug_init_state_indices)) != 5:
+ raise ValueError("debug init-state indices must be distinct")
+ if self.scored_init_state_index in self.debug_init_state_indices:
+ raise ValueError("scored init-state index must not be used for debugging")
+ return self
+
+
+class CameraContract(LiberoModel):
+ name: Literal["agentview", "robot0_eye_in_hand"]
+ width: Literal[128]
+ height: Literal[128]
+
+
+class ComparisonContract(LiberoModel):
+ robot: Literal["Panda"]
+ controller: Literal["OSC_POSE"]
+ cameras: tuple[CameraContract, CameraContract]
+ control_frequency_hz: Literal[20]
+ settling_ticks: Literal[5]
+ horizon_ticks: int = Field(gt=0)
+ clock: Literal["continuous_real_time"]
+ success: Literal["native_bddl_goal_predicates"]
+
+ @model_validator(mode="after")
+ def cameras_are_exact(self) -> ComparisonContract:
+ if tuple(camera.name for camera in self.cameras) != (
+ "agentview",
+ "robot0_eye_in_hand",
+ ):
+ raise ValueError("the two benchmark camera contracts must be ordered and complete")
+ return self
+
+
+class LiberoTaskManifest(LiberoModel):
+ schema_version: Literal["1.0"] = "1.0"
+ case_id: str = Field(min_length=1)
+ source: SourceIdentity
+ task: TaskIdentity
+ assets: TaskAssets
+ episodes: EpisodeSelection
+ contract: ComparisonContract
+
+
+class LiberoProConfig(LiberoModel):
+ task_manifest: str = Field(min_length=1)
diff --git a/dimos/benchmark/libero_pro/panda.urdf b/dimos/benchmark/libero_pro/panda.urdf
new file mode 100644
index 0000000000..0995d22b9e
--- /dev/null
+++ b/dimos/benchmark/libero_pro/panda.urdf
@@ -0,0 +1,52 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/dimos/benchmark/libero_pro/podman.py b/dimos/benchmark/libero_pro/podman.py
new file mode 100644
index 0000000000..f7c7fdab0c
--- /dev/null
+++ b/dimos/benchmark/libero_pro/podman.py
@@ -0,0 +1,162 @@
+"""Rootless Podman lifecycle for one isolated LIBERO-PRO trial."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+import hashlib
+from pathlib import Path
+import secrets
+import subprocess
+from uuid import uuid4
+
+from dimos.benchmark.libero_pro.assets import PreparedAssets
+from dimos.benchmark.libero_pro.models import LiberoTaskManifest
+from dimos.constants import DIMOS_PROJECT_ROOT
+
+
+def _image_source_digest() -> str:
+ """Identify the exact local sources copied into the trial image."""
+ sources = [
+ path
+ for root in (
+ DIMOS_PROJECT_ROOT / "docker" / "libero-pro",
+ DIMOS_PROJECT_ROOT / "dimos" / "benchmark" / "libero_pro" / "proto",
+ )
+ for path in root.rglob("*")
+ if path.is_file()
+ and (path.name == "Dockerfile" or path.suffix in {".proto", ".py", ".pyi", ".txt", ".yaml"})
+ ]
+ digest = hashlib.sha256()
+ for path in sorted(sources):
+ digest.update(path.relative_to(DIMOS_PROJECT_ROOT).as_posix().encode())
+ digest.update(path.read_bytes())
+ return digest.hexdigest()[:12]
+
+
+IMAGE = f"localhost/dimos-libero-pro:{_image_source_digest()}"
+POLICY_CONTAINER_PORT = 50051
+CONTROL_CONTAINER_PORT = 50052
+
+
+class PodmanError(RuntimeError):
+ """Podman could not build or run the LIBERO-PRO image."""
+
+
+@dataclass(frozen=True)
+class ContainerEndpoints:
+ policy: str
+ control: str
+ control_token: str
+
+
+class LiberoPodmanContainer:
+ def __init__(
+ self,
+ manifest: LiberoTaskManifest,
+ assets: PreparedAssets,
+ *,
+ artifact_dir: Path,
+ ) -> None:
+ self.manifest = manifest
+ self.assets = assets
+ self.artifact_dir = artifact_dir
+ self.name = f"dimos-libero-{uuid4().hex}"
+ self.token = secrets.token_urlsafe(32)
+ self._running = False
+
+ @staticmethod
+ def ensure_image() -> None:
+ exists = subprocess.run(
+ ["podman", "image", "exists", IMAGE],
+ check=False,
+ )
+ if exists.returncode == 0:
+ return
+ _run(
+ [
+ "podman",
+ "build",
+ "--tag",
+ IMAGE,
+ "--file",
+ str(DIMOS_PROJECT_ROOT / "docker" / "libero-pro" / "Dockerfile"),
+ str(DIMOS_PROJECT_ROOT),
+ ]
+ )
+
+ def start(self) -> ContainerEndpoints:
+ if self._running:
+ raise RuntimeError("LIBERO-PRO container is already running")
+ self.artifact_dir.mkdir(parents=True, exist_ok=True)
+ manifest_path = self.artifact_dir / "task.json"
+ manifest_path.write_text(self.manifest.model_dump_json(indent=2) + "\n")
+ _run(
+ [
+ "podman",
+ "run",
+ "--detach",
+ "--rm",
+ "--name",
+ self.name,
+ "--publish",
+ f"127.0.0.1::{POLICY_CONTAINER_PORT}",
+ "--publish",
+ f"127.0.0.1::{CONTROL_CONTAINER_PORT}",
+ "--env",
+ f"DIMOS_LIBERO_CONTROL_TOKEN={self.token}",
+ "--volume",
+ f"{self.assets.bddl}:/task/task.bddl:ro,Z",
+ "--volume",
+ f"{self.assets.init_states}:/task/init_states.pruned_init:ro,Z",
+ "--volume",
+ f"{manifest_path}:/task/task.json:ro,Z",
+ IMAGE,
+ ]
+ )
+ self._running = True
+ return ContainerEndpoints(
+ policy=f"127.0.0.1:{_published_port(self.name, POLICY_CONTAINER_PORT)}",
+ control=f"127.0.0.1:{_published_port(self.name, CONTROL_CONTAINER_PORT)}",
+ control_token=self.token,
+ )
+
+ def stop(self) -> None:
+ if not self._running:
+ return
+ logs = subprocess.run(
+ ["podman", "logs", self.name],
+ check=False,
+ capture_output=True,
+ text=True,
+ )
+ (self.artifact_dir / "container.log").write_text(
+ logs.stdout + logs.stderr,
+ encoding="utf-8",
+ )
+ subprocess.run(["podman", "stop", "--time", "2", self.name], check=False)
+ subprocess.run(["podman", "rm", "--force", self.name], check=False)
+ self._running = False
+
+ def __enter__(self) -> LiberoPodmanContainer:
+ self.start()
+ return self
+
+ def __exit__(self, *_args: object) -> None:
+ self.stop()
+
+
+def _published_port(name: str, container_port: int) -> int:
+ output = _run(["podman", "port", name, f"{container_port}/tcp"])
+ try:
+ return int(output.rsplit(":", 1)[1])
+ except (IndexError, ValueError) as exc:
+ raise PodmanError(f"invalid Podman port mapping: {output!r}") from exc
+
+
+def _run(command: list[str]) -> str:
+ try:
+ result = subprocess.run(command, check=True, capture_output=True, text=True)
+ except (OSError, subprocess.CalledProcessError) as exc:
+ stderr = getattr(exc, "stderr", "")
+ raise PodmanError(f"Podman command failed: {stderr or exc}") from exc
+ return result.stdout.strip()
diff --git a/dimos/benchmark/libero_pro/proto/libero_pro.proto b/dimos/benchmark/libero_pro/proto/libero_pro.proto
new file mode 100644
index 0000000000..ec6f085d39
--- /dev/null
+++ b/dimos/benchmark/libero_pro/proto/libero_pro.proto
@@ -0,0 +1,67 @@
+syntax = "proto3";
+
+package dimos.benchmark.libero_pro.v1;
+
+message Empty {}
+message Health { bool ready = 1; string detail = 2; }
+message WatchRequest {}
+message CameraFrame {
+ string camera = 1;
+ uint32 width = 2;
+ uint32 height = 3;
+ bytes rgb = 4;
+ bytes depth_meters = 5;
+ repeated double intrinsic = 6;
+ repeated double camera_to_robot_base = 7;
+}
+message RobotSnapshot {
+ uint64 tick = 1;
+ double timestamp_s = 2;
+ repeated double joint_position = 3;
+ repeated double joint_velocity = 4;
+ double gripper_position = 5;
+ repeated CameraFrame cameras = 6;
+}
+message JointTargets {
+ repeated double joint_position = 1;
+ double gripper_position = 2;
+ uint64 sequence = 3;
+}
+message Ack { uint64 sequence = 1; }
+message InitializeTrialRequest {
+ string suite = 1;
+ uint32 task_order_index = 2;
+ uint32 task_index = 3;
+ uint32 init_state_index = 4;
+ uint32 horizon_ticks = 5;
+ uint32 control_frequency_hz = 6;
+ uint32 settling_ticks = 7;
+}
+message TrialReady {
+ string task_name = 1;
+ string instruction = 2;
+}
+message TerminalResult {
+ bool success = 1;
+ double score = 2;
+ double reward = 3;
+ string terminal_reason = 4;
+ uint32 policy_ticks = 5;
+ uint32 backend_ticks = 6;
+ string error = 7;
+}
+
+service PolicyInterface {
+ rpc GetHealth(Empty) returns (Health);
+ rpc WatchState(WatchRequest) returns (stream RobotSnapshot);
+ rpc SetJointTargets(JointTargets) returns (Ack);
+}
+
+service EvaluationControl {
+ rpc GetHealth(Empty) returns (Health);
+ rpc InitializeTrial(InitializeTrialRequest) returns (TrialReady);
+ rpc StartTrial(Empty) returns (Empty);
+ rpc WaitForTerminal(Empty) returns (TerminalResult);
+ rpc CancelTrial(Empty) returns (TerminalResult);
+ rpc GetNativeResult(Empty) returns (TerminalResult);
+}
diff --git a/dimos/benchmark/libero_pro/proto/libero_pro_pb2.py b/dimos/benchmark/libero_pro/proto/libero_pro_pb2.py
new file mode 100644
index 0000000000..4a09ebda71
--- /dev/null
+++ b/dimos/benchmark/libero_pro/proto/libero_pro_pb2.py
@@ -0,0 +1,49 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# NO CHECKED-IN PROTOBUF GENCODE
+# source: libero_pro.proto
+# Protobuf Python Version: 7.35.1
+"""Generated protocol buffer code."""
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import descriptor_pool as _descriptor_pool
+from google.protobuf import symbol_database as _symbol_database
+from google.protobuf.internal import builder as _builder
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+
+
+DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x10libero_pro.proto\x12\x1d\x64imos.benchmark.libero_pro.v1\"\x07\n\x05\x45mpty\"\'\n\x06Health\x12\r\n\x05ready\x18\x01 \x01(\x08\x12\x0e\n\x06\x64\x65tail\x18\x02 \x01(\t\"\x0e\n\x0cWatchRequest\"\x90\x01\n\x0b\x43\x61meraFrame\x12\x0e\n\x06\x63\x61mera\x18\x01 \x01(\t\x12\r\n\x05width\x18\x02 \x01(\r\x12\x0e\n\x06height\x18\x03 \x01(\r\x12\x0b\n\x03rgb\x18\x04 \x01(\x0c\x12\x14\n\x0c\x64\x65pth_meters\x18\x05 \x01(\x0c\x12\x11\n\tintrinsic\x18\x06 \x03(\x01\x12\x1c\n\x14\x63\x61mera_to_robot_base\x18\x07 \x03(\x01\"\xb9\x01\n\rRobotSnapshot\x12\x0c\n\x04tick\x18\x01 \x01(\x04\x12\x13\n\x0btimestamp_s\x18\x02 \x01(\x01\x12\x16\n\x0ejoint_position\x18\x03 \x03(\x01\x12\x16\n\x0ejoint_velocity\x18\x04 \x03(\x01\x12\x18\n\x10gripper_position\x18\x05 \x01(\x01\x12;\n\x07\x63\x61meras\x18\x06 \x03(\x0b\x32*.dimos.benchmark.libero_pro.v1.CameraFrame\"R\n\x0cJointTargets\x12\x16\n\x0ejoint_position\x18\x01 \x03(\x01\x12\x18\n\x10gripper_position\x18\x02 \x01(\x01\x12\x10\n\x08sequence\x18\x03 \x01(\x04\"\x17\n\x03\x41\x63k\x12\x10\n\x08sequence\x18\x01 \x01(\x04\"\xbc\x01\n\x16InitializeTrialRequest\x12\r\n\x05suite\x18\x01 \x01(\t\x12\x18\n\x10task_order_index\x18\x02 \x01(\r\x12\x12\n\ntask_index\x18\x03 \x01(\r\x12\x18\n\x10init_state_index\x18\x04 \x01(\r\x12\x15\n\rhorizon_ticks\x18\x05 \x01(\r\x12\x1c\n\x14\x63ontrol_frequency_hz\x18\x06 \x01(\r\x12\x16\n\x0esettling_ticks\x18\x07 \x01(\r\"4\n\nTrialReady\x12\x11\n\ttask_name\x18\x01 \x01(\t\x12\x13\n\x0binstruction\x18\x02 \x01(\t\"\x95\x01\n\x0eTerminalResult\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\r\n\x05score\x18\x02 \x01(\x01\x12\x0e\n\x06reward\x18\x03 \x01(\x01\x12\x17\n\x0fterminal_reason\x18\x04 \x01(\t\x12\x14\n\x0cpolicy_ticks\x18\x05 \x01(\r\x12\x15\n\rbackend_ticks\x18\x06 \x01(\r\x12\r\n\x05\x65rror\x18\x07 \x01(\t2\xba\x02\n\x0fPolicyInterface\x12X\n\tGetHealth\x12$.dimos.benchmark.libero_pro.v1.Empty\x1a%.dimos.benchmark.libero_pro.v1.Health\x12i\n\nWatchState\x12+.dimos.benchmark.libero_pro.v1.WatchRequest\x1a,.dimos.benchmark.libero_pro.v1.RobotSnapshot0\x01\x12\x62\n\x0fSetJointTargets\x12+.dimos.benchmark.libero_pro.v1.JointTargets\x1a\".dimos.benchmark.libero_pro.v1.Ack2\xf0\x04\n\x11\x45valuationControl\x12X\n\tGetHealth\x12$.dimos.benchmark.libero_pro.v1.Empty\x1a%.dimos.benchmark.libero_pro.v1.Health\x12s\n\x0fInitializeTrial\x12\x35.dimos.benchmark.libero_pro.v1.InitializeTrialRequest\x1a).dimos.benchmark.libero_pro.v1.TrialReady\x12X\n\nStartTrial\x12$.dimos.benchmark.libero_pro.v1.Empty\x1a$.dimos.benchmark.libero_pro.v1.Empty\x12\x66\n\x0fWaitForTerminal\x12$.dimos.benchmark.libero_pro.v1.Empty\x1a-.dimos.benchmark.libero_pro.v1.TerminalResult\x12\x62\n\x0b\x43\x61ncelTrial\x12$.dimos.benchmark.libero_pro.v1.Empty\x1a-.dimos.benchmark.libero_pro.v1.TerminalResult\x12\x66\n\x0fGetNativeResult\x12$.dimos.benchmark.libero_pro.v1.Empty\x1a-.dimos.benchmark.libero_pro.v1.TerminalResultb\x06proto3')
+
+_globals = globals()
+_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
+_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'libero_pro_pb2', _globals)
+if not _descriptor._USE_C_DESCRIPTORS:
+ DESCRIPTOR._loaded_options = None
+ _globals['_EMPTY']._serialized_start=51
+ _globals['_EMPTY']._serialized_end=58
+ _globals['_HEALTH']._serialized_start=60
+ _globals['_HEALTH']._serialized_end=99
+ _globals['_WATCHREQUEST']._serialized_start=101
+ _globals['_WATCHREQUEST']._serialized_end=115
+ _globals['_CAMERAFRAME']._serialized_start=118
+ _globals['_CAMERAFRAME']._serialized_end=262
+ _globals['_ROBOTSNAPSHOT']._serialized_start=265
+ _globals['_ROBOTSNAPSHOT']._serialized_end=450
+ _globals['_JOINTTARGETS']._serialized_start=452
+ _globals['_JOINTTARGETS']._serialized_end=534
+ _globals['_ACK']._serialized_start=536
+ _globals['_ACK']._serialized_end=559
+ _globals['_INITIALIZETRIALREQUEST']._serialized_start=562
+ _globals['_INITIALIZETRIALREQUEST']._serialized_end=750
+ _globals['_TRIALREADY']._serialized_start=752
+ _globals['_TRIALREADY']._serialized_end=804
+ _globals['_TERMINALRESULT']._serialized_start=807
+ _globals['_TERMINALRESULT']._serialized_end=956
+ _globals['_POLICYINTERFACE']._serialized_start=959
+ _globals['_POLICYINTERFACE']._serialized_end=1273
+ _globals['_EVALUATIONCONTROL']._serialized_start=1276
+ _globals['_EVALUATIONCONTROL']._serialized_end=1900
+# @@protoc_insertion_point(module_scope)
diff --git a/dimos/benchmark/libero_pro/proto/libero_pro_pb2.pyi b/dimos/benchmark/libero_pro/proto/libero_pro_pb2.pyi
new file mode 100644
index 0000000000..8283b5da21
--- /dev/null
+++ b/dimos/benchmark/libero_pro/proto/libero_pro_pb2.pyi
@@ -0,0 +1,117 @@
+from google.protobuf.internal import containers as _containers
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from collections.abc import Iterable as _Iterable, Mapping as _Mapping
+from typing import ClassVar as _ClassVar, Optional as _Optional, Union as _Union
+
+DESCRIPTOR: _descriptor.FileDescriptor
+
+class Empty(_message.Message):
+ __slots__ = ()
+ def __init__(self) -> None: ...
+
+class Health(_message.Message):
+ __slots__ = ("ready", "detail")
+ READY_FIELD_NUMBER: _ClassVar[int]
+ DETAIL_FIELD_NUMBER: _ClassVar[int]
+ ready: bool
+ detail: str
+ def __init__(self, ready: _Optional[bool] = ..., detail: _Optional[str] = ...) -> None: ...
+
+class WatchRequest(_message.Message):
+ __slots__ = ()
+ def __init__(self) -> None: ...
+
+class CameraFrame(_message.Message):
+ __slots__ = ("camera", "width", "height", "rgb", "depth_meters", "intrinsic", "camera_to_robot_base")
+ CAMERA_FIELD_NUMBER: _ClassVar[int]
+ WIDTH_FIELD_NUMBER: _ClassVar[int]
+ HEIGHT_FIELD_NUMBER: _ClassVar[int]
+ RGB_FIELD_NUMBER: _ClassVar[int]
+ DEPTH_METERS_FIELD_NUMBER: _ClassVar[int]
+ INTRINSIC_FIELD_NUMBER: _ClassVar[int]
+ CAMERA_TO_ROBOT_BASE_FIELD_NUMBER: _ClassVar[int]
+ camera: str
+ width: int
+ height: int
+ rgb: bytes
+ depth_meters: bytes
+ intrinsic: _containers.RepeatedScalarFieldContainer[float]
+ camera_to_robot_base: _containers.RepeatedScalarFieldContainer[float]
+ def __init__(self, camera: _Optional[str] = ..., width: _Optional[int] = ..., height: _Optional[int] = ..., rgb: _Optional[bytes] = ..., depth_meters: _Optional[bytes] = ..., intrinsic: _Optional[_Iterable[float]] = ..., camera_to_robot_base: _Optional[_Iterable[float]] = ...) -> None: ...
+
+class RobotSnapshot(_message.Message):
+ __slots__ = ("tick", "timestamp_s", "joint_position", "joint_velocity", "gripper_position", "cameras")
+ TICK_FIELD_NUMBER: _ClassVar[int]
+ TIMESTAMP_S_FIELD_NUMBER: _ClassVar[int]
+ JOINT_POSITION_FIELD_NUMBER: _ClassVar[int]
+ JOINT_VELOCITY_FIELD_NUMBER: _ClassVar[int]
+ GRIPPER_POSITION_FIELD_NUMBER: _ClassVar[int]
+ CAMERAS_FIELD_NUMBER: _ClassVar[int]
+ tick: int
+ timestamp_s: float
+ joint_position: _containers.RepeatedScalarFieldContainer[float]
+ joint_velocity: _containers.RepeatedScalarFieldContainer[float]
+ gripper_position: float
+ cameras: _containers.RepeatedCompositeFieldContainer[CameraFrame]
+ def __init__(self, tick: _Optional[int] = ..., timestamp_s: _Optional[float] = ..., joint_position: _Optional[_Iterable[float]] = ..., joint_velocity: _Optional[_Iterable[float]] = ..., gripper_position: _Optional[float] = ..., cameras: _Optional[_Iterable[_Union[CameraFrame, _Mapping]]] = ...) -> None: ...
+
+class JointTargets(_message.Message):
+ __slots__ = ("joint_position", "gripper_position", "sequence")
+ JOINT_POSITION_FIELD_NUMBER: _ClassVar[int]
+ GRIPPER_POSITION_FIELD_NUMBER: _ClassVar[int]
+ SEQUENCE_FIELD_NUMBER: _ClassVar[int]
+ joint_position: _containers.RepeatedScalarFieldContainer[float]
+ gripper_position: float
+ sequence: int
+ def __init__(self, joint_position: _Optional[_Iterable[float]] = ..., gripper_position: _Optional[float] = ..., sequence: _Optional[int] = ...) -> None: ...
+
+class Ack(_message.Message):
+ __slots__ = ("sequence",)
+ SEQUENCE_FIELD_NUMBER: _ClassVar[int]
+ sequence: int
+ def __init__(self, sequence: _Optional[int] = ...) -> None: ...
+
+class InitializeTrialRequest(_message.Message):
+ __slots__ = ("suite", "task_order_index", "task_index", "init_state_index", "horizon_ticks", "control_frequency_hz", "settling_ticks")
+ SUITE_FIELD_NUMBER: _ClassVar[int]
+ TASK_ORDER_INDEX_FIELD_NUMBER: _ClassVar[int]
+ TASK_INDEX_FIELD_NUMBER: _ClassVar[int]
+ INIT_STATE_INDEX_FIELD_NUMBER: _ClassVar[int]
+ HORIZON_TICKS_FIELD_NUMBER: _ClassVar[int]
+ CONTROL_FREQUENCY_HZ_FIELD_NUMBER: _ClassVar[int]
+ SETTLING_TICKS_FIELD_NUMBER: _ClassVar[int]
+ suite: str
+ task_order_index: int
+ task_index: int
+ init_state_index: int
+ horizon_ticks: int
+ control_frequency_hz: int
+ settling_ticks: int
+ def __init__(self, suite: _Optional[str] = ..., task_order_index: _Optional[int] = ..., task_index: _Optional[int] = ..., init_state_index: _Optional[int] = ..., horizon_ticks: _Optional[int] = ..., control_frequency_hz: _Optional[int] = ..., settling_ticks: _Optional[int] = ...) -> None: ...
+
+class TrialReady(_message.Message):
+ __slots__ = ("task_name", "instruction")
+ TASK_NAME_FIELD_NUMBER: _ClassVar[int]
+ INSTRUCTION_FIELD_NUMBER: _ClassVar[int]
+ task_name: str
+ instruction: str
+ def __init__(self, task_name: _Optional[str] = ..., instruction: _Optional[str] = ...) -> None: ...
+
+class TerminalResult(_message.Message):
+ __slots__ = ("success", "score", "reward", "terminal_reason", "policy_ticks", "backend_ticks", "error")
+ SUCCESS_FIELD_NUMBER: _ClassVar[int]
+ SCORE_FIELD_NUMBER: _ClassVar[int]
+ REWARD_FIELD_NUMBER: _ClassVar[int]
+ TERMINAL_REASON_FIELD_NUMBER: _ClassVar[int]
+ POLICY_TICKS_FIELD_NUMBER: _ClassVar[int]
+ BACKEND_TICKS_FIELD_NUMBER: _ClassVar[int]
+ ERROR_FIELD_NUMBER: _ClassVar[int]
+ success: bool
+ score: float
+ reward: float
+ terminal_reason: str
+ policy_ticks: int
+ backend_ticks: int
+ error: str
+ def __init__(self, success: _Optional[bool] = ..., score: _Optional[float] = ..., reward: _Optional[float] = ..., terminal_reason: _Optional[str] = ..., policy_ticks: _Optional[int] = ..., backend_ticks: _Optional[int] = ..., error: _Optional[str] = ...) -> None: ...
diff --git a/dimos/benchmark/libero_pro/proto/libero_pro_pb2_grpc.py b/dimos/benchmark/libero_pro/proto/libero_pro_pb2_grpc.py
new file mode 100644
index 0000000000..6cd1558927
--- /dev/null
+++ b/dimos/benchmark/libero_pro/proto/libero_pro_pb2_grpc.py
@@ -0,0 +1,102 @@
+# mypy: ignore-errors
+"""gRPC bindings generated from libero_pro.proto."""
+
+import grpc
+
+from . import libero_pro_pb2 as pb2
+
+
+class PolicyInterfaceStub:
+ def __init__(self, channel: grpc.Channel) -> None:
+ self.GetHealth = channel.unary_unary(
+ "/dimos.benchmark.libero_pro.v1.PolicyInterface/GetHealth",
+ request_serializer=pb2.Empty.SerializeToString,
+ response_deserializer=pb2.Health.FromString,
+ )
+ self.WatchState = channel.unary_stream(
+ "/dimos.benchmark.libero_pro.v1.PolicyInterface/WatchState",
+ request_serializer=pb2.WatchRequest.SerializeToString,
+ response_deserializer=pb2.RobotSnapshot.FromString,
+ )
+ self.SetJointTargets = channel.unary_unary(
+ "/dimos.benchmark.libero_pro.v1.PolicyInterface/SetJointTargets",
+ request_serializer=pb2.JointTargets.SerializeToString,
+ response_deserializer=pb2.Ack.FromString,
+ )
+
+
+class EvaluationControlStub:
+ def __init__(self, channel: grpc.Channel) -> None:
+ prefix = "/dimos.benchmark.libero_pro.v1.EvaluationControl/"
+ self.GetHealth = channel.unary_unary(
+ prefix + "GetHealth",
+ request_serializer=pb2.Empty.SerializeToString,
+ response_deserializer=pb2.Health.FromString,
+ )
+ self.InitializeTrial = channel.unary_unary(
+ prefix + "InitializeTrial",
+ request_serializer=pb2.InitializeTrialRequest.SerializeToString,
+ response_deserializer=pb2.TrialReady.FromString,
+ )
+ for name in ("StartTrial", "WaitForTerminal", "CancelTrial", "GetNativeResult"):
+ response = pb2.Empty if name == "StartTrial" else pb2.TerminalResult
+ setattr(
+ self,
+ name,
+ channel.unary_unary(
+ prefix + name,
+ request_serializer=pb2.Empty.SerializeToString,
+ response_deserializer=response.FromString,
+ ),
+ )
+
+
+def add_PolicyInterfaceServicer_to_server(servicer, server) -> None:
+ handlers = {
+ "GetHealth": grpc.unary_unary_rpc_method_handler(
+ servicer.GetHealth,
+ request_deserializer=pb2.Empty.FromString,
+ response_serializer=pb2.Health.SerializeToString,
+ ),
+ "WatchState": grpc.unary_stream_rpc_method_handler(
+ servicer.WatchState,
+ request_deserializer=pb2.WatchRequest.FromString,
+ response_serializer=pb2.RobotSnapshot.SerializeToString,
+ ),
+ "SetJointTargets": grpc.unary_unary_rpc_method_handler(
+ servicer.SetJointTargets,
+ request_deserializer=pb2.JointTargets.FromString,
+ response_serializer=pb2.Ack.SerializeToString,
+ ),
+ }
+ server.add_generic_rpc_handlers(
+ (
+ grpc.method_handlers_generic_handler(
+ "dimos.benchmark.libero_pro.v1.PolicyInterface", handlers
+ ),
+ )
+ )
+
+
+def add_EvaluationControlServicer_to_server(servicer, server) -> None:
+ handlers = {}
+ for name, request, response in (
+ ("GetHealth", pb2.Empty, pb2.Health),
+ ("InitializeTrial", pb2.InitializeTrialRequest, pb2.TrialReady),
+ ("StartTrial", pb2.Empty, pb2.Empty),
+ ("WaitForTerminal", pb2.Empty, pb2.TerminalResult),
+ ("CancelTrial", pb2.Empty, pb2.TerminalResult),
+ ("GetNativeResult", pb2.Empty, pb2.TerminalResult),
+ ):
+ handlers[name] = grpc.unary_unary_rpc_method_handler(
+ getattr(servicer, name),
+ request_deserializer=request.FromString,
+ response_serializer=response.SerializeToString,
+ )
+ server.add_generic_rpc_handlers(
+ (
+ grpc.method_handlers_generic_handler(
+ "dimos.benchmark.libero_pro.v1.EvaluationControl", handlers
+ ),
+ )
+ )
diff --git a/dimos/benchmark/libero_pro/proto/libero_pro_pb2_grpc.pyi b/dimos/benchmark/libero_pro/proto/libero_pro_pb2_grpc.pyi
new file mode 100644
index 0000000000..208508f0db
--- /dev/null
+++ b/dimos/benchmark/libero_pro/proto/libero_pro_pb2_grpc.pyi
@@ -0,0 +1,19 @@
+from typing import Any
+
+class PolicyInterfaceStub:
+ GetHealth: Any
+ WatchState: Any
+ SetJointTargets: Any
+ def __init__(self, channel: Any) -> None: ...
+
+class EvaluationControlStub:
+ GetHealth: Any
+ InitializeTrial: Any
+ StartTrial: Any
+ WaitForTerminal: Any
+ CancelTrial: Any
+ GetNativeResult: Any
+ def __init__(self, channel: Any) -> None: ...
+
+def add_PolicyInterfaceServicer_to_server(servicer: Any, server: Any) -> None: ...
+def add_EvaluationControlServicer_to_server(servicer: Any, server: Any) -> None: ...
diff --git a/dimos/benchmark/libero_pro/test_assets.py b/dimos/benchmark/libero_pro/test_assets.py
new file mode 100644
index 0000000000..3685715926
--- /dev/null
+++ b/dimos/benchmark/libero_pro/test_assets.py
@@ -0,0 +1,32 @@
+import hashlib
+
+import pytest
+
+from dimos.benchmark.libero_pro.assets import LiberoAssetError, _verify
+from dimos.benchmark.libero_pro.models import AssetReference
+
+
+def test_asset_verification_accepts_exact_size_and_digest(tmp_path) -> None:
+ payload = b"verified"
+ path = tmp_path / "task.bddl"
+ path.write_bytes(payload)
+ reference = AssetReference(
+ repository_path="bddl/task.bddl",
+ sha256=hashlib.sha256(payload).hexdigest(),
+ size_bytes=len(payload),
+ )
+
+ _verify(path, reference)
+
+
+def test_asset_verification_rejects_wrong_digest(tmp_path) -> None:
+ path = tmp_path / "task.bddl"
+ path.write_bytes(b"tampered")
+ reference = AssetReference(
+ repository_path="bddl/task.bddl",
+ sha256="0" * 64,
+ size_bytes=8,
+ )
+
+ with pytest.raises(LiberoAssetError, match="digest mismatch"):
+ _verify(path, reference)
diff --git a/dimos/benchmark/libero_pro/test_blueprint.py b/dimos/benchmark/libero_pro/test_blueprint.py
new file mode 100644
index 0000000000..8a3f746bc8
--- /dev/null
+++ b/dimos/benchmark/libero_pro/test_blueprint.py
@@ -0,0 +1,74 @@
+"""Blueprint surface tests for a complete LIBERO-PRO policy run."""
+
+from pathlib import Path
+
+from dimos.benchmark.libero_pro.blueprint import libero_trial_blueprint
+from dimos.benchmark.libero_pro.connection import LiberoConnection, LiberoRecorder
+from dimos.benchmark.libero_pro.video import LiberoVideoRecorder
+from dimos.control.coordinator import ControlCoordinator
+from dimos.manipulation.grasping.grasp_gen_x import GraspGenXModule
+from dimos.manipulation.manipulation_module import ManipulationModule
+from dimos.perception.grounded_segmentation import GroundedSegmentationModule
+
+
+def test_trial_blueprint_exposes_normal_control_manipulation_and_memory(
+ tmp_path: Path,
+) -> None:
+ blueprint = libero_trial_blueprint(
+ policy_endpoint="127.0.0.1:50051",
+ discovery_address=str(tmp_path / "panda-shm"),
+ memory_path=tmp_path / "recording.db",
+ video_path=tmp_path / "trial.mp4",
+ )
+ atoms = {atom.module: atom for atom in blueprint.blueprints}
+
+ assert {
+ LiberoConnection,
+ ControlCoordinator,
+ ManipulationModule,
+ GroundedSegmentationModule,
+ GraspGenXModule,
+ LiberoRecorder,
+ LiberoVideoRecorder,
+ } <= atoms.keys()
+ hardware = atoms[ControlCoordinator].kwargs["hardware"]
+ assert len(hardware) == 1
+ assert hardware[0].adapter_type == "sim_mujoco"
+ assert hardware[0].all_joints == [
+ "panda/joint1",
+ "panda/joint2",
+ "panda/joint3",
+ "panda/joint4",
+ "panda/joint5",
+ "panda/joint6",
+ "panda/joint7",
+ "panda/gripper",
+ ]
+ assert atoms[ControlCoordinator].kwargs["tick_rate"] == 20.0
+ robot_model = atoms[ManipulationModule].kwargs["robots"][0]
+ assert robot_model.planning_groups[0].tip_link == "tcp"
+ assert atoms[LiberoVideoRecorder].kwargs["output_path"] == tmp_path / "trial.mp4"
+ assert atoms[LiberoRecorder].kwargs["record_tf"] is True
+ assert atoms[LiberoRecorder].kwargs["stream_codecs"] == {
+ "agentview_depth_image": "pickle",
+ "eye_in_hand_depth_image": "pickle",
+ }
+ assert atoms[GraspGenXModule].kwargs["gripper"]["extents_open"] == (0.08, 0.04, 0.10)
+ assert atoms[GraspGenXModule].kwargs["max_candidates"] == 25
+
+
+def test_recorder_has_no_privileged_evaluation_streams() -> None:
+ public_inputs = {name for name in LiberoRecorder.__annotations__ if not name.startswith("_")}
+
+ assert public_inputs == {
+ "joint_state",
+ "agentview_color_image",
+ "agentview_depth_image",
+ "agentview_camera_info",
+ "eye_in_hand_color_image",
+ "eye_in_hand_depth_image",
+ "eye_in_hand_camera_info",
+ }
+ assert public_inputs.isdisjoint(
+ {"reward", "success", "goal_predicates", "terminal_reason", "native_result"}
+ )
diff --git a/dimos/benchmark/libero_pro/test_connection.py b/dimos/benchmark/libero_pro/test_connection.py
new file mode 100644
index 0000000000..3c92ead761
--- /dev/null
+++ b/dimos/benchmark/libero_pro/test_connection.py
@@ -0,0 +1,84 @@
+"""Tests for public LIBERO observation conversion."""
+
+import grpc
+import numpy as np
+
+from dimos.benchmark.libero_pro.connection import LiberoConnection, _decode_camera
+from dimos.benchmark.libero_pro.proto import libero_pro_pb2 as pb2
+from dimos.msgs.sensor_msgs.Image import ImageFormat
+
+
+def test_camera_messages_keep_the_shared_simulator_timestamp() -> None:
+ pixels = np.arange(12, dtype=np.uint8).reshape(2, 2, 3)
+ depth = np.array([[0.5, 0.75], [1.0, 1.25]], dtype=np.float32)
+ intrinsic = [100.0, 0.0, 1.0, 0.0, 101.0, 1.0, 0.0, 0.0, 1.0]
+ camera_to_robot_base = np.eye(4)
+ camera_to_robot_base[:3, 3] = [0.1, 0.2, 0.3]
+ frame = pb2.CameraFrame(
+ camera="agentview",
+ width=2,
+ height=2,
+ rgb=pixels.tobytes(),
+ depth_meters=depth.tobytes(),
+ intrinsic=intrinsic,
+ camera_to_robot_base=camera_to_robot_base.reshape(-1),
+ )
+
+ image, depth_image, info, transform = _decode_camera(frame, 123.5)
+
+ assert image.ts == 123.5
+ assert image.frame_id == "agentview_optical"
+ assert image.format is ImageFormat.RGB
+ assert np.array_equal(image.data, pixels)
+ assert depth_image.ts == 123.5
+ assert depth_image.frame_id == "agentview_optical"
+ assert depth_image.format is ImageFormat.DEPTH
+ assert np.array_equal(depth_image.data, depth)
+ assert info.ts == 123.5
+ assert info.frame_id == "agentview_optical"
+ assert info.K == intrinsic
+ assert transform.ts == 123.5
+ assert transform.frame_id == "world"
+ assert transform.child_frame_id == "agentview_optical"
+ assert np.allclose(transform.to_matrix(), camera_to_robot_base)
+
+
+def test_policy_snapshot_has_no_privileged_evaluation_fields() -> None:
+ fields = set(pb2.RobotSnapshot.DESCRIPTOR.fields_by_name)
+
+ assert fields == {
+ "tick",
+ "timestamp_s",
+ "joint_position",
+ "joint_velocity",
+ "gripper_position",
+ "cameras",
+ }
+ assert fields.isdisjoint({"reward", "success", "goal_predicates", "object_poses"})
+
+
+def test_watch_stream_cancellation_is_clean_during_stop() -> None:
+ class CancelledStub:
+ def WatchState(self, _request: pb2.WatchRequest): # type: ignore[no-untyped-def]
+ raise grpc.RpcError("channel closed")
+
+ connection = LiberoConnection(endpoint="unused", discovery_address="unused")
+ try:
+ connection._stub = CancelledStub() # type: ignore[assignment]
+ connection._stop.set()
+ connection._watch()
+ finally:
+ connection.stop()
+
+
+def test_gripper_only_update_retains_last_commanded_arm_target() -> None:
+ connection = LiberoConnection(endpoint="unused", discovery_address="unused")
+ try:
+ commanded = np.arange(7, dtype=np.float64)
+
+ assert np.array_equal(connection._resolve_arm_target(commanded), commanded)
+
+ commanded[:] = -1.0
+ assert np.array_equal(connection._resolve_arm_target(None), np.arange(7))
+ finally:
+ connection.stop()
diff --git a/dimos/benchmark/libero_pro/test_evaluation.py b/dimos/benchmark/libero_pro/test_evaluation.py
new file mode 100644
index 0000000000..c5b7a6a55f
--- /dev/null
+++ b/dimos/benchmark/libero_pro/test_evaluation.py
@@ -0,0 +1,148 @@
+"""Orchestration tests for one fresh LIBERO-PRO trial."""
+
+from pathlib import Path
+from types import SimpleNamespace
+
+from pytest_mock import MockerFixture
+
+from dimos.benchmark.evaluation.protocol import PolicyArtifact, PolicyExecution
+from dimos.benchmark.libero_pro.assets import PreparedAssets
+import dimos.benchmark.libero_pro.evaluation as evaluation
+from dimos.benchmark.libero_pro.models import LiberoTaskManifest
+from dimos.benchmark.libero_pro.podman import ContainerEndpoints
+
+CASE = Path(__file__).parent / "cases" / "goal-task-0-single-trial" / "task.json"
+
+
+def test_evaluation_protocol_requires_rich_visual_inspection_of_debug_trials() -> None:
+ protocol = evaluation.EVALUATION_PROTOCOL
+
+ assert "trial.open_memory()" in protocol
+ assert "from IPython.display import display" in protocol
+ assert 'memory.stream("agentview_color_image").last().data' in protocol
+ assert "display(PILImage.fromarray(frame.to_rgb().data))" in protocol
+ assert "ASCII art" in protocol
+ assert "pixel statistics" in protocol
+
+
+def test_trial_starts_clock_and_prepared_policy_only_after_blueprint_is_ready(
+ mocker: MockerFixture,
+ tmp_path: Path,
+) -> None:
+ events: list[str] = []
+ manifest = LiberoTaskManifest.model_validate_json(CASE.read_bytes())
+ assets = PreparedAssets(tmp_path / "task.bddl", tmp_path / "states.pt")
+ policy = PolicyArtifact(tmp_path / "policy.py", tmp_path / "policy.pkl", "digest")
+
+ class FakeContainer:
+ def __init__(self, *_args: object, **_kwargs: object) -> None:
+ pass
+
+ def start(self) -> ContainerEndpoints:
+ events.append("container.start")
+ return ContainerEndpoints("policy", "control", "token")
+
+ def stop(self) -> None:
+ events.append("container.stop")
+
+ class FakeControl:
+ def __init__(self, endpoint: str, token: str) -> None:
+ assert (endpoint, token) == ("control", "token")
+
+ def wait_ready(self) -> None:
+ events.append("control.ready")
+
+ def initialize(self, _manifest: LiberoTaskManifest, index: int) -> None:
+ assert index == 0
+ events.append("control.initialize")
+
+ def start(self) -> None:
+ events.append("control.start")
+
+ def wait_terminal(self, _timeout: float) -> SimpleNamespace:
+ events.append("control.wait")
+ return SimpleNamespace(
+ success=True,
+ score=1.0,
+ reward=1.0,
+ terminal_reason="success",
+ policy_ticks=210,
+ backend_ticks=215,
+ error="",
+ )
+
+ def cancel(self) -> None:
+ events.append("control.cancel")
+
+ def close(self) -> None:
+ events.append("control.close")
+
+ class FakeExecution:
+ def start(self) -> None:
+ events.append("policy.start")
+
+ def finish(self, *, grace_s: float = 1.0) -> PolicyExecution:
+ del grace_s
+ events.append("policy.finish")
+ return PolicyExecution("policy_error", 10.5, "RPC closed after native success")
+
+ class FakeRuntime:
+ def prepare(
+ self,
+ _policy: PolicyArtifact,
+ *,
+ memory_path: Path,
+ startup_timeout_s: float,
+ ) -> FakeExecution:
+ assert memory_path.name == "recording.db"
+ assert startup_timeout_s == 30.0
+ events.append("policy.prepare")
+ return FakeExecution()
+
+ coordinator = mocker.Mock()
+ coordinator.start_rpc_service.side_effect = lambda: events.append("blueprint.ready")
+
+ def stop_blueprint() -> None:
+ events.append("blueprint.stop")
+ (tmp_path / "trial" / "trial.mp4").write_bytes(b"video")
+
+ coordinator.stop.side_effect = stop_blueprint
+ mocker.patch.object(evaluation, "LiberoPodmanContainer", FakeContainer)
+ mocker.patch.object(evaluation, "EvaluationControlClient", FakeControl)
+ mocker.patch.object(
+ evaluation, "libero_trial_blueprint", return_value=mocker.sentinel.blueprint
+ )
+ mocker.patch.object(
+ evaluation.ModuleCoordinator,
+ "build",
+ side_effect=lambda _blueprint: (events.append("blueprint.build"), coordinator)[1],
+ )
+
+ trial, native = evaluation._run_trial(
+ manifest,
+ assets,
+ policy,
+ init_index=0,
+ path=tmp_path / "trial",
+ runtime=FakeRuntime(), # type: ignore[arg-type]
+ run_id="scored",
+ )
+
+ assert trial.outcome.status == "completed"
+ assert native["score"] == 1.0
+ assert native["policy_execution_status"] == "completed"
+ assert events == [
+ "container.start",
+ "control.ready",
+ "control.initialize",
+ "blueprint.build",
+ "blueprint.ready",
+ "policy.prepare",
+ "control.start",
+ "policy.start",
+ "control.wait",
+ "policy.finish",
+ "blueprint.stop",
+ "control.close",
+ "container.stop",
+ ]
diff --git a/dimos/benchmark/libero_pro/test_models.py b/dimos/benchmark/libero_pro/test_models.py
new file mode 100644
index 0000000000..6130a93def
--- /dev/null
+++ b/dimos/benchmark/libero_pro/test_models.py
@@ -0,0 +1,24 @@
+from pathlib import Path
+
+import pytest
+
+from dimos.benchmark.libero_pro.models import LiberoTaskManifest
+
+CASE = Path(__file__).parent / "cases" / "goal-task-0-single-trial" / "task.json"
+
+
+def test_smoke_manifest_selects_same_task_with_fresh_rows() -> None:
+ manifest = LiberoTaskManifest.model_validate_json(CASE.read_bytes())
+
+ assert manifest.task.instruction == "open the bottom drawer of the cabinet"
+ assert manifest.episodes.debug_init_state_indices == (1, 2, 3, 4, 5)
+ assert manifest.episodes.scored_init_state_index == 0
+ assert manifest.contract.horizon_ticks == 300
+
+
+def test_manifest_rejects_scored_row_used_for_debugging() -> None:
+ payload = LiberoTaskManifest.model_validate_json(CASE.read_bytes()).model_dump()
+ payload["episodes"]["scored_init_state_index"] = 1
+
+ with pytest.raises(ValueError, match="must not be used for debugging"):
+ LiberoTaskManifest.model_validate(payload)
diff --git a/dimos/benchmark/libero_pro/test_podman.py b/dimos/benchmark/libero_pro/test_podman.py
new file mode 100644
index 0000000000..e7b881033d
--- /dev/null
+++ b/dimos/benchmark/libero_pro/test_podman.py
@@ -0,0 +1,61 @@
+"""Hermetic tests for the rootless Podman lifecycle."""
+
+from pathlib import Path
+from subprocess import CompletedProcess
+
+from pytest_mock import MockerFixture
+
+from dimos.benchmark.libero_pro.assets import PreparedAssets
+from dimos.benchmark.libero_pro.models import LiberoTaskManifest
+from dimos.benchmark.libero_pro.podman import IMAGE, LiberoPodmanContainer
+
+CASE = Path(__file__).parent / "cases" / "goal-task-0-single-trial" / "task.json"
+
+
+def test_container_uses_podman_with_isolated_ports_and_read_only_assets(
+ mocker: MockerFixture,
+ tmp_path: Path,
+) -> None:
+ manifest = LiberoTaskManifest.model_validate_json(CASE.read_bytes())
+ assets = PreparedAssets(bddl=tmp_path / "task.bddl", init_states=tmp_path / "states.pt")
+ assets.bddl.touch()
+ assets.init_states.touch()
+ calls: list[list[str]] = []
+
+ def run(command: list[str], **_kwargs: object) -> CompletedProcess[str]:
+ calls.append(command)
+ if command[1] == "port":
+ port = "41001" if command[-1] == "50051/tcp" else "41002"
+ return CompletedProcess(command, 0, f"127.0.0.1:{port}\n", "")
+ return CompletedProcess(command, 0, "container-id\n", "")
+
+ mocker.patch("subprocess.run", side_effect=run)
+ container = LiberoPodmanContainer(manifest, assets, artifact_dir=tmp_path / "artifacts")
+
+ endpoints = container.start()
+ container.stop()
+
+ run_command = calls[0]
+ assert run_command[:3] == ["podman", "run", "--detach"]
+ assert "--rm" in run_command
+ assert "127.0.0.1::50051" in run_command
+ assert "127.0.0.1::50052" in run_command
+ assert f"{assets.bddl}:/task/task.bddl:ro,Z" in run_command
+ assert f"{assets.init_states}:/task/init_states.pruned_init:ro,Z" in run_command
+ assert run_command[-1] == IMAGE
+ assert endpoints.policy == "127.0.0.1:41001"
+ assert endpoints.control == "127.0.0.1:41002"
+ assert endpoints.control_token
+ assert calls[-2][:3] == ["podman", "stop", "--time"]
+ assert calls[-1][:3] == ["podman", "rm", "--force"]
+
+
+def test_existing_image_skips_build(mocker: MockerFixture) -> None:
+ run = mocker.patch(
+ "subprocess.run",
+ return_value=CompletedProcess(["podman"], 0, "", ""),
+ )
+
+ LiberoPodmanContainer.ensure_image()
+
+ run.assert_called_once_with(["podman", "image", "exists", IMAGE], check=False)
diff --git a/dimos/benchmark/libero_pro/test_server_runtime.py b/dimos/benchmark/libero_pro/test_server_runtime.py
new file mode 100644
index 0000000000..00b2649efe
--- /dev/null
+++ b/dimos/benchmark/libero_pro/test_server_runtime.py
@@ -0,0 +1,149 @@
+"""Threading regression tests for the container-owned LIBERO runtime."""
+
+from importlib.util import module_from_spec, spec_from_file_location
+from pathlib import Path
+import threading
+
+import numpy as np
+from pytest_mock import MockerFixture
+
+SERVER_PATH = Path(__file__).parents[3] / "docker" / "libero-pro" / "server.py"
+SERVER_SPEC = spec_from_file_location("libero_pro_container_server", SERVER_PATH)
+assert SERVER_SPEC is not None
+assert SERVER_SPEC.loader is not None
+SERVER = module_from_spec(SERVER_SPEC)
+SERVER_SPEC.loader.exec_module(SERVER)
+
+
+def test_environment_is_initialized_and_stepped_on_the_same_thread(
+ mocker: MockerFixture,
+) -> None:
+ simulator_thread_ids: list[int] = []
+ observation = {
+ "robot0_joint_pos": np.zeros(7),
+ "robot0_joint_vel": np.zeros(7),
+ "robot0_gripper_qpos": np.zeros(2),
+ "agentview_image": np.zeros((128, 128, 3), dtype=np.uint8),
+ "agentview_depth": np.ones((128, 128, 1), dtype=np.float32),
+ "robot0_eye_in_hand_image": np.zeros((128, 128, 3), dtype=np.uint8),
+ "robot0_eye_in_hand_depth": np.ones((128, 128, 1), dtype=np.float32),
+ }
+
+ class Environment:
+ def step(self, _action: np.ndarray) -> tuple[dict[str, np.ndarray], float, bool, dict]:
+ simulator_thread_ids.append(threading.get_ident())
+ return observation, 1.0, True, {"success": True}
+
+ def check_success(self) -> bool:
+ return True
+
+ def close(self) -> None:
+ simulator_thread_ids.append(threading.get_ident())
+
+ def initialize(runtime, _request) -> tuple[str, str]:
+ simulator_thread_ids.append(threading.get_ident())
+ runtime.environment = Environment()
+ runtime.environment.sim = mocker.sentinel.sim
+ runtime.get_real_depth_map = lambda _sim, depth: depth
+ runtime.get_camera_to_robot_base = lambda _sim, _camera: np.eye(4)
+ runtime.camera_intrinsics = {
+ "agentview": np.eye(3),
+ "robot0_eye_in_hand": np.eye(3),
+ }
+ runtime.frequency = 1_000
+ runtime.horizon = 1
+ runtime.target = np.zeros(8)
+ runtime._action = lambda: np.zeros(7)
+ runtime._publish(observation, 0)
+ return "task", "instruction"
+
+ mocker.patch.object(
+ SERVER.Runtime, "_initialize_environment", autospec=True, side_effect=initialize
+ )
+ runtime = SERVER.Runtime()
+ try:
+ ready = runtime.initialize(object())
+ runtime.start()
+ result = runtime.wait_result()
+ finally:
+ runtime.stop()
+
+ assert ready == ("task", "instruction")
+ assert result.success
+ assert len(simulator_thread_ids) == 3
+ assert len(set(simulator_thread_ids)) == 1
+ assert simulator_thread_ids[0] != threading.get_ident()
+
+
+def test_camera_pose_is_expressed_in_the_robot_base_frame() -> None:
+ class Model:
+ @staticmethod
+ def body_name2id(name: str) -> int:
+ assert name == "robot0_base"
+ return 0
+
+ class Data:
+ xpos = np.array([[0.4, -0.2, 0.5]])
+ xmat = np.array([np.eye(3).reshape(-1)])
+
+ class Simulator:
+ model = Model()
+ data = Data()
+
+ camera_to_world = np.eye(4)
+ camera_to_world[:3, 3] = [0.6, 0.1, 1.2]
+
+ camera_to_robot_base = SERVER._camera_to_robot_base(Simulator(), camera_to_world)
+
+ assert np.allclose(camera_to_robot_base[:3, 3], [0.2, 0.3, 0.7])
+
+
+def test_camera_rendering_is_cached_outside_the_control_step() -> None:
+ calls: list[str] = []
+
+ class Simulator:
+ @staticmethod
+ def render(*, width: int, height: int, camera_name: str, depth: bool):
+ assert (width, height, depth) == (128, 128, True)
+ calls.append(camera_name)
+ return (
+ np.full((128, 128, 3), len(calls), dtype=np.uint8),
+ np.full((128, 128), len(calls), dtype=np.float32),
+ )
+
+ runtime = object.__new__(SERVER.Runtime)
+ runtime.environment = type("Environment", (), {"sim": Simulator()})()
+
+ runtime._render_cameras()
+
+ assert calls == ["agentview", "robot0_eye_in_hand"]
+ assert runtime.camera_observation["agentview_image"][0, 0, 0] == 1
+ assert runtime.camera_observation["robot0_eye_in_hand_depth"][0, 0] == 2
+ assert SERVER.CAMERA_RENDER_INTERVAL_TICKS == 4
+
+
+def test_joint_target_is_adapted_to_native_osc_pose_action() -> None:
+ runtime = object.__new__(SERVER.Runtime)
+ runtime.snapshot = type(
+ "Snapshot",
+ (),
+ {"gripper_position": 0.04},
+ )()
+ runtime.target = np.array([0.1] * 7 + [0.04])
+ controller = type(
+ "Controller",
+ (),
+ {"ee_pos": np.zeros(3), "ee_ori_mat": np.eye(3)},
+ )()
+ robot = type("Robot", (), {"controller": controller})()
+ runtime.environment = type(
+ "Environment",
+ (),
+ {"env": type("Inner", (), {"robots": [robot]})()},
+ )()
+ runtime._target_eef_pose = lambda _joints: (np.array([0.025, -0.05, 0.1]), np.eye(3))
+ runtime.orientation_error = lambda _target, _measured: np.array([0.25, -0.5, 1.0])
+
+ action = runtime._action()
+
+ assert np.allclose(action, [0.5, -1.0, 2.0, 0.5, -1.0, 2.0, 0.0])
diff --git a/dimos/benchmark/libero_pro/test_video.py b/dimos/benchmark/libero_pro/test_video.py
new file mode 100644
index 0000000000..d099268b52
--- /dev/null
+++ b/dimos/benchmark/libero_pro/test_video.py
@@ -0,0 +1,83 @@
+"""Hermetic tests for LIBERO trial video artifacts."""
+
+from pathlib import Path
+
+import numpy as np
+import pytest
+from pytest_mock import MockerFixture
+
+from dimos.benchmark.libero_pro import video
+from dimos.benchmark.libero_pro.video import SideBySideVideo
+from dimos.msgs.sensor_msgs.Image import Image, ImageFormat
+
+
+def test_video_pairs_matching_timestamps_and_finalizes_atomically(
+ mocker: MockerFixture,
+ tmp_path: Path,
+) -> None:
+ writer = mocker.Mock()
+ writer.isOpened.return_value = True
+
+ def open_writer(path: Path, fps: float) -> object:
+ assert fps == 20.0
+ path.touch()
+ return writer
+
+ mocker.patch.object(video, "_open_writer", side_effect=open_writer)
+ output = tmp_path / "trial.mp4"
+ recording = SideBySideVideo(output)
+ red = np.zeros((128, 128, 3), dtype=np.uint8)
+ red[:, :, 0] = 255
+ green = np.zeros((128, 128, 3), dtype=np.uint8)
+ green[:, :, 1] = 255
+
+ recording.add(
+ "robot0_eye_in_hand",
+ Image(data=green, format=ImageFormat.RGB, ts=42.0),
+ )
+ writer.write.assert_not_called()
+ recording.add(
+ "agentview",
+ Image(data=red, format=ImageFormat.RGB, ts=42.0),
+ )
+ recording.add(
+ "agentview",
+ Image(data=red, format=ImageFormat.RGB, ts=41.0),
+ )
+ recording.finish()
+
+ frame = writer.write.call_args.args[0]
+ assert recording.frames_written == 1
+ assert frame.shape == (128, 256, 3)
+ assert frame[0, 0].tolist() == [0, 0, 255]
+ assert frame[0, 255].tolist() == [0, 255, 0]
+ assert output.is_file()
+ assert not (tmp_path / "trial.partial.mp4").exists()
+ writer.write.assert_called_once()
+ writer.release.assert_called_once_with()
+
+
+def test_video_without_a_complete_pair_does_not_publish_artifact(
+ mocker: MockerFixture,
+ tmp_path: Path,
+) -> None:
+ writer = mocker.Mock()
+ writer.isOpened.return_value = True
+
+ def open_writer(path: Path, _fps: float) -> object:
+ path.touch()
+ return writer
+
+ mocker.patch.object(video, "_open_writer", side_effect=open_writer)
+ output = tmp_path / "trial.mp4"
+ recording = SideBySideVideo(output)
+ recording.add(
+ "agentview",
+ Image(data=np.zeros((128, 128, 3), dtype=np.uint8), ts=1.0),
+ )
+
+ with pytest.raises(RuntimeError, match="no synchronized video frames"):
+ recording.finish()
+
+ assert not output.exists()
+ assert not (tmp_path / "trial.partial.mp4").exists()
diff --git a/dimos/benchmark/libero_pro/video.py b/dimos/benchmark/libero_pro/video.py
new file mode 100644
index 0000000000..508f652547
--- /dev/null
+++ b/dimos/benchmark/libero_pro/video.py
@@ -0,0 +1,145 @@
+"""Side-by-side MP4 recording for normal LIBERO camera streams."""
+
+from __future__ import annotations
+
+from pathlib import Path
+import threading
+from typing import Any
+
+import numpy as np
+from reactivex.disposable import Disposable
+
+from dimos.core.core import rpc
+from dimos.core.module import Module, ModuleConfig
+from dimos.core.stream import In
+from dimos.msgs.sensor_msgs.Image import Image
+
+VIDEO_FPS = 20.0
+CAMERA_WIDTH = 128
+CAMERA_HEIGHT = 128
+MAX_PENDING_TIMESTAMPS = 8
+
+
+class LiberoVideoRecorderConfig(ModuleConfig):
+ output_path: Path
+ fps: float = VIDEO_FPS
+
+
+class LiberoVideoRecorder(Module):
+ """Record synchronized public camera observations as one diagnostic MP4."""
+
+ config: LiberoVideoRecorderConfig
+ agentview_color_image: In[Image]
+ eye_in_hand_color_image: In[Image]
+
+ def __init__(self, **kwargs: Any) -> None:
+ super().__init__(**kwargs)
+ self._video: SideBySideVideo | None = None
+
+ @rpc
+ def start(self) -> None:
+ super().start()
+ self._video = SideBySideVideo(self.config.output_path, fps=self.config.fps)
+ self.register_disposable(
+ Disposable(
+ self.agentview_color_image.subscribe(lambda image: self._add("agentview", image))
+ )
+ )
+ self.register_disposable(
+ Disposable(
+ self.eye_in_hand_color_image.subscribe(
+ lambda image: self._add("robot0_eye_in_hand", image)
+ )
+ )
+ )
+
+ @rpc
+ def stop(self) -> None:
+ video, self._video = self._video, None
+ try:
+ super().stop()
+ finally:
+ if video is not None:
+ video.finish()
+
+ def _add(self, camera: str, image: Image) -> None:
+ if self._video is not None:
+ self._video.add(camera, image)
+
+
+class SideBySideVideo:
+ """Pair same-timestamp camera frames and atomically publish an MP4."""
+
+ def __init__(self, output_path: Path, *, fps: float = VIDEO_FPS) -> None:
+ self.output_path = output_path
+ self.partial_path = output_path.with_name(f"{output_path.stem}.partial{output_path.suffix}")
+ self.output_path.parent.mkdir(parents=True, exist_ok=True)
+ self.output_path.unlink(missing_ok=True)
+ self.partial_path.unlink(missing_ok=True)
+ self._writer = _open_writer(self.partial_path, fps)
+ if not self._writer.isOpened():
+ self._writer.release()
+ raise RuntimeError(f"Failed to open LIBERO video writer for {self.output_path}")
+ self._lock = threading.Lock()
+ self._pending: dict[float, dict[str, Image]] = {}
+ self._last_timestamp = float("-inf")
+ self._frames_written = 0
+ self._finished = False
+
+ @property
+ def frames_written(self) -> int:
+ return self._frames_written
+
+ def add(self, camera: str, image: Image) -> None:
+ if camera not in {"agentview", "robot0_eye_in_hand"}:
+ raise ValueError(f"Unknown LIBERO camera: {camera}")
+ with self._lock:
+ if self._finished or image.ts <= self._last_timestamp:
+ return
+ pair = self._pending.setdefault(image.ts, {})
+ pair[camera] = image
+ if pair.keys() >= {"agentview", "robot0_eye_in_hand"}:
+ frame = _compose(pair["agentview"], pair["robot0_eye_in_hand"])
+ self._writer.write(frame)
+ self._frames_written += 1
+ self._last_timestamp = image.ts
+ self._pending = {
+ timestamp: value
+ for timestamp, value in self._pending.items()
+ if timestamp > self._last_timestamp
+ }
+ while len(self._pending) > MAX_PENDING_TIMESTAMPS:
+ self._pending.pop(min(self._pending))
+
+ def finish(self) -> None:
+ with self._lock:
+ if self._finished:
+ return
+ self._finished = True
+ self._writer.release()
+ if self._frames_written == 0:
+ self.partial_path.unlink(missing_ok=True)
+ raise RuntimeError("LIBERO trial produced no synchronized video frames")
+ self.partial_path.replace(self.output_path)
+
+
+def _compose(agentview: Image, eye_in_hand: Image) -> np.ndarray[Any, np.dtype[np.uint8]]:
+ frames = []
+ for name, image in (("agentview", agentview), ("robot0_eye_in_hand", eye_in_hand)):
+ if image.data.shape != (CAMERA_HEIGHT, CAMERA_WIDTH, 3):
+ raise ValueError(
+ f"{name} frame must be {CAMERA_WIDTH}x{CAMERA_HEIGHT} RGB, got {image.data.shape}"
+ )
+ frames.append(np.asarray(image.to_opencv(), dtype=np.uint8))
+ return np.ascontiguousarray(np.concatenate(frames, axis=1))
+
+
+def _open_writer(path: Path, fps: float) -> Any:
+ import cv2
+
+ return cv2.VideoWriter(
+ str(path),
+ cv2.VideoWriter.fourcc(*"mp4v"),
+ fps,
+ (CAMERA_WIDTH * 2, CAMERA_HEIGHT),
+ )
diff --git a/dimos/benchmark/vlnce_r2r/container/runtime/vlnce_runtime/video.py b/dimos/benchmark/vlnce_r2r/container/runtime/vlnce_runtime/video.py
index ead158c901..1b16b60c0e 100644
--- a/dimos/benchmark/vlnce_r2r/container/runtime/vlnce_runtime/video.py
+++ b/dimos/benchmark/vlnce_r2r/container/runtime/vlnce_runtime/video.py
@@ -17,7 +17,6 @@
import os
import tempfile
-import cv2
from habitat.utils.visualizations.utils import images_to_video, observations_to_image
import numpy as np
@@ -75,6 +74,8 @@ def close(self):
_write_json_atomic(self.metadata_path, payload)
def _capture(self, repeats=1):
+ import cv2
+
if self.diagnostic is not None:
return
try:
diff --git a/dimos/benchmark/vlnce_r2r/evaluation.py b/dimos/benchmark/vlnce_r2r/evaluation.py
index 7712cfb436..67ad46e2eb 100644
--- a/dimos/benchmark/vlnce_r2r/evaluation.py
+++ b/dimos/benchmark/vlnce_r2r/evaluation.py
@@ -42,7 +42,7 @@
SYSTEM_PROMPT = """You are the live policy for a real-time navigation Evaluation.
-Use python_exec throughout the episode. The persistent session provides `memory` for
+Use python_exec throughout the episode. The persistent session provides `app.memory` for
public observations and `app` for ordinary DimOS RPCs. The relevant Memory2 streams are
`color_image`, `depth_image`, `depth_pointcloud`, `global_costmap`, and `odom`.
@@ -50,15 +50,15 @@
from IPython.display import display
from PIL import Image as PILImage
- display(PILImage.fromarray(memory.stream("color_image").last().data.data))
+ display(PILImage.fromarray(app.memory.stream("color_image").last().data.data))
Displayed images are delivered to you visually. The `global_costmap` stream is the
complete world-frame traversability grid; `depth_pointcloud` is only the latest
camera-local depth geometry. These expressions access the map and robot pose:
- costmap = memory.stream("global_costmap").last().data
+ costmap = app.memory.stream("global_costmap").last().data
cells = costmap.grid
- pose = memory.stream("odom").last().data
+ pose = app.memory.stream("odom").last().data
cell = costmap.world_to_grid((pose.x, pose.y)) # cell.x is column, cell.y is row
world = costmap.grid_to_world((column, row))
diff --git a/dimos/cli/test_eval.py b/dimos/cli/test_eval.py
new file mode 100644
index 0000000000..f2429432b0
--- /dev/null
+++ b/dimos/cli/test_eval.py
@@ -0,0 +1,50 @@
+"""CLI contract tests for executable evaluations."""
+
+import json
+from pathlib import Path
+from types import SimpleNamespace
+
+from pytest_mock import MockerFixture
+from typer.testing import CliRunner
+
+from dimos.cli import eval as eval_cli
+from dimos.cli.dimos import main
+
+CASE = (
+ Path(__file__).parents[1]
+ / "benchmark"
+ / "libero_pro"
+ / "cases"
+ / "goal-task-0-single-trial"
+ / "evaluation.json"
+)
+
+
+def test_libero_pro_smoke_json_treats_zero_score_as_completed(
+ mocker: MockerFixture,
+ tmp_path: Path,
+) -> None:
+ payload = {
+ "status": "completed",
+ "report": {"native_result": {"kind": "inline", "value": {"score": 0.0}}},
+ }
+ result = SimpleNamespace(
+ status="completed",
+ model_dump_json=lambda: json.dumps(payload),
+ )
+ execute = mocker.patch.object(eval_cli, "execute_evaluation", return_value=result)
+ output = tmp_path / "output"
+
+ invocation = CliRunner().invoke(
+ main,
+ ["eval", "run", str(CASE), "--output", str(output), "--json", "--quiet"],
+ )
+
+ assert invocation.exit_code == 0
+ assert json.loads(invocation.stdout) == payload
+ execute.assert_called_once_with(
+ CASE,
+ output=output,
+ api_key_env="OPENAI_API_KEY",
+ progress=None,
+ )
diff --git a/dimos/control/coordinator.py b/dimos/control/coordinator.py
index 66ff9ea295..463a29d573 100644
--- a/dimos/control/coordinator.py
+++ b/dimos/control/coordinator.py
@@ -886,7 +886,7 @@ def set_gripper_position(self, hardware_id: str, position: float) -> bool:
if isinstance(hw, ConnectedTwistBase):
logger.warning(f"Hardware '{hardware_id}' is a twist base, no gripper support")
return False
- return hw.adapter.write_gripper_position(position)
+ return hw.set_gripper_position(position)
@rpc
def get_gripper_position(self, hardware_id: str) -> float | None:
diff --git a/dimos/control/hardware_interface.py b/dimos/control/hardware_interface.py
index 3a7c74f430..831cb10f10 100644
--- a/dimos/control/hardware_interface.py
+++ b/dimos/control/hardware_interface.py
@@ -202,6 +202,17 @@ def write_command(self, commands: dict[str, float], mode: ControlMode) -> bool:
return arm_ok and gripper_ok
+ def set_gripper_position(self, position: float) -> bool:
+ """Set a physical gripper position and retain it across arm-only commands."""
+ if not self._gripper_joints:
+ return False
+ if not self._initialized:
+ self._initialize_last_commanded()
+ normalized = self._physical_to_normalized(position)
+ for joint_name in self._gripper_joints:
+ self._last_commanded[joint_name] = normalized
+ return self._adapter.write_gripper_position(position)
+
def _initialize_last_commanded(self) -> None:
"""Initialize last_commanded with current hardware positions."""
for _ in range(10):
diff --git a/dimos/control/tasks/trajectory_task/trajectory_task.py b/dimos/control/tasks/trajectory_task/trajectory_task.py
index 85630e7930..7237c65423 100644
--- a/dimos/control/tasks/trajectory_task/trajectory_task.py
+++ b/dimos/control/tasks/trajectory_task/trajectory_task.py
@@ -55,6 +55,7 @@ def joint_trajectory_task(
joint_names: Sequence[str],
priority: int = 10,
start_position_tolerance: float = 0.05,
+ goal_position_tolerance: float = 0.05,
) -> TaskConfig:
"""Build the coordinator's single canonical joint-trajectory task."""
# The coordinator imports this module to recognize the canonical JTT.
@@ -65,7 +66,10 @@ def joint_trajectory_task(
type="trajectory",
joint_names=list(joint_names),
priority=priority,
- params={"start_position_tolerance": start_position_tolerance},
+ params={
+ "start_position_tolerance": start_position_tolerance,
+ "goal_position_tolerance": goal_position_tolerance,
+ },
)
@@ -131,6 +135,8 @@ class JointTrajectoryTaskConfig:
priority: Priority for arbitration (higher wins)
start_position_tolerance: Maximum difference between current joint
position and the first trajectory point.
+ goal_position_tolerance: Maximum difference between measured and
+ commanded joint positions before execution completes.
"""
joint_names: Annotated[
@@ -143,6 +149,11 @@ class JointTrajectoryTaskConfig:
ge=0.0,
allow_inf_nan=False,
)
+ goal_position_tolerance: float = Field(
+ default=0.05,
+ ge=0.0,
+ allow_inf_nan=False,
+ )
class JointTrajectoryTask(BaseControlTask):
@@ -226,14 +237,21 @@ def compute(self, state: CoordinatorState) -> JointCommandOutput | None:
t_elapsed = state.t_now - self._start_time
self._last_elapsed = max(0.0, t_elapsed)
- # Check completion - clamp to final position to ensure we reach goal
+ # Clamp to the final command after its nominal duration, but do not
+ # report success until feedback confirms that the hardware reached it.
if t_elapsed >= self._trajectory.duration:
- self._state = TrajectoryState.COMPLETED
- logger.info(f"Trajectory {self._name} completed after {t_elapsed:.3f}s")
- # Return final position to hold at goal
q_ref, _ = self._trajectory.sample(self._trajectory.duration)
final_names = list(self._trajectory.joint_names)
- self._clear_active_trajectory()
+ reached_goal = all(
+ (measured := state.joints.get_position(name)) is not None
+ and math.isfinite(measured)
+ and abs(measured - target) <= self._config.goal_position_tolerance
+ for name, target in zip(final_names, q_ref, strict=True)
+ )
+ if reached_goal:
+ self._state = TrajectoryState.COMPLETED
+ logger.info(f"Trajectory {self._name} completed after {t_elapsed:.3f}s")
+ self._clear_active_trajectory()
return JointCommandOutput(
joint_names=final_names,
positions=list(q_ref),
@@ -459,6 +477,11 @@ class JointTrajectoryTaskParams(BaseConfig):
ge=0.0,
allow_inf_nan=False,
)
+ goal_position_tolerance: float = Field(
+ default=0.05,
+ ge=0.0,
+ allow_inf_nan=False,
+ )
def create_task(cfg: Any, hardware: Any) -> JointTrajectoryTask:
@@ -472,5 +495,6 @@ def create_task(cfg: Any, hardware: Any) -> JointTrajectoryTask:
joint_names=cfg.joint_names,
priority=cfg.priority,
start_position_tolerance=params.start_position_tolerance,
+ goal_position_tolerance=params.goal_position_tolerance,
),
)
diff --git a/dimos/control/test_control.py b/dimos/control/test_control.py
index 37de9469a7..ac07b9f43e 100644
--- a/dimos/control/test_control.py
+++ b/dimos/control/test_control.py
@@ -208,6 +208,26 @@ def test_normalized_gripper_commands_are_mapped_at_hardware_boundary(self, mock_
((0.07,), {}),
]
+ def test_direct_gripper_command_is_retained_by_arm_only_commands(self, mock_adapter):
+ mock_adapter.read_gripper_position.return_value = 0.035
+ component = HardwareComponent(
+ hardware_id="arm",
+ hardware_type=HardwareType.MANIPULATOR,
+ joints=make_joints("arm", 6),
+ gripper_joints=["arm/gripper"],
+ gripper_open_position=0.07,
+ gripper_closed_position=0.0,
+ )
+ hardware = ConnectedHardware(mock_adapter, component)
+
+ assert hardware.set_gripper_position(0.0)
+ hardware.write_command({"arm/joint1": 0.5}, ControlMode.POSITION)
+
+ assert mock_adapter.write_gripper_position.call_args_list == [
+ ((0.0,), {}),
+ ((0.0,), {}),
+ ]
+
def test_joint_names_prefixed(self, connected_hardware):
names = connected_hardware.joint_names
assert names == [
@@ -444,7 +464,10 @@ def test_joint_trajectory_task_factory(self):
assert config.type == "trajectory"
assert config.joint_names == ["arm/joint1", "arm/joint2"]
assert config.priority == 7
- assert config.params == {"start_position_tolerance": 0.02}
+ assert config.params == {
+ "start_position_tolerance": 0.02,
+ "goal_position_tolerance": 0.05,
+ }
def test_removing_trajectory_task_allows_replacement(self, make_coordinator):
coordinator = make_coordinator()
@@ -501,6 +524,17 @@ def test_config_requires_finite_non_negative_start_tolerance(self, tolerance):
start_position_tolerance=tolerance,
)
+ @pytest.mark.parametrize(
+ "tolerance",
+ [-0.01, math.nan, math.inf, -math.inf],
+ )
+ def test_config_requires_finite_non_negative_goal_tolerance(self, tolerance):
+ with pytest.raises(ValueError):
+ JointTrajectoryTaskConfig(
+ joint_names=["arm/joint1"],
+ goal_position_tolerance=tolerance,
+ )
+
def test_initial_state(self, trajectory_task):
assert trajectory_task.name == JOINT_TRAJECTORY_TASK_NAME
assert not trajectory_task.is_active()
@@ -523,13 +557,22 @@ def test_execute_trajectory(self, trajectory_task, simple_trajectory):
def test_status_snapshot_is_non_destructive(self, trajectory_task, simple_trajectory):
trajectory_task.execute(simple_trajectory, trajectory_start_positions(simple_trajectory))
- trajectory_task.compute(CoordinatorState(joints=MagicMock(), t_now=10.0, dt=0.01))
+ reached = JointStateSnapshot(
+ joint_positions=dict(
+ zip(
+ simple_trajectory.joint_names,
+ simple_trajectory.points[-1].positions,
+ strict=True,
+ )
+ )
+ )
+ trajectory_task.compute(CoordinatorState(joints=reached, t_now=10.0, dt=0.01))
active = trajectory_task.get_status(10.25)
assert active.state is TrajectoryState.EXECUTING
assert active.progress == pytest.approx(0.25)
- trajectory_task.compute(CoordinatorState(joints=MagicMock(), t_now=11.5, dt=0.01))
+ trajectory_task.compute(CoordinatorState(joints=reached, t_now=11.5, dt=0.01))
terminal = trajectory_task.get_status(11.5)
assert terminal.state is TrajectoryState.COMPLETED
assert terminal.progress == pytest.approx(1.0)
@@ -663,18 +706,19 @@ def test_compute_emits_active_subset_only_and_clears_on_completion(self, traject
trajectory_task.execute(trajectory, trajectory_start_positions(trajectory)).status
is TrajectoryExecutionStatus.ACCEPTED
)
- trajectory_task.compute(CoordinatorState(joints=MagicMock(), t_now=10.0, dt=0.01))
- output = trajectory_task.compute(CoordinatorState(joints=MagicMock(), t_now=10.5, dt=0.01))
+ reached = JointStateSnapshot(joint_positions={"arm/joint2": 1.0})
+ trajectory_task.compute(CoordinatorState(joints=reached, t_now=10.0, dt=0.01))
+ output = trajectory_task.compute(CoordinatorState(joints=reached, t_now=10.5, dt=0.01))
assert output is not None
assert output.joint_names == ["arm/joint2"]
assert output.positions == [pytest.approx(0.5)]
- final = trajectory_task.compute(CoordinatorState(joints=MagicMock(), t_now=11.5, dt=0.01))
+ final = trajectory_task.compute(CoordinatorState(joints=reached, t_now=11.5, dt=0.01))
assert final is not None
assert final.joint_names == ["arm/joint2"]
assert trajectory_task.get_state() == TrajectoryState.COMPLETED
assert (
- trajectory_task.compute(CoordinatorState(joints=MagicMock(), t_now=12.0, dt=0.01))
+ trajectory_task.compute(CoordinatorState(joints=reached, t_now=12.0, dt=0.01))
is None
)
@@ -753,8 +797,18 @@ def test_trajectory_completes(self, trajectory_task, simple_trajectory, coordina
trajectory_task.compute(state0)
# Compute past trajectory duration
+ reached_joints = JointStateSnapshot(
+ joint_positions={
+ name: position
+ for name, position in zip(
+ simple_trajectory.joint_names,
+ simple_trajectory.points[-1].positions,
+ strict=True,
+ )
+ }
+ )
state = CoordinatorState(
- joints=coordinator_state.joints,
+ joints=reached_joints,
t_now=t_start + 1.5,
dt=0.01,
)
@@ -766,6 +820,27 @@ def test_trajectory_completes(self, trajectory_task, simple_trajectory, coordina
assert not trajectory_task.is_active()
assert trajectory_task.get_state() == TrajectoryState.COMPLETED
+ def test_trajectory_holds_final_command_until_measured_goal_is_reached(
+ self, trajectory_task, simple_trajectory, coordinator_state
+ ):
+ t_start = time.perf_counter()
+ trajectory_task.execute(simple_trajectory, trajectory_start_positions(simple_trajectory))
+ trajectory_task.compute(
+ CoordinatorState(joints=coordinator_state.joints, t_now=t_start, dt=0.01)
+ )
+
+ output = trajectory_task.compute(
+ CoordinatorState(
+ joints=coordinator_state.joints,
+ t_now=t_start + 1.5,
+ dt=0.01,
+ )
+ )
+
+ assert output is not None
+ assert output.positions == simple_trajectory.points[-1].positions
+ assert trajectory_task.get_state() is TrajectoryState.EXECUTING
+
def test_cancel_trajectory(self, trajectory_task, simple_trajectory):
trajectory_task.execute(simple_trajectory, trajectory_start_positions(simple_trajectory))
assert trajectory_task.is_active()
@@ -980,6 +1055,14 @@ def test_tick_loop_calls_compute(self, mock_adapter, wait_until):
class TestIntegration:
def test_full_trajectory_execution(self, mock_adapter, wait_until):
+ measured_positions = [0.0] * 6
+ mock_adapter.read_joint_positions.side_effect = lambda: measured_positions.copy()
+
+ def track_command(positions):
+ measured_positions[:] = positions
+ return True
+
+ mock_adapter.write_joint_positions.side_effect = track_command
component = HardwareComponent(
hardware_id="arm",
hardware_type=HardwareType.MANIPULATOR,
diff --git a/dimos/manipulation/manipulation_module.py b/dimos/manipulation/manipulation_module.py
index 52805b6f8b..5847bf866b 100644
--- a/dimos/manipulation/manipulation_module.py
+++ b/dimos/manipulation/manipulation_module.py
@@ -74,7 +74,7 @@
RoboPlanPlannerConfig,
)
from dimos.manipulation.planning.spec.config import RobotModelConfig
-from dimos.manipulation.planning.spec.enums import IKStatus, ObstacleType
+from dimos.manipulation.planning.spec.enums import IKStatus, ObstacleType, PlanningStatus
from dimos.manipulation.planning.spec.models import (
DEFAULT_OBSTACLE_RGBA,
CartesianTarget,
@@ -774,6 +774,7 @@ def inverse_kinematics_single(
{group_id: target_pose}, seed=seed, check_collision=check_collision
)
+ @rpc
def solve_ik(
self,
pose: Pose,
@@ -1053,6 +1054,86 @@ def generate_cartesian_plan(
return None
return self._store_generated_plan(group_ids, result, planning_epoch, resolved_speed_scale)
+ @rpc
+ def move_to_pose(
+ self,
+ target: PoseStamped,
+ planning_group: PlanningGroupID | None = None,
+ check_collision: bool = False,
+ speed_scale: float | None = None,
+ blocking: bool = True,
+ timeout: float | None = None,
+ ) -> MoveResult:
+ """Move one end effector to an absolute world-frame pose."""
+
+ self._clear_pending_plan()
+ group = self._resolve_pose_group(planning_group)
+ if isinstance(group, CommandResult):
+ plan_result = PlanResult(PlanStatus.AMBIGUOUS_GROUP, group.message)
+ return MoveResult(plan_result, None, (0.0, 0.0, 0.0), check_collision)
+ current = self.get_state().groups[group.id].end_effector_pose
+ if current is None:
+ plan_result = PlanResult(PlanStatus.FAILED, "End-effector pose is unavailable")
+ return MoveResult(plan_result, None, (0.0, 0.0, 0.0), check_collision)
+ delta = (
+ target.position.x - current.position.x,
+ target.position.y - current.position.y,
+ target.position.z - current.position.z,
+ )
+ resolved_speed = self.config.linear_speed_scale if speed_scale is None else speed_scale
+ start_marker = PoseStamped(
+ frame_id="world",
+ position=current.position,
+ orientation=current.orientation,
+ )
+ if check_collision:
+ plan = self.generate_cartesian_plan(
+ {group.id: (start_marker, target)},
+ RoboPlanCartesianPathConfig(),
+ speed_scale=resolved_speed,
+ check_collision=True,
+ )
+ else:
+ planning = self._begin_group_planning(resolved_speed)
+ if planning is None:
+ plan = None
+ else:
+ planning_epoch, resolved_speed = planning
+ resolved = self._resolve_group_plan_start((group.id,), planning_epoch)
+ if resolved is None:
+ plan = None
+ else:
+ _selection, start = resolved
+ ik = self.inverse_kinematics(
+ {group.id: target},
+ seed=start,
+ check_collision=False,
+ )
+ if not ik.is_success() or ik.joint_state is None:
+ detail = f": {ik.message}" if ik.message else ""
+ self._fail_planning_epoch(
+ planning_epoch,
+ f"IK failed: {ik.status.name}{detail}",
+ )
+ plan = None
+ else:
+ direct_path = PlanningResult(
+ status=PlanningStatus.SUCCESS,
+ path=[start, ik.joint_state],
+ )
+ plan = self._store_generated_plan(
+ (group.id,),
+ direct_path,
+ planning_epoch,
+ resolved_speed,
+ )
+ if plan is None:
+ plan_result = PlanResult(PlanStatus.FAILED, self._error_message or "Planning failed")
+ return MoveResult(plan_result, None, delta, check_collision)
+ plan_result = PlanResult(PlanStatus.SUCCEEDED, plan.message, plan)
+ execution = self.execute(blocking=blocking, timeout=timeout)
+ return MoveResult(plan_result, execution, delta, check_collision)
+
@rpc
def move_linear(
self,
diff --git a/dimos/manipulation/manipulation_spec.py b/dimos/manipulation/manipulation_spec.py
index 9a5aaa3624..85759b99fc 100644
--- a/dimos/manipulation/manipulation_spec.py
+++ b/dimos/manipulation/manipulation_spec.py
@@ -247,6 +247,16 @@ def move_linear(
timeout: float | None = None,
) -> MoveResult: ...
+ def move_to_pose(
+ self,
+ target: PoseStamped,
+ planning_group: PlanningGroupID | None = None,
+ check_collision: bool = False,
+ speed_scale: float | None = None,
+ blocking: bool = True,
+ timeout: float | None = None,
+ ) -> MoveResult: ...
+
def set_gripper_position(
self,
position: float,
diff --git a/dimos/manipulation/test_manipulation_primitives.py b/dimos/manipulation/test_manipulation_primitives.py
index 7986c4d7ed..9839f55e8d 100644
--- a/dimos/manipulation/test_manipulation_primitives.py
+++ b/dimos/manipulation/test_manipulation_primitives.py
@@ -37,6 +37,9 @@
from dimos.manipulation.planning.spec.enums import PlanningStatus
from dimos.manipulation.planning.spec.models import GeneratedPlan
from dimos.msgs.geometry_msgs.Transform import Transform
+from dimos.msgs.geometry_msgs.Pose import Pose
+from dimos.msgs.geometry_msgs.PoseStamped import PoseStamped
+from dimos.msgs.geometry_msgs.Vector3 import Vector3
from dimos.msgs.sensor_msgs.JointState import JointState
from dimos.msgs.trajectory_msgs.JointTrajectory import JointTrajectory
from dimos.msgs.trajectory_msgs.TrajectoryPoint import TrajectoryPoint
@@ -117,6 +120,36 @@ def test_move_linear_uses_world_relative_target_and_default_speed(
execute.assert_called_once_with(blocking=False, timeout=None)
+def test_move_to_pose_uses_current_tcp_as_absolute_path_start(
+ module_factory,
+ mocker: MockerFixture,
+) -> None:
+ module = module_factory()
+ _set_groups(module, _robot())
+ current = Pose(position=Vector3(0.1, 0.2, 0.3))
+ target = PoseStamped(frame_id="world", position=Vector3(0.4, 0.5, 0.6))
+ module._world_monitor.current_group_joint_state.return_value = JointState(
+ name=["arm/j0"], position=[0.1]
+ )
+ module._world_monitor.get_group_ee_pose.return_value = current
+ generate = mocker.patch.object(module, "generate_cartesian_plan", return_value=_plan())
+ mocker.patch.object(
+ module,
+ "execute",
+ return_value=ExecutionResult(ExecutionStatus.ACCEPTED),
+ )
+
+ result = module.move_to_pose(target, check_collision=True)
+
+ assert result.succeeded
+ targets, _config = generate.call_args.args
+ start, goal = targets["arm/tool"]
+ assert start.position == current.position
+ assert start.orientation == current.orientation
+ assert goal == target
+ assert generate.call_args.kwargs["check_collision"] is True
+
+
def test_get_state_returns_every_group_with_presets(module_factory) -> None:
module = module_factory()
robot = _robot(gripper=True, home=[0.3])
diff --git a/dimos/memory2/module.py b/dimos/memory2/module.py
index 146cc4cd27..c66a446f4f 100644
--- a/dimos/memory2/module.py
+++ b/dimos/memory2/module.py
@@ -443,6 +443,8 @@ async def _resolve_pose(self, name: str, msg: Any, ts: float) -> Pose | None:
"""Pose to anchor *msg* with. Dispatches to the stream's (async)
``@pose_setter_for`` if one is defined, else falls back to a
``world <- frame_id`` tf lookup."""
+ if name in self.config.poseless_streams:
+ return None
setter = self._pose_setters.get(name)
if setter is not None:
return cast("Pose | None", await setter(msg))
diff --git a/dimos/memory2/test_module.py b/dimos/memory2/test_module.py
index eb3e2d6c54..1bab6c9b60 100644
--- a/dimos/memory2/test_module.py
+++ b/dimos/memory2/test_module.py
@@ -16,13 +16,15 @@
from __future__ import annotations
+import asyncio
from collections.abc import Iterator
+from typing import Any, cast
import pytest
from dimos.core.module import ModuleConfig
from dimos.core.stream import In, Out
-from dimos.memory2.module import StreamModule
+from dimos.memory2.module import Recorder, StreamModule
from dimos.memory2.stream import Stream
from dimos.memory2.transform import Transformer
from dimos.memory2.type.observation import Observation
@@ -93,3 +95,19 @@ def test_blueprint_ports(module_cls: type[StreamModule]) -> None:
stream_names = {s.name for s in atom.streams}
assert "numbers" in stream_names
assert "doubled" in stream_names
+
+
+def test_poseless_recorder_stream_skips_tf_lookup() -> None:
+ class UnexpectedTF:
+ def get(self, *_args: object, **_kwargs: object) -> None:
+ raise AssertionError("poseless streams must not query TF")
+
+ def dispose(self) -> None:
+ pass
+
+ recorder = Recorder(poseless_streams=["measurements"])
+ try:
+ recorder._tf = cast("Any", UnexpectedTF())
+ assert asyncio.run(recorder._resolve_pose("measurements", object(), 1.0)) is None
+ finally:
+ recorder.stop()
diff --git a/dimos/perception/grounded_segmentation.py b/dimos/perception/grounded_segmentation.py
new file mode 100644
index 0000000000..bf93d3a8ba
--- /dev/null
+++ b/dimos/perception/grounded_segmentation.py
@@ -0,0 +1,99 @@
+# Copyright 2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Explicit-image language grounding followed by promptable segmentation."""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from typing import Protocol
+
+from dimos.core.core import rpc
+from dimos.core.module import Module, ModuleConfig
+from dimos.models.segmentation.edge_tam import EdgeTAMImageSegmenter
+from dimos.models.vl.base import VlModel
+from dimos.models.vl.moondream import MoondreamVlModel
+from dimos.msgs.sensor_msgs.Image import Image
+from dimos.perception.detection.type.detection2d.bbox import Detection2DBBox
+from dimos.perception.detection.type.detection2d.imageDetections2D import ImageDetections2D
+from dimos.perception.detection.type.detection2d.seg import Detection2DSeg
+from dimos.spec.utils import Spec
+
+
+class GroundedSegmentationSpec(Spec, Protocol):
+ """General on-demand text-to-mask perception interface."""
+
+ def segment(
+ self,
+ image: Image,
+ prompts: list[str],
+ ) -> ImageDetections2D[Detection2DSeg]: ...
+
+
+class GroundedSegmentationConfig(ModuleConfig):
+ grounder: Callable[[], VlModel] = MoondreamVlModel
+ segmenter: Callable[[], EdgeTAMImageSegmenter] = EdgeTAMImageSegmenter
+
+
+class GroundedSegmentationModule(Module, GroundedSegmentationSpec):
+ """Ground text in one supplied image and return typed segmentation masks."""
+
+ dedicated_worker = True
+ config: GroundedSegmentationConfig
+
+ def __init__(self, **kwargs) -> None: # type: ignore[no-untyped-def]
+ super().__init__(**kwargs)
+ self._grounder: VlModel | None = None
+ self._segmenter: EdgeTAMImageSegmenter | None = None
+
+ @rpc
+ def start(self) -> None:
+ super().start()
+ self._grounder = self.config.grounder()
+ self._segmenter = self.config.segmenter()
+ self._grounder.start()
+
+ @rpc
+ def stop(self) -> None:
+ if self._grounder is not None:
+ self._grounder.stop()
+ self._grounder = None
+ self._segmenter = None
+ super().stop()
+
+ @rpc
+ def segment(
+ self,
+ image: Image,
+ prompts: list[str],
+ ) -> ImageDetections2D[Detection2DSeg]:
+ """Return masks for objects matching text prompts in the supplied image."""
+ if self._grounder is None or self._segmenter is None:
+ raise RuntimeError("Grounded segmentation module has not been started")
+ normalized = tuple(dict.fromkeys(prompt.strip() for prompt in prompts if prompt.strip()))
+ if not normalized:
+ raise ValueError("prompts must contain at least one non-empty description")
+ boxes: list[Detection2DBBox] = []
+ for prompt in normalized:
+ grounded = self._grounder.query_detections(image, prompt)
+ for detection in grounded:
+ detection.track_id = len(boxes)
+ boxes.append(detection)
+ if not boxes:
+ return ImageDetections2D(image)
+ segmented = self._segmenter.segment(ImageDetections2D(image, boxes))
+ return ImageDetections2D(
+ image,
+ [detection for detection in segmented if isinstance(detection, Detection2DSeg)],
+ )
diff --git a/dimos/perception/rgbd.py b/dimos/perception/rgbd.py
new file mode 100644
index 0000000000..abb52f6f06
--- /dev/null
+++ b/dimos/perception/rgbd.py
@@ -0,0 +1,170 @@
+# Copyright 2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Synchronized RGB-D observations over ordinary Memory2 streams."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+import math
+from typing import TYPE_CHECKING, TypeVar
+
+import numpy as np
+
+from dimos.memory2.tf import StreamTF
+from dimos.msgs.sensor_msgs.CameraInfo import CameraInfo
+from dimos.msgs.sensor_msgs.Image import Image, ImageFormat
+from dimos.perception.detection.type.detection3d.imageDetections3DPC import (
+ ImageDetections3DPC,
+)
+from dimos.perception.detection.type.detection2d.imageDetections2D import ImageDetections2D
+from dimos.perception.detection.type.detection2d.seg import Detection2DSeg
+
+if TYPE_CHECKING:
+ from dimos.memory2.store.base import Store
+ from dimos.memory2.stream import Stream
+ from dimos.msgs.geometry_msgs.Transform import Transform
+ from dimos.perception.detection.type.detection3d.pointcloud_filters import (
+ PointCloudFilter,
+ )
+
+T = TypeVar("T")
+
+
+@dataclass(frozen=True)
+class RGBDObservation:
+ """One timestamp-aligned calibrated RGB-D camera observation."""
+
+ color: Image
+ depth: Image
+ camera_info: CameraInfo
+ world_to_optical: Transform
+
+ @property
+ def timestamp(self) -> float:
+ return self.color.ts
+
+
+def latest_rgbd(
+ store: Store,
+ *,
+ color_stream: str,
+ depth_stream: str,
+ camera_info_stream: str,
+ optical_frame: str,
+ world_frame: str = "world",
+ tolerance_s: float = 0.05,
+) -> RGBDObservation:
+ """Read the latest color frame and its nearest depth, calibration, and TF."""
+ if tolerance_s <= 0:
+ raise ValueError("tolerance_s must be positive")
+ color_obs = store.stream(color_stream, Image).last()
+ depth_obs = _nearest(store.stream(depth_stream, Image), color_obs.ts, tolerance_s)
+ info_obs = _nearest(store.stream(camera_info_stream, CameraInfo), color_obs.ts, tolerance_s)
+ color = color_obs.data
+ depth = depth_obs.data
+ info = info_obs.data
+ if color.frame_id != optical_frame:
+ raise ValueError(
+ f"color frame {color.frame_id!r} does not match optical frame {optical_frame!r}"
+ )
+ if depth.frame_id != optical_frame or info.frame_id != optical_frame:
+ raise ValueError("RGB-D calibration frame IDs are not aligned")
+ if color.width != depth.width or color.height != depth.height:
+ raise ValueError("color and depth image dimensions do not match")
+ if info.width != color.width or info.height != color.height:
+ raise ValueError("camera calibration dimensions do not match the images")
+ if depth.format is not ImageFormat.DEPTH:
+ raise ValueError("depth image must contain metric floating-point depth")
+ tf = StreamTF.from_store(store)
+ if tf is None:
+ raise LookupError("Memory2 store has no recorded TF stream")
+ world_to_optical = tf.get(
+ optical_frame,
+ world_frame,
+ time_point=color_obs.ts,
+ time_tolerance=tolerance_s,
+ )
+ if world_to_optical is None:
+ raise LookupError(f"No {world_frame!r} to {optical_frame!r} transform near {color_obs.ts}")
+ return RGBDObservation(color, depth, info, world_to_optical)
+
+
+def project_depth(
+ detections: ImageDetections2D,
+ observation: RGBDObservation,
+ filters: list[PointCloudFilter] | None = None,
+) -> ImageDetections3DPC:
+ """Lift masks or boxes from one RGB-D observation into world-frame point clouds."""
+ if detections.image.ts != observation.color.ts:
+ raise ValueError("detections and RGB-D observation have different timestamps")
+ if detections.image.frame_id != observation.color.frame_id:
+ raise ValueError("detections and RGB-D observation have different camera frames")
+ return ImageDetections3DPC.from_depth(
+ detections,
+ observation.depth,
+ observation.camera_info,
+ observation.world_to_optical,
+ filters,
+ )
+
+
+def crop_masks(
+ detections: ImageDetections2D[Detection2DSeg],
+ *,
+ x_range: tuple[float, float] = (0.0, 1.0),
+ y_range: tuple[float, float] = (0.0, 1.0),
+) -> ImageDetections2D[Detection2DSeg]:
+ """Keep a normalized image-space region of every segmentation mask."""
+ _validate_normalized_range("x_range", x_range)
+ _validate_normalized_range("y_range", y_range)
+ cropped: list[Detection2DSeg] = []
+ for detection in detections:
+ rows, columns = np.nonzero(detection.mask)
+ if not len(rows):
+ continue
+ x_min, x_max = int(columns.min()), int(columns.max()) + 1
+ y_min, y_max = int(rows.min()), int(rows.max()) + 1
+ left = x_min + math.floor((x_max - x_min) * x_range[0])
+ right = x_min + math.ceil((x_max - x_min) * x_range[1])
+ top = y_min + math.floor((y_max - y_min) * y_range[0])
+ bottom = y_min + math.ceil((y_max - y_min) * y_range[1])
+ mask = np.zeros_like(detection.mask)
+ mask[top:bottom, left:right] = detection.mask[top:bottom, left:right]
+ if not mask.any():
+ continue
+ cropped.append(
+ Detection2DSeg.from_sam2_result(
+ mask,
+ detection.track_id,
+ detections.image,
+ class_id=detection.class_id,
+ name=detection.name,
+ confidence=detection.confidence,
+ )
+ )
+ return ImageDetections2D(detections.image, cropped)
+
+
+def _validate_normalized_range(name: str, value: tuple[float, float]) -> None:
+ low, high = value
+ if not 0.0 <= low < high <= 1.0:
+ raise ValueError(f"{name} must satisfy 0 <= low < high <= 1")
+
+
+def _nearest(stream: Stream[T], timestamp: float, tolerance_s: float): # type: ignore[no-untyped-def]
+ candidates = list(stream.at(timestamp, tolerance_s))
+ if not candidates:
+ raise LookupError(f"No {stream.name!r} observation near {timestamp}")
+ return min(candidates, key=lambda observation: abs(observation.ts - timestamp))
diff --git a/dimos/perception/test_grounded_segmentation.py b/dimos/perception/test_grounded_segmentation.py
new file mode 100644
index 0000000000..c7b456925c
--- /dev/null
+++ b/dimos/perception/test_grounded_segmentation.py
@@ -0,0 +1,100 @@
+# Copyright 2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import numpy as np
+import pytest
+
+from dimos.msgs.sensor_msgs.Image import Image, ImageFormat
+from dimos.perception.detection.type.detection2d.bbox import Detection2DBBox
+from dimos.perception.detection.type.detection2d.imageDetections2D import ImageDetections2D
+from dimos.perception.detection.type.detection2d.seg import Detection2DSeg
+from dimos.perception.grounded_segmentation import GroundedSegmentationModule
+
+
+class FakeGrounder:
+ def __init__(self) -> None:
+ self.queries: list[tuple[Image, str]] = []
+
+ def start(self) -> None:
+ pass
+
+ def stop(self) -> None:
+ pass
+
+ def query_detections(
+ self, image: Image, query: str, **_kwargs: object
+ ) -> ImageDetections2D[Detection2DBBox]:
+ self.queries.append((image, query))
+ return ImageDetections2D(
+ image,
+ [Detection2DBBox((1.0, 1.0, 3.0, 3.0), 99, -1, 1.0, query, image.ts, image)],
+ )
+
+
+class FakeSegmenter:
+ def segment(self, detections: ImageDetections2D) -> ImageDetections2D[Detection2DSeg]:
+ masks = []
+ for detection in detections:
+ mask = np.zeros((4, 4), dtype=np.uint8)
+ mask[1:3, 1:3] = 255
+ masks.append(
+ Detection2DSeg.from_sam2_result(
+ mask,
+ detection.track_id,
+ detections.image,
+ name=detection.name,
+ )
+ )
+ return ImageDetections2D(detections.image, masks)
+
+
+def test_segment_uses_explicit_image_and_preserves_prompt_labels() -> None:
+ image = Image(
+ np.zeros((4, 4, 3), dtype=np.uint8),
+ ImageFormat.RGB,
+ "camera_optical",
+ 42.0,
+ )
+ grounder = FakeGrounder()
+ module = GroundedSegmentationModule(
+ grounder=lambda: grounder,
+ segmenter=FakeSegmenter,
+ )
+
+ module.start()
+ try:
+ result = module.segment(image, ["drawer handle", " drawer handle ", "cabinet"])
+ finally:
+ module.stop()
+
+ assert grounder.queries == [(image, "drawer handle"), (image, "cabinet")]
+ assert [detection.name for detection in result] == ["drawer handle", "cabinet"]
+ assert [detection.track_id for detection in result] == [0, 1]
+ assert all(detection.ts == 42.0 for detection in result)
+ assert all(np.count_nonzero(detection.mask) == 4 for detection in result)
+
+
+def test_segment_rejects_empty_prompts() -> None:
+ image = Image(np.zeros((2, 2, 3), dtype=np.uint8), ImageFormat.RGB, "camera", 1.0)
+ module = GroundedSegmentationModule(
+ grounder=FakeGrounder,
+ segmenter=FakeSegmenter,
+ )
+
+ module.start()
+ try:
+ with pytest.raises(ValueError, match="at least one"):
+ module.segment(image, [" "])
+ finally:
+ module.stop()
diff --git a/dimos/perception/test_rgbd.py b/dimos/perception/test_rgbd.py
new file mode 100644
index 0000000000..b0f89abc4e
--- /dev/null
+++ b/dimos/perception/test_rgbd.py
@@ -0,0 +1,124 @@
+# Copyright 2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from dimos.memory2.store.sqlite import SqliteStore
+from dimos.msgs.geometry_msgs.Transform import Transform
+from dimos.msgs.geometry_msgs.Vector3 import Vector3
+from dimos.msgs.sensor_msgs.CameraInfo import CameraInfo
+from dimos.msgs.sensor_msgs.Image import Image, ImageFormat
+from dimos.msgs.tf2_msgs.TFMessage import TFMessage
+from dimos.perception.detection.type.detection2d.imageDetections2D import ImageDetections2D
+from dimos.perception.detection.type.detection2d.seg import Detection2DSeg
+from dimos.perception.rgbd import crop_masks, latest_rgbd
+
+
+def test_latest_rgbd_returns_nearest_calibrated_observation(tmp_path: Path) -> None:
+ path = tmp_path / "rgbd.db"
+ with SqliteStore(path=str(path)) as store:
+ color = store.stream("front_color", Image, codec="pickle")
+ depth = store.stream("front_depth", Image, codec="pickle")
+ info = store.stream("front_info", CameraInfo)
+ tf = store.stream("tf", TFMessage)
+ color.append(
+ Image(np.zeros((2, 3, 3), dtype=np.uint8), ImageFormat.RGB, "front_optical", 10.0),
+ ts=10.0,
+ )
+ depth.append(
+ Image(np.full((2, 3), 2.0, dtype=np.float32), ImageFormat.DEPTH, "front_optical", 9.98),
+ ts=9.98,
+ )
+ info.append(
+ CameraInfo.from_intrinsics(100.0, 101.0, 1.5, 1.0, 3, 2, "front_optical").with_ts(
+ 10.01
+ ),
+ ts=10.01,
+ )
+ tf.append(
+ TFMessage(
+ Transform(
+ translation=Vector3(1.0, 0.0, 0.0),
+ frame_id="world",
+ child_frame_id="front_optical",
+ ts=10.0,
+ )
+ ),
+ ts=10.0,
+ )
+
+ observation = latest_rgbd(
+ store,
+ color_stream="front_color",
+ depth_stream="front_depth",
+ camera_info_stream="front_info",
+ optical_frame="front_optical",
+ )
+
+ assert observation.timestamp == 10.0
+ assert np.array_equal(observation.depth.data, np.full((2, 3), 2.0, dtype=np.float32))
+ assert observation.camera_info.K[0] == 100.0
+ assert observation.world_to_optical.frame_id == "front_optical"
+ assert observation.world_to_optical.child_frame_id == "world"
+
+
+def test_latest_rgbd_rejects_unaligned_depth(tmp_path: Path) -> None:
+ path = tmp_path / "rgbd.db"
+ with SqliteStore(path=str(path)) as store:
+ store.stream("color", Image, codec="pickle").append(
+ Image(np.zeros((2, 2, 3), dtype=np.uint8), ImageFormat.RGB, "camera", 10.0),
+ ts=10.0,
+ )
+ store.stream("depth", Image, codec="pickle").append(
+ Image(np.ones((2, 2), dtype=np.float32), ImageFormat.DEPTH, "camera", 9.0),
+ ts=9.0,
+ )
+ store.stream("info", CameraInfo).append(
+ CameraInfo.from_intrinsics(1.0, 1.0, 1.0, 1.0, 2, 2, "camera").with_ts(10.0),
+ ts=10.0,
+ )
+
+ with pytest.raises(LookupError, match="No 'depth' observation"):
+ latest_rgbd(
+ store,
+ color_stream="color",
+ depth_stream="depth",
+ camera_info_stream="info",
+ optical_frame="camera",
+ )
+
+
+def test_crop_masks_selects_a_normalized_spatial_subregion() -> None:
+ image = Image(np.zeros((8, 8, 3), dtype=np.uint8), ImageFormat.RGB, "camera", 10.0)
+ mask = np.zeros((8, 8), dtype=np.uint8)
+ mask[1:7, 2:6] = 255
+ detection = Detection2DSeg.from_sam2_result(
+ mask,
+ obj_id=4,
+ image=image,
+ name="stacked handles",
+ )
+
+ result = crop_masks(
+ ImageDetections2D(image, [detection]),
+ x_range=(0.25, 0.75),
+ y_range=(0.5, 1.0),
+ )
+
+ assert len(result) == 1
+ assert result[0].bbox == (3.0, 4.0, 4.0, 6.0)
+ assert np.array_equal(np.argwhere(result[0].mask), np.argwhere(mask[4:7, 3:5]) + [4, 3])
diff --git a/dimos/porcelain/dimos.py b/dimos/porcelain/dimos.py
index 53be50f4b4..7ccb684f4c 100644
--- a/dimos/porcelain/dimos.py
+++ b/dimos/porcelain/dimos.py
@@ -18,7 +18,7 @@
import importlib
import inspect
import threading
-from typing import Any, TypeAlias
+from typing import TYPE_CHECKING, Any, TypeAlias
from dimos.core.coordination.blueprints import Blueprint
from dimos.core.coordination.module_coordinator import ModuleCoordinator, ModuleDescriptor
@@ -34,12 +34,16 @@
from dimos.robot.all_blueprints import all_modules
from dimos.robot.get_all_blueprints import class_name_to_registry_key, get_by_name
+if TYPE_CHECKING:
+ from dimos.memory2.store.base import Store
+
DescribeTarget: TypeAlias = str | ModuleHandle | RpcCall | ModuleInfo | RpcInfo
class Dimos:
- def __init__(self, **config_overrides: Any) -> None:
+ def __init__(self, *, memory: Store | None = None, **config_overrides: Any) -> None:
self._config_overrides = config_overrides
+ self._memory = memory
self._coordinator: ModuleCoordinator | None = None
self._source: ModuleSource | None = None
self._lock = threading.RLock()
@@ -104,7 +108,7 @@ def restart(self, module_class: type[ModuleBase], *, reload_source: bool = True)
self._coordinator.restart_module(module_class, reload_source=reload_source)
@classmethod
- def connect(cls, *, timeout: float = 5.0) -> Dimos:
+ def connect(cls, *, timeout: float = 5.0, memory: Store | None = None) -> Dimos:
"""Connect to the running DimOS coordinator on the current transport bus.
One coordinator serves the configured bus. This works for both
@@ -117,10 +121,21 @@ def connect(cls, *, timeout: float = 5.0) -> Dimos:
`stop()` closes the connection without terminating the remote process.
"""
source = RemoteModuleSource(timeout=timeout)
- instance = cls()
+ instance = cls(memory=memory)
instance._source = source
return instance
+ @property
+ def memory(self) -> Store:
+ """Return the Memory2 store attached to this runtime capability.
+
+ Evaluation policy runtimes attach the active read-only recording so the
+ same query interface works during execution and after a trial.
+ """
+ if self._memory is None:
+ raise RuntimeError("No Memory2 store is attached to this Dimos instance")
+ return self._memory
+
def list_modules(self) -> list[ModuleInfo]:
"""Return structured information for the currently deployed module instances.
@@ -277,6 +292,10 @@ def stop(self) -> None:
self._coordinator.stop()
self._coordinator = None
+ if self._memory is not None:
+ self._memory.stop()
+ self._memory = None
+
@property
def is_running(self) -> bool:
with self._lock:
diff --git a/dimos/porcelain/test_dimos.py b/dimos/porcelain/test_dimos.py
index 9c71ecd602..afa0369ed7 100644
--- a/dimos/porcelain/test_dimos.py
+++ b/dimos/porcelain/test_dimos.py
@@ -95,6 +95,21 @@ def test_construction_with_overrides():
instance.stop()
+def test_attached_memory_is_exposed_and_owned(mocker):
+ memory = mocker.Mock()
+ instance = Dimos(memory=memory)
+
+ assert instance.memory is memory
+ instance.stop()
+
+ memory.stop.assert_called_once_with()
+
+
+def test_memory_without_attachment_raises(app):
+ with pytest.raises(RuntimeError, match="No Memory2 store"):
+ app.memory # noqa: B018
+
+
def test_repr_when_stopped(app):
assert "stopped" in repr(app)
diff --git a/dimos/robot/all_blueprints.py b/dimos/robot/all_blueprints.py
index b1b76be819..77e6924981 100644
--- a/dimos/robot/all_blueprints.py
+++ b/dimos/robot/all_blueprints.py
@@ -221,6 +221,7 @@
"gps-nav-skill-container": "dimos.agents.skills.gps_nav_skill.GpsNavSkillContainer",
"grasp-gen-x-module": "dimos.manipulation.grasping.grasp_gen_x.GraspGenXModule",
"grasping-module": "dimos.manipulation.grasping.grasping.GraspingModule",
+ "grounded-segmentation-module": "dimos.perception.grounded_segmentation.GroundedSegmentationModule",
"gstreamer-camera-module": "dimos.hardware.sensors.camera.gstreamer.gstreamer_camera.GstreamerCameraModule",
"hand-teleop-module": "dimos.teleop.quest.quest_extensions.HandTeleopModule",
"hosted-stats-module": "dimos.teleop.hosted.hosted_stats.HostedStatsModule",
@@ -228,6 +229,9 @@
"joystick-module": "dimos.robot.unitree.b1.joystick_module.JoystickModule",
"keyboard-teleop": "dimos.robot.unitree.keyboard_teleop.KeyboardTeleop",
"keyboard-teleop-module": "dimos.teleop.keyboard.keyboard_teleop_module.KeyboardTeleopModule",
+ "libero-connection": "dimos.benchmark.libero_pro.connection.LiberoConnection",
+ "libero-recorder": "dimos.benchmark.libero_pro.connection.LiberoRecorder",
+ "libero-video-recorder": "dimos.benchmark.libero_pro.video.LiberoVideoRecorder",
"local-planner": "dimos.navigation.cmu_nav.modules.local_planner.local_planner.LocalPlanner",
"manipulation-module": "dimos.manipulation.manipulation_module.ManipulationModule",
"manipulation-skills": "dimos.manipulation.manipulation_skills.ManipulationSkills",
diff --git a/dimos/robot/manipulators/common/blueprints.py b/dimos/robot/manipulators/common/blueprints.py
index 634b14959c..18c7c16095 100644
--- a/dimos/robot/manipulators/common/blueprints.py
+++ b/dimos/robot/manipulators/common/blueprints.py
@@ -63,12 +63,14 @@ def trajectory_task(
*additional_hardware: HardwareComponent,
priority: int = 10,
start_position_tolerance: float = 0.05,
+ goal_position_tolerance: float = 0.05,
) -> TaskConfig:
hardware_components = (hardware, *additional_hardware)
return joint_trajectory_task(
[joint_name for component in hardware_components for joint_name in component.joints],
priority=priority,
start_position_tolerance=start_position_tolerance,
+ goal_position_tolerance=goal_position_tolerance,
)
diff --git a/docker/libero-pro/Dockerfile b/docker/libero-pro/Dockerfile
new file mode 100644
index 0000000000..b1fb62046c
--- /dev/null
+++ b/docker/libero-pro/Dockerfile
@@ -0,0 +1,28 @@
+FROM python:3.10-slim-bookworm
+
+ARG LIBERO_PRO_REVISION=eafdb809426b13153aa1e4c42d6601844217dfec
+
+ENV DEBIAN_FRONTEND=noninteractive \
+ MUJOCO_GL=egl \
+ PYTHONDONTWRITEBYTECODE=1 \
+ PYTHONUNBUFFERED=1
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+ git libegl1 libgl1 libglib2.0-0 libglfw3 libosmesa6 && \
+ rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+COPY docker/libero-pro/requirements.txt /app/requirements.txt
+RUN pip install --no-cache-dir -r /app/requirements.txt
+RUN git clone --depth 1 https://github.com/Zxy-MLlab/LIBERO-PRO.git /opt/libero-pro && \
+ git -C /opt/libero-pro checkout "$LIBERO_PRO_REVISION" && \
+ test "$(git -C /opt/libero-pro rev-parse HEAD)" = "$LIBERO_PRO_REVISION" && \
+ pip install --no-cache-dir -e /opt/libero-pro
+ENV LIBERO_CONFIG_PATH=/opt/libero-config
+COPY docker/libero-pro/config.yaml /opt/libero-config/config.yaml
+
+COPY docker/libero-pro/server.py /app/server.py
+COPY dimos/benchmark/libero_pro/proto /app/dimos/benchmark/libero_pro/proto
+
+EXPOSE 50051 50052
+CMD ["python", "/app/server.py"]
diff --git a/docker/libero-pro/config.yaml b/docker/libero-pro/config.yaml
new file mode 100644
index 0000000000..17b5df0749
--- /dev/null
+++ b/docker/libero-pro/config.yaml
@@ -0,0 +1,5 @@
+assets: /opt/libero-pro/libero/libero/assets
+bddl_files: /opt/libero-pro/libero/libero/bddl_files
+benchmark_root: /opt/libero-pro/libero/libero
+datasets: /opt/libero-pro/libero/datasets
+init_states: /opt/libero-pro/libero/libero/init_files
diff --git a/docker/libero-pro/requirements.txt b/docker/libero-pro/requirements.txt
new file mode 100644
index 0000000000..48d0dcd147
--- /dev/null
+++ b/docker/libero-pro/requirements.txt
@@ -0,0 +1,14 @@
+grpcio==1.81.1
+protobuf==6.33.5
+bddl==1.0.1
+easydict==1.9
+future==0.18.2
+gym==0.25.2
+glfw==2.10.2
+h5py==3.8.0
+matplotlib==3.5.3
+mujoco==2.3.7
+numpy==1.22.4
+opencv-python==4.6.0.66
+robosuite==1.4.0
+termcolor==1.1.0
diff --git a/docker/libero-pro/server.py b/docker/libero-pro/server.py
new file mode 100644
index 0000000000..c7031d84b1
--- /dev/null
+++ b/docker/libero-pro/server.py
@@ -0,0 +1,440 @@
+"""Container-owned LIBERO-PRO simulator and split gRPC interfaces."""
+
+from __future__ import annotations
+
+from concurrent import futures
+import json
+import os
+from pathlib import Path
+import pickle
+import signal
+import threading
+import time
+import zipfile
+
+import grpc # type: ignore[import-untyped]
+import numpy as np
+
+from dimos.benchmark.libero_pro.proto import libero_pro_pb2 as pb2, libero_pro_pb2_grpc as pb2_grpc
+
+GRIPPER_APERTURE_TOLERANCE_M = 1e-4
+CAMERA_RENDER_INTERVAL_TICKS = 4
+OSC_POSITION_DELTA_M = 0.05
+OSC_ORIENTATION_DELTA_RAD = 0.5
+
+
+class Runtime:
+ def __init__(self) -> None:
+ self.condition = threading.Condition()
+ self.environment = None
+ self.snapshot = None
+ self.snapshot_version = 0
+ self.target = None
+ self.result = None
+ self.active = False
+ self.stop_requested = False
+ self.horizon = 0
+ self.frequency = 20
+ self.policy_ticks = 0
+ self.backend_ticks = 0
+ self.task_name = ""
+ self.instruction = ""
+ self.initialize_request = None
+ self.initialize_result = None
+ self.initialize_error = None
+ self.initialization_requested = False
+ self.initialize_done = threading.Event()
+ self.camera_intrinsics = {}
+ self.camera_observation = {}
+ self.get_camera_to_robot_base = None
+ self.get_real_depth_map = None
+ self.thread = threading.Thread(target=self._run, daemon=True)
+ self.thread.start()
+
+ def initialize(self, request) -> tuple[str, str]:
+ with self.condition:
+ if self.initialization_requested:
+ raise RuntimeError("trial initialization has already been requested")
+ self.initialization_requested = True
+ self.initialize_request = request
+ self.condition.notify_all()
+ self.initialize_done.wait()
+ if self.initialize_error is not None:
+ raise self.initialize_error
+ return self.initialize_result
+
+ def _initialize_environment(self, request) -> tuple[str, str]:
+ try:
+ from libero.envs import OffScreenRenderEnv
+ except ImportError:
+ from libero.libero.envs import OffScreenRenderEnv
+ from robosuite.utils.camera_utils import (
+ get_camera_extrinsic_matrix,
+ get_camera_intrinsic_matrix,
+ get_real_depth_map,
+ )
+
+ manifest = json.loads(Path("/task/task.json").read_text())
+ task = manifest["task"]
+ if (request.suite, request.task_order_index, request.task_index) != (
+ task["suite"],
+ task["task_order_index"],
+ task["task_index"],
+ ):
+ raise ValueError("trial selection does not match mounted task manifest")
+ self.task_name = str(task["task_name"])
+ self.instruction = str(task["instruction"])
+ self.environment = OffScreenRenderEnv(
+ bddl_file_name="/task/task.bddl",
+ robots=["Panda"],
+ use_camera_obs=False,
+ has_renderer=False,
+ has_offscreen_renderer=True,
+ camera_heights=128,
+ camera_widths=128,
+ camera_names=["agentview", "robot0_eye_in_hand"],
+ camera_depths=True,
+ controller="OSC_POSE",
+ control_freq=request.control_frequency_hz,
+ horizon=request.horizon_ticks + request.settling_ticks,
+ )
+ from robosuite.utils.control_utils import orientation_error
+
+ self.orientation_error = orientation_error
+ self.get_camera_to_robot_base = lambda sim, camera: _camera_to_robot_base(
+ sim,
+ get_camera_extrinsic_matrix(sim, camera),
+ )
+ self.get_real_depth_map = get_real_depth_map
+ self.camera_intrinsics = {
+ camera: get_camera_intrinsic_matrix(self.environment.sim, camera, 128, 128)
+ for camera in ("agentview", "robot0_eye_in_hand")
+ }
+ with zipfile.ZipFile("/task/init_states.pruned_init") as archive:
+ states = pickle.loads(archive.read("archive/data.pkl"))
+ if request.init_state_index >= len(states):
+ raise IndexError("initial-state index is outside the mounted tensor")
+ self.environment.reset()
+ observation = self.environment.set_init_state(states[request.init_state_index])
+ for _ in range(request.settling_ticks):
+ observation, _, _, _ = self.environment.step(
+ np.zeros(self.environment.env.action_dim, dtype=np.float64)
+ )
+ self.backend_ticks += 1
+ self._render_cameras()
+ self.frequency = request.control_frequency_hz
+ self.horizon = request.horizon_ticks
+ self.policy_ticks = 0
+ self.result = None
+ self.active = False
+ self.target = self._measured_target(observation)
+ self._publish(observation, 0)
+ return self.task_name, self.instruction
+
+ def stop(self) -> None:
+ with self.condition:
+ self.active = False
+ self.stop_requested = True
+ self.condition.notify_all()
+ self.thread.join()
+
+ def start(self) -> None:
+ with self.condition:
+ if self.environment is None or self.result is not None:
+ raise RuntimeError("trial is not ready")
+ self.active = True
+ self.condition.notify_all()
+
+ def cancel(self) -> object:
+ with self.condition:
+ if self.result is None:
+ self.result = self._terminal(False, 0.0, "cancelled")
+ self.active = False
+ self.condition.notify_all()
+ return self.result
+
+ def wait_result(self) -> object:
+ with self.condition:
+ while self.result is None:
+ self.condition.wait()
+ return self.result
+
+ def update_target(self, request) -> None:
+ if len(request.joint_position) != 7:
+ raise ValueError("exactly seven Panda joint targets are required")
+ with self.condition:
+ self.target = np.array([*request.joint_position, request.gripper_position])
+
+ def _run(self) -> None:
+ try:
+ while True:
+ with self.condition:
+ while (
+ self.initialize_request is None
+ and not self.active
+ and not self.stop_requested
+ ):
+ self.condition.wait()
+ if self.stop_requested:
+ return
+ initialize_request = self.initialize_request
+ self.initialize_request = None
+ if initialize_request is not None:
+ try:
+ self.initialize_result = self._initialize_environment(initialize_request)
+ except BaseException as exc:
+ self.initialize_error = exc
+ finally:
+ self.initialize_done.set()
+ continue
+ deadline = time.monotonic()
+ while self.active:
+ deadline += 1.0 / self.frequency
+ time.sleep(max(0.0, deadline - time.monotonic()))
+ if not self.active:
+ break
+ try:
+ observation, reward, done, info = self.environment.step(self._action())
+ self.policy_ticks += 1
+ self.backend_ticks += 1
+ if self.policy_ticks % CAMERA_RENDER_INTERVAL_TICKS == 0:
+ self._render_cameras()
+ self._publish(observation, self.policy_ticks)
+ check_success = getattr(self.environment, "check_success", None)
+ if check_success is None:
+ check_success = self.environment._check_success
+ success = bool(
+ info.get("success") or info.get("is_success") or check_success()
+ )
+ reason = None
+ if success:
+ reason = "success"
+ elif self.policy_ticks >= self.horizon:
+ reason = "horizon"
+ elif done:
+ reason = "backend_done"
+ if reason is not None:
+ with self.condition:
+ self.result = self._terminal(success, float(reward), reason)
+ self.active = False
+ self.condition.notify_all()
+ break
+ except BaseException as exc:
+ with self.condition:
+ self.result = self._terminal(False, 0.0, "failure", str(exc))
+ self.active = False
+ self.condition.notify_all()
+ break
+ finally:
+ if self.environment is not None:
+ self.environment.close()
+
+ def _action(self) -> np.ndarray:
+ observation = self.snapshot
+ target = np.asarray(self.target, dtype=np.float64)
+ target_position, target_orientation = self._target_eef_pose(target[:7])
+ controller = self.environment.env.robots[0].controller
+ position = (target_position - controller.ee_pos) / OSC_POSITION_DELTA_M
+ orientation = (
+ self.orientation_error(
+ target_orientation,
+ controller.ee_ori_mat,
+ )
+ / OSC_ORIENTATION_DELTA_RAD
+ )
+ arm = np.concatenate([position, orientation])
+ gripper_error = target[7] - observation.gripper_position
+ if abs(gripper_error) <= GRIPPER_APERTURE_TOLERANCE_M:
+ gripper = 0.0
+ else:
+ gripper = -1.0 if gripper_error > 0.0 else 1.0
+ return np.concatenate([arm, [gripper]])
+
+ def _target_eef_pose(self, joint_position: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+ """Evaluate a DimOS joint target as an OSC end-effector target."""
+ sim = self.environment.sim
+ robot = self.environment.env.robots[0]
+ state = sim.get_state()
+ try:
+ sim.data.qpos[robot._ref_joint_pos_indexes] = joint_position
+ sim.data.qvel[robot._ref_joint_vel_indexes] = 0.0
+ sim.forward()
+ position = np.asarray(sim.data.site_xpos[robot.eef_site_id]).copy()
+ orientation = np.asarray(sim.data.site_xmat[robot.eef_site_id]).reshape(3, 3).copy()
+ finally:
+ sim.set_state(state)
+ sim.forward()
+ return position, orientation
+
+ def _publish(self, observation, tick: int) -> None:
+ joints = np.asarray(observation["robot0_joint_pos"], dtype=np.float64)[:7]
+ velocities = np.asarray(observation["robot0_joint_vel"], dtype=np.float64)[:7]
+ gripper = float(np.mean(np.abs(observation["robot0_gripper_qpos"])))
+ cameras = []
+ if self.get_real_depth_map is None or self.get_camera_to_robot_base is None:
+ raise RuntimeError("camera calibration helpers are not initialized")
+ camera_observation = self.camera_observation or observation
+ for camera in ("agentview", "robot0_eye_in_hand"):
+ rgb = np.ascontiguousarray(
+ np.flipud(camera_observation[f"{camera}_image"]), dtype=np.uint8
+ )
+ depth = self.get_real_depth_map(
+ self.environment.sim,
+ camera_observation[f"{camera}_depth"],
+ )
+ depth = np.ascontiguousarray(np.flipud(depth).reshape(128, 128), dtype=np.float32)
+ camera_to_robot_base = self.get_camera_to_robot_base(
+ self.environment.sim,
+ camera,
+ )
+ cameras.append(
+ pb2.CameraFrame(
+ camera=camera,
+ width=128,
+ height=128,
+ rgb=rgb.tobytes(),
+ depth_meters=depth.tobytes(),
+ intrinsic=np.asarray(self.camera_intrinsics[camera]).reshape(-1).tolist(),
+ camera_to_robot_base=np.asarray(camera_to_robot_base).reshape(-1).tolist(),
+ )
+ )
+ with self.condition:
+ self.snapshot = pb2.RobotSnapshot(
+ tick=tick,
+ timestamp_s=time.time(),
+ joint_position=joints,
+ joint_velocity=velocities,
+ gripper_position=gripper,
+ cameras=cameras,
+ )
+ self.snapshot_version += 1
+ self.condition.notify_all()
+
+ def _render_cameras(self) -> None:
+ """Refresh RGB-D independently of the 20 Hz physics/control loop."""
+ frames = {}
+ for camera in ("agentview", "robot0_eye_in_hand"):
+ rgb, depth = self.environment.sim.render(
+ width=128,
+ height=128,
+ camera_name=camera,
+ depth=True,
+ )
+ frames[f"{camera}_image"] = rgb
+ frames[f"{camera}_depth"] = depth
+ self.camera_observation = frames
+
+ def _measured_target(self, observation) -> np.ndarray:
+ return np.array(
+ [
+ *np.asarray(observation["robot0_joint_pos"], dtype=np.float64)[:7],
+ float(np.mean(np.abs(observation["robot0_gripper_qpos"]))),
+ ]
+ )
+
+ def _terminal(self, success, reward, reason, error=""):
+ return pb2.TerminalResult(
+ success=success,
+ score=1.0 if success else 0.0,
+ reward=reward,
+ terminal_reason=reason,
+ policy_ticks=self.policy_ticks,
+ backend_ticks=self.backend_ticks,
+ error=error,
+ )
+
+
+def _camera_to_robot_base(sim, camera_to_world: np.ndarray) -> np.ndarray:
+ """Express an OpenCV camera pose in the Panda base frame."""
+ base_id = sim.model.body_name2id("robot0_base")
+ robot_base_to_world = np.eye(4, dtype=np.float64)
+ robot_base_to_world[:3, :3] = np.asarray(sim.data.xmat[base_id]).reshape(3, 3)
+ robot_base_to_world[:3, 3] = np.asarray(sim.data.xpos[base_id])
+ return np.linalg.inv(robot_base_to_world) @ np.asarray(camera_to_world)
+
+
+class PolicyService:
+ def __init__(self, runtime: Runtime) -> None:
+ self.runtime = runtime
+
+ def GetHealth(self, _request, _context):
+ return pb2.Health(ready=True, detail="policy interface ready")
+
+ def WatchState(self, _request, context):
+ version = -1
+ while context.is_active():
+ with self.runtime.condition:
+ while self.runtime.snapshot_version == version and context.is_active():
+ self.runtime.condition.wait(timeout=1)
+ if not context.is_active():
+ return
+ version = self.runtime.snapshot_version
+ snapshot = self.runtime.snapshot
+ if snapshot is not None:
+ yield snapshot
+
+ def SetJointTargets(self, request, _context):
+ self.runtime.update_target(request)
+ return pb2.Ack(sequence=request.sequence)
+
+
+class ControlService:
+ def __init__(self, runtime: Runtime, token: str) -> None:
+ self.runtime = runtime
+ self.token = token
+
+ def _authorize(self, context) -> None:
+ metadata = dict(context.invocation_metadata())
+ if metadata.get("authorization") != f"Bearer {self.token}":
+ context.abort(grpc.StatusCode.PERMISSION_DENIED, "invalid control capability")
+
+ def GetHealth(self, _request, context):
+ self._authorize(context)
+ return pb2.Health(ready=True, detail="evaluation control ready")
+
+ def InitializeTrial(self, request, context):
+ self._authorize(context)
+ name, instruction = self.runtime.initialize(request)
+ return pb2.TrialReady(task_name=name, instruction=instruction)
+
+ def StartTrial(self, _request, context):
+ self._authorize(context)
+ self.runtime.start()
+ return pb2.Empty()
+
+ def WaitForTerminal(self, _request, context):
+ self._authorize(context)
+ return self.runtime.wait_result()
+
+ def CancelTrial(self, _request, context):
+ self._authorize(context)
+ return self.runtime.cancel()
+
+ def GetNativeResult(self, _request, context):
+ self._authorize(context)
+ return self.runtime.wait_result()
+
+
+def main() -> None:
+ runtime = Runtime()
+ policy_server = grpc.server(futures.ThreadPoolExecutor(max_workers=8))
+ control_server = grpc.server(futures.ThreadPoolExecutor(max_workers=4))
+ pb2_grpc.add_PolicyInterfaceServicer_to_server(PolicyService(runtime), policy_server)
+ pb2_grpc.add_EvaluationControlServicer_to_server(
+ ControlService(runtime, os.environ["DIMOS_LIBERO_CONTROL_TOKEN"]), control_server
+ )
+ policy_server.add_insecure_port("0.0.0.0:50051")
+ control_server.add_insecure_port("0.0.0.0:50052")
+ policy_server.start()
+ control_server.start()
+ stopped = threading.Event()
+ signal.signal(signal.SIGTERM, lambda *_: stopped.set())
+ signal.signal(signal.SIGINT, lambda *_: stopped.set())
+ stopped.wait()
+ policy_server.stop(2)
+ control_server.stop(2)
+ runtime.stop()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/docs/adr/0001-preserve-libero-pro-comparability.md b/docs/adr/0001-preserve-libero-pro-comparability.md
new file mode 100644
index 0000000000..4791dd8257
--- /dev/null
+++ b/docs/adr/0001-preserve-libero-pro-comparability.md
@@ -0,0 +1,7 @@
+---
+status: superseded by ADR-0006
+---
+
+# Preserve LIBERO-PRO comparability
+
+The LIBERO-PRO integration will preserve the original benchmark's episode, observation, action, rollout, success, and aggregation semantics as closely as possible so its score can be compared with other reported LIBERO-PRO scores. We reject a separate DimOS-native track with continuous real-time stepping, Panda joint-position control, and a custom horizon because those changes would produce a different benchmark even if it reused LIBERO-PRO tasks and native success predicates.
diff --git a/docs/adr/0002-retain-code-policy-exploration.md b/docs/adr/0002-retain-code-policy-exploration.md
new file mode 100644
index 0000000000..4d41f6fcc5
--- /dev/null
+++ b/docs/adr/0002-retain-code-policy-exploration.md
@@ -0,0 +1,3 @@
+# Retain code-policy exploration
+
+The evaluated system includes PR #3434's task-specific Exploration Stage: Pi may submit policies, observe unscored debug-trial outcomes, and revise the Policy Artifact before measured LIBERO-PRO rollouts. We treat this as the code-as-policy system's training phase, analogous to the training available to learned policies, because evaluating only a pre-existing callable would remove the approach's intended learning mechanism and make code-as-policy itself largely irrelevant. Comparability therefore depends on declaring and enforcing the evidence available during exploration, not on deleting exploration.
diff --git a/docs/adr/0003-seal-libero-pro-evaluation-cases.md b/docs/adr/0003-seal-libero-pro-evaluation-cases.md
new file mode 100644
index 0000000000..2febe62d1c
--- /dev/null
+++ b/docs/adr/0003-seal-libero-pro-evaluation-cases.md
@@ -0,0 +1,7 @@
+---
+status: superseded by ADR-0016
+---
+
+# Seal LIBERO-PRO evaluation cases
+
+The single LIBERO-PRO integration configures code-policy exploration with an unperturbed training suite from the pinned LIBERO-PRO source, while perturbed evaluation suites, evaluation initial states, and native outcome feedback remain sealed until the scored Evaluation Stage. The Policy Artifact is frozen before it encounters that held-out configuration. This preserves task-specific interactive learning while retaining LIBERO-PRO's intended measurement of generalization under perturbation; it does not introduce a separate LIBERO integration.
diff --git a/docs/adr/0004-scope-first-libero-pro-change-to-one-trial.md b/docs/adr/0004-scope-first-libero-pro-change-to-one-trial.md
new file mode 100644
index 0000000000..af20cdf691
--- /dev/null
+++ b/docs/adr/0004-scope-first-libero-pro-change-to-one-trial.md
@@ -0,0 +1,3 @@
+# Scope the first LIBERO-PRO change to one trial
+
+The first LIBERO-PRO integration is complete when one code-policy Exploration Stage runs against a selected task from the pinned LIBERO-PRO source, its frozen Policy Artifact executes one fresh trial of that same suite and task under the declared unified real-time contract, and the native score is generated, retained, and reported through `dimos eval`. The agent receives the exact task language used by the scored trial, while the scored episode uses a fresh initial state and keeps native scoring truth evaluator-only. Full task matrices, repeated rollouts, aggregate reporting, multi-container scheduling, and distributed batching are deferred; consequently, this change demonstrates benchmark integration but does not claim a comparable aggregate LIBERO-PRO result.
diff --git a/docs/adr/0005-isolate-every-policy-submission-in-a-complete-run.md b/docs/adr/0005-isolate-every-policy-submission-in-a-complete-run.md
new file mode 100644
index 0000000000..ae69af07b2
--- /dev/null
+++ b/docs/adr/0005-isolate-every-policy-submission-in-a-complete-run.md
@@ -0,0 +1,3 @@
+# Isolate every policy submission in a complete run
+
+Every `submit_policy()` call builds a brand-new complete DimOS blueprint, spawns a fresh LIBERO episode within that run, executes the submitted policy against the connected blueprint, stops the entire run, and returns its immutable outcome, logs, Memory2 recording, and artifacts. No simulator, blueprint, module state, or policy process is reused between submissions. The scored LIBERO-PRO trial uses the same fresh-run harness without the exploration agent, concentrating lifecycle and cleanup semantics in one path while preventing state leakage between training attempts and evaluation.
diff --git a/docs/adr/0006-run-unified-policies-in-real-time.md b/docs/adr/0006-run-unified-policies-in-real-time.md
new file mode 100644
index 0000000000..e24ae263ec
--- /dev/null
+++ b/docs/adr/0006-run-unified-policies-in-real-time.md
@@ -0,0 +1,3 @@
+# Run unified policies in real time
+
+The LIBERO-PRO integration evaluates the same ordinary DimOS Policy Artifact shape used in simulation and on real robots, so the simulator advances freely while `policy(app)` computes and commands the running blueprint. We accept and explicitly report the resulting temporal deviation from upstream LIBERO, whose synchronous evaluator pauses between native actions, because a policy that depends on paused inference cannot satisfy the unified real/sim product requirement. Tasks, permitted observations, native action semantics, initial states, privileged BDDL success, and score reporting remain benchmark-faithful wherever they do not conflict with continuous execution.
diff --git a/docs/adr/0007-count-libero-horizons-in-simulator-ticks.md b/docs/adr/0007-count-libero-horizons-in-simulator-ticks.md
new file mode 100644
index 0000000000..df4b2f0029
--- /dev/null
+++ b/docs/adr/0007-count-libero-horizons-in-simulator-ticks.md
@@ -0,0 +1,3 @@
+# Count LIBERO horizons in simulator ticks
+
+The unified real-time LIBERO-PRO evaluation interprets each published suite action-step limit as the same number of continuously advancing 20 Hz simulator control ticks. Blueprint and simulator startup plus evaluator-owned settling finish before the counter starts; afterward every tick consumes the horizon even when the policy has not issued a new command. This preserves the native horizon's simulated physical duration, penalizes policy latency as real execution would, and avoids recreating paused inference by counting only policy updates.
diff --git a/docs/adr/0008-isolate-libero-in-a-container.md b/docs/adr/0008-isolate-libero-in-a-container.md
new file mode 100644
index 0000000000..ede233cde0
--- /dev/null
+++ b/docs/adr/0008-isolate-libero-in-a-container.md
@@ -0,0 +1,3 @@
+# Isolate LIBERO in a container
+
+LIBERO, robosuite, MuJoCo, benchmark assets, and their pinned legacy dependencies run in a dedicated container rather than the DimOS Python environment. Each debug or measured trial creates a fresh LIBERO container and a fresh complete DimOS blueprint; an ordinary Python `LiberoConnection` module carries non-privileged robot control and observations between them. This preserves one policy interface across simulation and real hardware, keeps incompatible dependencies out of core DimOS, and gives later batch execution a reproducible container artifact without requiring a generic Docker module worker in this PR.
diff --git a/docs/adr/0009-separate-policy-and-evaluation-container-interfaces.md b/docs/adr/0009-separate-policy-and-evaluation-container-interfaces.md
new file mode 100644
index 0000000000..648a09ce52
--- /dev/null
+++ b/docs/adr/0009-separate-policy-and-evaluation-container-interfaces.md
@@ -0,0 +1,3 @@
+# Separate policy and evaluation container interfaces
+
+The LIBERO container exposes two interfaces with different owners. `LiberoConnection` adapts a non-privileged Policy Interface into the complete DimOS blueprint, making ordinary observations, robot state, commands, and operational health available to `policy(app)`. In parallel, the Evaluation exclusively uses an Evaluation Control Interface for suite and task configuration, reset and initial-state selection, readiness and clock control, native terminal state, privileged diagnostics, and scoring. The Evaluation owns the container lifecycle; no control-interface capability is registered in the blueprint, recorded in Memory2, or returned through debug-trial artifacts.
diff --git a/docs/adr/0010-defer-policy-sandbox-to-parallel-evaluation.md b/docs/adr/0010-defer-policy-sandbox-to-parallel-evaluation.md
new file mode 100644
index 0000000000..97fa053633
--- /dev/null
+++ b/docs/adr/0010-defer-policy-sandbox-to-parallel-evaluation.md
@@ -0,0 +1,3 @@
+# Defer the policy sandbox to parallel evaluation
+
+The single-trial LIBERO-PRO integration keeps the Evaluation Control Interface out of the blueprint, policy environment, Memory2, logs, and trial artifacts and protects it with an unshared per-trial capability, but it does not claim containment against deliberately hostile same-user Python. A policy sandbox with process and network isolation is required before parallel or untrusted evaluation and will provide that stronger guarantee later. This PR preserves an interface seam that can cross the future sandbox without expanding the first vertical slice into a general execution-security project.
diff --git a/docs/adr/0011-keep-libero-pro-in-repo.md b/docs/adr/0011-keep-libero-pro-in-repo.md
new file mode 100644
index 0000000000..d070650813
--- /dev/null
+++ b/docs/adr/0011-keep-libero-pro-in-repo.md
@@ -0,0 +1,3 @@
+# Keep LIBERO-PRO integration in the DimOS repository
+
+The first LIBERO-PRO integration is in-repo DimOS work: its Evaluation, `LiberoConnection`, blueprint composition, benchmark manifests, tests, and container definition live in the main repository and use the existing built-in evaluation resolution path. We will not create a separate Python distribution, installation workflow, or external `dimos.evaluations` entry point for this change. Outside registration remains the registry's concern and is not an abstraction the LIBERO vertical slice needs to introduce.
diff --git a/docs/adr/0012-keep-libero-connection-with-the-evaluation.md b/docs/adr/0012-keep-libero-connection-with-the-evaluation.md
new file mode 100644
index 0000000000..80fb5ede86
--- /dev/null
+++ b/docs/adr/0012-keep-libero-connection-with-the-evaluation.md
@@ -0,0 +1,3 @@
+# Keep LiberoConnection with the evaluation
+
+`LiberoConnection` is an ordinary DimOS `Module`, but its domain role is exclusively to adapt the LIBERO container's Policy Interface into the LIBERO-PRO evaluation blueprint. It therefore lives under the in-repo LIBERO-PRO evaluation rather than the general robot/manipulator connection hierarchy. We do not create or support a speculative standalone use; the module remains composable and testable through normal DimOS interfaces inside the fresh debug and measured blueprints that need it.
diff --git a/docs/adr/0013-use-grpc-protobuf-for-the-container-protocol.md b/docs/adr/0013-use-grpc-protobuf-for-the-container-protocol.md
new file mode 100644
index 0000000000..14ed2dc289
--- /dev/null
+++ b/docs/adr/0013-use-grpc-protobuf-for-the-container-protocol.md
@@ -0,0 +1,3 @@
+# Use gRPC and protobuf for the LIBERO container protocol
+
+The LIBERO container and in-process evaluation modules communicate through a checked-in protobuf contract implemented with gRPC. `LiberoConnection` translates the Policy Interface's streamed observations, robot state, actuator commands, readiness, and health into ordinary DimOS behavior, while the Evaluation uses the separate control service. The repository will declare `grpcio` directly rather than rely on its current transitive lock entry, keep LIBERO dependencies inside the image, and check in generated Python stubs so ordinary runtime and image startup do not require a protobuf compiler.
diff --git a/docs/adr/0014-separate-the-grpc-listeners.md b/docs/adr/0014-separate-the-grpc-listeners.md
new file mode 100644
index 0000000000..6f88d89542
--- /dev/null
+++ b/docs/adr/0014-separate-the-grpc-listeners.md
@@ -0,0 +1,3 @@
+# Separate the gRPC listeners
+
+The LIBERO container binds the Policy Interface and Evaluation Control Interface to separate gRPC endpoints. `LiberoConnection` receives only the policy endpoint, while the Evaluation receives the control endpoint and its per-trial capability. This makes the privilege split structural today and allows the future policy sandbox to reach only the policy listener without redesigning the protobuf services or depending on method-level filtering at one shared address.
diff --git a/docs/adr/0015-mount-verified-task-assets-per-trial.md b/docs/adr/0015-mount-verified-task-assets-per-trial.md
new file mode 100644
index 0000000000..97bd9bb4ea
--- /dev/null
+++ b/docs/adr/0015-mount-verified-task-assets-per-trial.md
@@ -0,0 +1,3 @@
+# Mount verified task assets per trial
+
+The pinned LIBERO-PRO container image contains the Python runtime, LIBERO-PRO source, robosuite, MuJoCo, gRPC server, and locked dependencies, while the Evaluation supplies the selected BDDL, initialization tensor, and task manifest as verified read-only mounts. Debug and measured containers use the same image, protocol, suite, task, and task language. Each trial selects a fresh initial state from that task's benchmark data. This keeps task identity explicit, avoids rebuilding the runtime image for asset changes, and lets future parallel workers reuse one content-addressed image without introducing a separate LIBERO integration.
diff --git a/docs/adr/0016-align-evaluation-semantics-with-cap-x.md b/docs/adr/0016-align-evaluation-semantics-with-cap-x.md
new file mode 100644
index 0000000000..631829d56f
--- /dev/null
+++ b/docs/adr/0016-align-evaluation-semantics-with-cap-x.md
@@ -0,0 +1,5 @@
+# Align evaluation semantics with CaP-X
+
+When an evaluation-design question has an analogous choice in CaP-X, inspect CaP-X's current implementation first and follow its semantics unless they conflict with an explicit DimOS product requirement. Any necessary deviation must be deliberate and documented. CaP-X is the closest reference for code-as-policy evaluation and therefore provides a better default than inventing a new split or interpreting the learned-policy protocol in isolation.
+
+For LIBERO-PRO task disclosure, exploration and scoring use the same suite, task identity, and exact task language, matching CaP-X's practice of injecting the selected LIBERO task language into the code-generation prompt for each trial. Every `submit_policy()` debug trial and the final scored run still receives a fresh environment and initial state. The Policy Artifact is frozen before the scored run, and privileged native outcome state remains evaluator-only. We reject the superseded design that explored an unperturbed task and then scored the artifact against an undisclosed perturbed task, because that would measure hidden-task transfer rather than CaP-X-style task-conditioned code generation.
diff --git a/docs/capabilities/agents/evaluation.md b/docs/capabilities/agents/evaluation.md
index f55b88bf6b..25dec3514c 100644
--- a/docs/capabilities/agents/evaluation.md
+++ b/docs/capabilities/agents/evaluation.md
@@ -77,10 +77,19 @@ artifacts. It does not expose simulator state or scorer internals.
The Evaluation reuses one task-level policy across all held-out cases. For each
case it starts a fresh environment and policy-only blueprint, waits for DimOS to
-be ready, and invokes the serialized callable in a clean process:
+be ready, and loads the serialized callable in a clean process behind a start
+gate:
```python
-execution = context.runtime.execute(policy, timeout_s=case.timeout_s)
+execution = context.runtime.prepare(
+ policy,
+ memory_path=recording_path,
+ startup_timeout_s=30.0,
+)
+evaluator.start_trial()
+execution.start()
+terminal = evaluator.wait_for_terminal()
+policy_result = execution.finish()
```
Pi and the production `McpClient` are absent from measured execution. The task
@@ -117,6 +126,34 @@ Run an installed Evaluation from a strict JSON specification:
dimos eval run specification.json --output evaluation-run
```
+The in-repo LIBERO-PRO smoke case runs each debug submission and the scored
+trial in a fresh rootless Podman container and a fresh DimOS blueprint. Podman
+owns the pinned simulator environment; LIBERO is never imported into the host
+DimOS Python environment.
+
+```bash
+dimos eval run \
+ dimos/benchmark/libero_pro/cases/goal-task-0-single-trial/evaluation.json \
+ --output /tmp/dimos-libero-pro-smoke \
+ --json --quiet
+```
+
+A completed trial exits successfully whether its native score is `0.0` or
+`1.0`. Infrastructure and policy execution failures remain failed runs.
+
+Every debug submission and the scored trial records the two public camera
+streams side by side at 20 FPS. Debug videos live beside their corresponding
+trial diagnostics; the scored video is also listed in the evaluation report:
+
+```text
+runtime/exploration-0001/submission-0001/trial/trial.mp4
+scored-trial/trial.mp4
+```
+
+The left half is `agentview` and the right half is `robot0_eye_in_hand`. The
+video contains the post-settling initial observation followed by one frame for
+each simulator policy tick.
+
The run specification pins the model condition but cannot switch runtime
profiles. The Evaluation owns that choice, and `evaluation-run/run.json`
records both the requested condition and resolved profile.
@@ -125,7 +162,7 @@ records both the requested condition and resolved profile.
The `live-agent-v1` profile prepares one Pi session and one persistent Python
workspace before the evaluator's start gate. The workspace provides `app` for
-ordinary DimOS RPCs and `memory` for read-only public observations. Pi can make
+ordinary DimOS RPCs and `app.memory` for read-only public observations. Pi can make
repeated `python_exec` calls while the native environment advances. The native
terminal condition or evaluator timeout stops Pi and closes the workspace.
diff --git a/docs/research/libero-pro-native-evaluation-call-graph.md b/docs/research/libero-pro-native-evaluation-call-graph.md
new file mode 100644
index 0000000000..fa7f8a1087
--- /dev/null
+++ b/docs/research/libero-pro-native-evaluation-call-graph.md
@@ -0,0 +1,350 @@
+# LIBERO-PRO native evaluation call graph
+
+Date: 2026-08-10
+
+> Design update: the original sealed-task recommendation in this investigation
+> was superseded after checking CaP-X's task-conditioned evaluation flow. See
+> ADR-0016. The upstream LIBERO-PRO call-graph findings remain valid.
+
+## Question
+
+What evaluation behavior is actually defined by LIBERO-PRO at commit
+[`eafdb80`](https://github.com/Zxy-MLlab/LIBERO-PRO/tree/eafdb809426b13153aa1e4c42d6601844217dfec),
+what behavior is inherited from original LIBERO at commit
+[`8f1084e`](https://github.com/Lifelong-Robot-Learning/LIBERO/tree/8f1084e3132a39270c3a13ebe37270a43ece2a01),
+and can PR #3434 preserve it in a one-task, one-trial vertical slice?
+
+Only first-party source, official configuration, CaP-X at commit
+[`53e9966`](https://github.com/capgym/cap-x/tree/53e9966d7a8e2fa7494676772bccc35280f5c0ed),
+and the two benchmark papers were used.
+
+## Conclusions
+
+1. **LIBERO-PRO does not ship a standalone canonical rollout program.** It
+ ships perturbed suite registrations/assets, a perturbation generator, and a
+ README patch recipe for an external OpenVLA evaluator. Its simulator call
+ graph remains LIBERO's synchronous `reset -> set_init_state -> five settling
+ steps -> policy/action/step loop -> BDDL success` call graph.
+2. **The horizon is a count of policy actions, not wall time.** Model inference
+ occurs before each `env.step(action)` and therefore does not consume the
+ step budget. `control_freq=20` controls how much simulated time robosuite
+ advances per action; it does not require a continuously running 20 Hz host
+ thread.
+3. **The original and PRO suites are distinct artifacts, but they do not define
+ an exploration/scoring split.** Following CaP-X, DimOS selects the concrete
+ PRO suite and task first and discloses its exact language during exploration.
+ Debug and scored trials use fresh initial states of that same task.
+4. **One task and one trial can preserve episode semantics**, including the
+ native success bit, but it is only an integration result. A published-style
+ task success rate requires 50 episodes, and the leaderboard total is the
+ mean over suite-by-perturbation success rates.
+5. **No change to `Evaluation.run()` or the debug-trial callback is proven
+ necessary.** The evaluator can own the fresh environment and blueprint and
+ invoke `runtime.execute()` beside them. However, a continuous wall-clock
+ simulator design would change benchmark semantics. If early policy-process
+ cancellation is required for operational cleanup, the current synchronous
+ `CodePolicyRuntime.execute()` lacks that lifecycle control; that is an
+ operational gap, not a reason to move LIBERO's horizon or scorer into the
+ generic runtime.
+
+## Native call graph
+
+LIBERO-PRO's README tells integrations to select or generate a perturbed suite,
+then hand that suite to an external evaluator; it does not replace LIBERO's
+rollout loop ([PRO README, evaluation patch recipe](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/README.md#evaluation-on-openvla)).
+There are therefore two relevant entry points, neither a PRO-generic CLI:
+
+- LIBERO's reusable rollout function is `evaluate_one_task_success()` in
+ `libero/lifelong/metric.py`.
+- `libero/lifelong/evaluate.py` is a checkpoint- and algorithm-specific CLI
+ around the same environment calls
+ ([source](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/lifelong/evaluate.py#L72-L304)).
+ The PRO instructions instead ask users to modify an OpenVLA repository's
+ `run_libero_eval.py`, which is not present in LIBERO-PRO.
+
+The inherited native path is:
+
+```text
+get_benchmark(suite)(task_order_index)
+ -> benchmark.get_task(task_id)
+ -> Task(problem_folder, bddl_file, init_states_file, language)
+
+OffScreenRenderEnv(
+ bddl_file_name=/.bddl,
+ camera_heights=128,
+ camera_widths=128,
+)
+ -> ControlEnv defaults:
+ Panda, OSC_POSE, 20 Hz, hard_reset=True,
+ agentview + robot0_eye_in_hand RGB
+
+for evaluation batch:
+ env.reset()
+ init_states = torch.load(/.pruned_init)
+ indices = arange(batch offset) % len(init_states)
+ obs = env.set_init_state(init_states[indices])
+ repeat 5 times:
+ obs = env.step(zeros(7))
+
+ steps = 0
+ dones = false
+ while steps < max_steps:
+ steps += 1
+ policy_input = transform(obs, task_embedding)
+ action = policy.get_action(policy_input)
+ obs, reward, done, info = env.step(action)
+ dones |= done
+ if all(dones): break
+
+ task_success_rate = sum(dones) / episode_count
+```
+
+The loop above is directly implemented by original LIBERO and is unchanged in
+the pinned PRO fork
+([original `evaluate_one_task_success`](https://github.com/Lifelong-Robot-Learning/LIBERO/blob/8f1084e3132a39270c3a13ebe37270a43ece2a01/libero/lifelong/metric.py#L52-L161),
+[PRO copy](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/lifelong/metric.py#L52-L161)).
+
+### Task, reset, init state, and seed
+
+The benchmark `Task` record binds a task name, language, problem folder, BDDL
+file, and matching `.pruned_init` file. Original 10-task suites can apply one
+of the fixed task-order permutations, whereas PRO's suffixed suites use their
+task-map order because they are not in `standard_10_task_suites`
+([PRO benchmark registry](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/libero/benchmark/__init__.py#L36-L43),
+[ordering behavior](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/libero/benchmark/__init__.py#L185-L231)).
+The first vertical slice should therefore use task-order index `0` (the
+identity order) and record the exact task name rather than assuming arbitrary
+ordered indices correspond across original and PRO suites.
+
+Each evaluation batch first performs a normal hard reset, then overwrites the
+MuJoCo state with a deterministic tensor chosen by index. `set_init_state`
+sets the flattened state, forwards MuJoCo, refreshes object state and
+observables, and returns a new observation
+([environment wrapper](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/libero/envs/env_wrapper.py#L90-L145)).
+The loop deterministically walks init-state rows modulo the tensor length
+([metric](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/lifelong/metric.py#L102-L123)).
+Consequently, the comparable episode identity is primarily the BDDL hash,
+init-state tensor hash, and row index. The environment `seed()` only calls
+NumPy's global seed; it must be recorded, but it is not a substitute for the
+fixed state identity
+([seed implementation](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/libero/envs/bddl_base_domain.py#L162-L164)).
+
+A fresh environment/process per DimOS submission is stricter isolation than
+upstream, which reuses an environment across batches. It preserves episode
+semantics as long as the same artifacts, state row, and settling sequence are
+used.
+
+### Observations
+
+The official LIBERO data configuration maps the policy surface to:
+
+| Modality | Environment keys |
+|---|---|
+| RGB, 128 x 128 | `agentview_image`, `robot0_eye_in_hand_image` |
+| Proprioception | `robot0_gripper_qpos`, `robot0_joint_pos` |
+| Depth | none |
+
+These mappings are unchanged in the pinned PRO fork
+([original data config](https://github.com/Lifelong-Robot-Learning/LIBERO/blob/8f1084e3132a39270c3a13ebe37270a43ece2a01/libero/configs/data/default.yaml#L1-L39),
+[PRO data config](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/configs/data/default.yaml#L1-L39)).
+The exact BDDL language string is a legitimate input. Reward, `done`, BDDL
+goal predicates, raw simulator state, object poses, depth, and segmentation are
+not part of this configured policy surface and must remain evaluator-private.
+Model-side transforms computed from the allowed inputs remain policy behavior.
+
+The frozen `policy(app)` signature has no language argument. Therefore the
+DimOS blueprint must expose the current exact instruction through a
+non-privileged normal runtime interface if semantic or task perturbations are
+to be evaluated. Baking only the original debug instruction into generated
+code would make those PRO tracks structurally impossible to solve.
+
+### Action and controller
+
+`ControlEnv` defaults to a Panda using robosuite's fixed-impedance `OSC_POSE`
+controller and a default gripper. It sets `control_freq=20` and creates the two
+camera observations at 128 x 128
+([environment defaults](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/libero/envs/env_wrapper.py#L12-L80)).
+LIBERO's five settling actions are explicitly seven-dimensional
+([metric](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/lifelong/metric.py#L119-L123)).
+For the default controller this is six end-effector pose deltas (translation
+and axis-angle rotation) plus one gripper command; robosuite documents six
+fixed-impedance `OSC_POSE` arm dimensions and delta control semantics
+([robosuite controller documentation](https://robosuite.ai/docs/modules/controllers.html#operational-space-control-pose-with-fixed-impedance)).
+
+Replacing this with Panda `JOINT_POSITION` changes the benchmark action space
+and should not be reported as directly comparable. DimOS may translate a
+normal manipulation API into the native seven-dimensional commands, but the
+commands delivered at the environment boundary and their per-step semantics
+must remain native.
+
+### Horizon and start point
+
+The benchmark step budget starts **after** `reset`, `set_init_state`, and five
+settling actions. The counter increments immediately before policy inference
+and the subsequent action step
+([metric loop](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/lifelong/metric.py#L109-L157)).
+It is a simulator-policy-step count. There is no wall-clock deadline in this
+loop.
+
+Original LIBERO's checked-in evaluator defaults to 600 policy steps and 20
+episodes
+([original eval config](https://github.com/Lifelong-Robot-Learning/LIBERO/blob/8f1084e3132a39270c3a13ebe37270a43ece2a01/libero/configs/eval/default.yaml#L1-L10)).
+LIBERO-PRO's OpenVLA integration recipe instead assigns suite-specific limits:
+
+| Suite family | PRO action-step limit |
+|---|---:|
+| Goal | 300 |
+| Spatial | 220 |
+| LIBERO-10 | 520 |
+| Object | 280 |
+
+The same family limit is used for all five perturbation suffixes
+([PRO README](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/README.md#evaluation-on-openvla)).
+These limits are therefore part of the published PRO integration protocol,
+not `ControlEnv.horizon` (whose default is 1000). The LIBERO task wrapper
+replaces robosuite's returned `done` with BDDL success, so the outer evaluation
+loop owns the actual failure horizon
+([step override](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/libero/envs/bddl_base_domain.py#L800-L809)).
+
+A continuously advancing wall-clock 20 Hz simulator would penalize inference
+latency with lost policy steps, unlike native evaluation. The benchmark-faithful
+bridge must advance exactly one native `env.step(action)` per policy tick and
+must not start that tick counter during blueprint startup or policy-process
+connection.
+
+### Success, score, and aggregation
+
+The task environment parses goal predicates from the selected BDDL and requires
+their conjunction. Its `step()` returns that private predicate as `done`; sparse
+reward is also 1 on success
+([goal conjunction](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/libero/envs/problems/libero_tabletop_manipulation.py#L135-L155),
+[reward](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/libero/envs/bddl_base_domain.py#L165-L189)).
+The evaluator latches `done` across steps and computes the task success rate as
+successful episodes divided by requested episodes
+([metric](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/lifelong/metric.py#L145-L161)).
+
+The PRO paper evaluates 50 episodes per task
+([paper, section 5.1](https://arxiv.org/pdf/2510.03827)), and the public
+leaderboard reports one normalized success rate for every combination of four
+suite families and five perturbation types. Its displayed `Total` is the
+arithmetic mean of those 20 cells (for example, the shown OpenVLA cells average
+to 0.5165, displayed as 0.52)
+([PRO leaderboard](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/README.md#libero-pro-model-leaderboard)).
+Although the paper calls 50 episodes consistent with original LIBERO, the
+pinned original and PRO config files both still say `n_eval: 20`; use the
+paper's explicit 50 for a PRO aggregate and preserve this discrepancy in the
+run manifest rather than silently treating the checked-in default as canonical.
+
+Thus a one-trial report should publish `success: 0|1`, exact episode identity,
+and `completed`/infrastructure status. It must be labeled a vertical-slice
+result, not a task success rate or LIBERO-PRO total.
+
+## Perturbations and task disclosure
+
+The PRO paper defines a task as instruction, environment (visual context,
+objects, and initial configuration), and binary goal predicate. Its
+perturbations change object attributes, initial configuration, language,
+task goal/object set, or environment; task perturbation is excluded from
+cross-type combinations
+([paper, sections 3.1 and 4](https://arxiv.org/pdf/2510.03827)).
+The repository implements these changes by rewriting BDDL and generating
+matching init-state tensors. Task perturbation changes language, goal, and
+objects of interest; other perturbators alter the relevant BDDL elements
+([perturbation pipeline](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/perturbation.py#L425-L538),
+[asset generation](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/perturbation.py#L541-L670)).
+
+The earlier proposal to treat these artifacts as a hidden transfer boundary
+was rejected after checking CaP-X. CaP-X selects the concrete LIBERO suite and
+task, injects that task's language into the code-generation prompt, and scores
+the generated code on that task. The DimOS boundary is therefore:
+
+```text
+Exploration (visible to Pi) Measured trial
+----------------------------------- -----------------------------------
+selected PRO suite and task same PRO suite and task
+exact selected task instruction same task instruction
+fresh debug init-state rows fresh scored init-state row
+native debug success feedback native result evaluator-private
+up to five fresh blueprints -> frozen policy, fresh blueprint
+```
+
+The task identity and instruction are not secrets. Privileged BDDL predicate
+state and the scored episode's native result remain evaluator-only while the
+policy runs. The Policy Artifact cannot be regenerated after observing the
+scored result.
+
+## Source and asset preparation
+
+The pinned commit's active benchmark task map is
+`libero/libero/benchmark/libero_suite_task_map.py`. It contains the complete
+set of registered suffixed suites, including `*_lan`, `*_object`, `*_swap`,
+`*_task`, and `*_env`
+([suffixed mappings](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/libero/libero/benchmark/libero_suite_task_map.py#L1102-L1389)).
+The repository also contains an older, shorter similarly named file one level
+above `benchmark/`; integrations must follow the import in
+`benchmark/__init__.py` and must not select that stale-looking path by filename
+alone.
+
+The BDDL and init-state corpus is distributed separately. Upstream instructs
+users to download both directories from the official dataset and move them
+under `libero/libero`; the pinned March 2026 source already contains the
+post-November-2025 benchmark registry update
+([preparation instructions](https://github.com/Zxy-MLlab/LIBERO-PRO/blob/eafdb809426b13153aa1e4c42d6601844217dfec/README.md#L157-L173)).
+Source preparation must verify the requested commit and hash the separately
+installed episode assets.
+
+## Mapping to PR #3434
+
+The existing contracts are sufficient for the vertical slice:
+
+```text
+Evaluation.run(config, context)
+ -> context.runtime.explore(
+ task_input=,
+ submit_debug_trial=,
+ )
+ -> freeze ExplorationOutcome.policy
+ -> create fresh same-task PRO environment + fresh DimOS blueprint
+ -> run policy process beside evaluator-owned synchronized native step loop
+ -> stop blueprint and environment
+ -> EvaluationReport(native success bit + episode manifest)
+```
+
+The debug callback already returns a fully stopped `TrialRun`, and
+`EvaluationReport` can carry native structured output. Neither requires
+LIBERO-specific fields.
+
+Two constraints should shape implementation without widening the generic
+interface:
+
+- Blueprint and simulator readiness must complete before the five settling
+ actions and measured policy-step counter.
+- The evaluator/bridge, not `CodePolicyRuntime`, must own BDDL selection,
+ init-state loading, step synchronization, horizon, native predicate, and
+ scoring.
+
+`CodePolicyRuntime.execute()` is synchronous and exposes no start barrier or
+cancel handle. The evaluator can run it concurrently with its bridge, so this
+does not by itself block native stepping. It does mean the evaluator cannot
+promptly terminate a still-running policy process on early native success; it
+can only stop the blueprint and wait for completion/timeout. Add generic
+cancellation only if the vertical slice proves this cleanup behavior is
+unacceptable. Do not use a wall-clock timeout as the benchmark horizon.
+
+## Vertical-slice acceptance criteria
+
+One trial preserves native semantics if it records and verifies all of the
+following:
+
+- exact selected PRO suite/task shared by debug and scored runs;
+- PRO source revision plus BDDL and init-state hashes;
+- task-order index `0`, exact task name, exact instruction, and init row;
+- Panda, default `OSC_POSE`, seven-dimensional action, and 20 Hz simulated
+ policy frequency;
+- two 128 x 128 RGB views plus joint/gripper proprioception only;
+- reset, exact state restoration, five zero-action settling steps, then the
+ suite-specific action-step horizon;
+- evaluator-private BDDL conjunction success, exposed only after shutdown;
+- a fresh environment and complete fresh DimOS blueprint for every debug and
+ measured attempt; and
+- a final report explicitly labeled `single_trial`, with no aggregate claim.
diff --git a/pyproject.toml b/pyproject.toml
index 4a480aa22d..11948aa805 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -157,6 +157,7 @@ dependencies = [
# DimSim scene client (dimos/simulation/dimsim/scene_client.py) imports `websocket`
# at module load; DimSim is a non-extra-gated robot connection backend.
"websocket-client>=1.8",
+ "grpcio>=1.81.1",
]
diff --git a/uv.lock b/uv.lock
index 3de78d55da..fae810e555 100644
--- a/uv.lock
+++ b/uv.lock
@@ -1642,6 +1642,7 @@ dependencies = [
{ name = "dimos-viewer" },
{ name = "eclipse-zenoh" },
{ name = "filelock" },
+ { name = "grpcio" },
{ name = "imagecodecs", version = "2025.3.30", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },
{ name = "imagecodecs", version = "2026.3.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.11.*'" },
{ name = "imagecodecs", version = "2026.6.26", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" },
@@ -2227,6 +2228,7 @@ requires-dist = [
{ name = "gdown", marker = "extra == 'misc'", specifier = ">=5.2.2" },
{ name = "googlemaps", marker = "extra == 'misc'", specifier = ">=4.10.0" },
{ name = "graspgenx", marker = "extra == 'graspgenx'", git = "https://github.com/NVlabs/GraspGenX.git?rev=b9429097728cb1c430dd78b92edf17ba318aad03" },
+ { name = "grpcio", specifier = ">=1.81.1" },
{ name = "gtsam-extended", marker = "extra == 'mapping'", specifier = ">=4.3a1.post1" },
{ name = "h5py", marker = "extra == 'learning'" },
{ name = "huggingface-hub", marker = "extra == 'graspgenx'", specifier = ">=0.30,<1" },