Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
0edda59
Add LLMAgent code
qianl-nv May 13, 2026
d4bdaa6
Add asset resolver
qianl-nv May 25, 2026
3162272
Move assert_unique_ids + assert_references_exist to post_init so it's…
qianl-nv May 26, 2026
06d4b0e
Update schema to better support unary relation
qianl-nv May 26, 2026
087c498
Simplify relation types in resolver
qianl-nv May 26, 2026
cf38ec3
Fix copyright years
qianl-nv May 26, 2026
7e5f58e
Add unit test
qianl-nv May 26, 2026
ae99097
Add pydantic to runtime deps
qianl-nv May 27, 2026
ef8f694
Use explicit ValueError for missing LLM API key
qianl-nv May 27, 2026
1a07ba6
Rewrite unary relation subjects in background override
qianl-nv May 27, 2026
7fc2ba0
Expose at_pose as an LLM-emittable relation kind
qianl-nv May 27, 2026
c941d64
Derive prompt kind enumerations from schema literals
qianl-nv May 27, 2026
2d27374
Replace _extract_json asserts with LLMResponseParseError
qianl-nv May 27, 2026
c6f2cfa
Document __init__ and generate_spec arguments on LLMAgent
qianl-nv May 27, 2026
9188c1b
Rename schema.py to llm_schema.py and SceneSpec to LLMEnvSpec
qianl-nv May 27, 2026
decd878
Surface resolution failures via has_resolution_errors
qianl-nv May 27, 2026
0ad51da
Add ping() health-check method to LLMAgent
qianl-nv May 27, 2026
16f8224
Add unit + remote-e2e tests for LLMAgent
qianl-nv May 27, 2026
85a7f83
Add llm_remote_e2e job to CI
qianl-nv May 27, 2026
472a48b
Remove Resolver into a new branch
qianl-nv May 27, 2026
d5cadd7
Do ont skip test_generate_spec_against_live_endpoint
qianl-nv May 27, 2026
f3a36c7
Revert "Move assert_unique_ids + assert_references_exist to post_init…
qianl-nv May 27, 2026
421a282
Revert ArenaEnvGraphSpec YAML-serialization additions
qianl-nv May 27, 2026
f04ba3e
Sanity-check LLM endpoint on LLMAgent construction
qianl-nv May 27, 2026
0409075
Fix llm_remote_e2e test
qianl-nv May 27, 2026
e409553
Use "atomic task" consistently in env-gen prompt and schema
qianl-nv May 28, 2026
0cde094
Drop implementation details from Item.scale comment
qianl-nv May 28, 2026
58dbc13
Drop Resolver cross-references from LLMEnvSpec schema
qianl-nv May 28, 2026
1989c8f
Move per-field LLM prompt rules into pydantic Field descriptions
qianl-nv May 28, 2026
3de65f8
Add required reasoning field to LLMEnvSpec for forced CoT
qianl-nv May 28, 2026
d67fd12
Move llm_env_gen package under environments/agentic_env_gen
qianl-nv May 28, 2026
1d1f8a4
Rename LLMEnvSpec to EnvIntentSpec
qianl-nv May 28, 2026
3336f26
Rename LLMAgent to EnvGenAgent and sweep LLM naming across env-gen pa…
qianl-nv May 28, 2026
fb42596
Adopt structured outputs and wire-validate at construction time
qianl-nv May 28, 2026
5cc6269
Guard empty choices list in check_structured_output_support
qianl-nv May 28, 2026
f92fa73
Allow empty tasks list in EnvIntentSpec
qianl-nv May 28, 2026
9cd53f4
Simplify check_structured_output_support to bool-or-raise
qianl-nv May 28, 2026
bd4293b
Mark inference e2e tests flacky; add TODO in code to handle retries l…
qianl-nv May 28, 2026
11fe53e
Guard empty choices list in ping
qianl-nv May 28, 2026
0a3c30a
Add todo for light/hdr images
qianl-nv May 28, 2026
dc1ed0d
Add TODO for relation params incompatibility with strict mode models
qianl-nv May 28, 2026
6dbeda4
Move e2e agentic env gen test back to default group
qianl-nv May 29, 2026
ab090cd
Simplify test comments and test cases
qianl-nv May 29, 2026
82708e2
Simplify comments
qianl-nv May 29, 2026
1f1f92d
Rename to initial_state_graph
qianl-nv May 29, 2026
0ddc52b
Further simplifies comments
qianl-nv May 29, 2026
3264581
Improve EnvIntentSpec descriptions
qianl-nv May 29, 2026
f9ecfdf
Test for env_intent relation is subset of all Relations
qianl-nv May 29, 2026
89ad100
Merge remote-tracking branch 'origin/main' into qianl/dev/agentic_llm
qianl-nv Jun 1, 2026
e41e465
Remove ci Verify ARENA_NV_API_KEY step
qianl-nv May 29, 2026
633f94a
Minor fixes from review comments
qianl-nv Jun 1, 2026
7367e8f
Update build_catalog_text to use baseclass
qianl-nv Jun 1, 2026
8d8edfd
More cleanups
qianl-nv Jun 1, 2026
8c8f9cd
Move folder
qianl-nv Jun 1, 2026
ada141d
Remove scale from spec
qianl-nv Jun 1, 2026
08fb51b
Merge remote-tracking branch 'origin/main' into qianl/dev/agentic_llm
qianl-nv Jun 1, 2026
92efc23
Agentic env gen pulls relations from registry
qianl-nv Jun 2, 2026
8d9bb93
Agent pulls task from registry; add decorate for agent-ready tasks
qianl-nv Jun 2, 2026
a184f25
Update IntentSpec Task to pull required params from runtime class
qianl-nv Jun 2, 2026
0d506e4
Rename generate_spec param catalog to asset_catalog
qianl-nv Jun 2, 2026
24bc9bb
Improves EnvironmentGenerationAgent docs
qianl-nv Jun 2, 2026
e5feae7
Rename to agent utils and remove check_structured_output helper
qianl-nv Jun 2, 2026
e58b1db
Merge origin/main into qianl/dev/agentic_llm
qianl-nv Jun 4, 2026
21aa0d2
Refactor RelationSpec/TaskSpec from ArenaEnvGraph types to reuse for …
qianl-nv Jun 4, 2026
e0f0ba6
Self review
qianl-nv Jun 4, 2026
eb8f802
Simplify validtor in IntentSpec
qianl-nv Jun 4, 2026
c2c3a52
Add doc explaining necessity of Item class
qianl-nv Jun 4, 2026
cf9b313
Drop unusage task_description
qianl-nv Jun 4, 2026
9dbd77f
Change taskspec from nesting to inheritance
qianl-nv Jun 4, 2026
7fb97c8
Merge remote-tracking branch 'origin/main' into qianl/dev/agentic_llm
qianl-nv Jun 5, 2026
54c16f6
Rename to SpatialRelationSpec
qianl-nv Jun 5, 2026
12079a0
Remove stale scale field from agent test fixture.
qianl-nv Jun 5, 2026
11b7421
Guard empty choices in generate_spec.
qianl-nv Jun 5, 2026
1536ce8
Restore inline comments in maple table graph test YAML.
qianl-nv Jun 5, 2026
d11cbe8
Require openai>=2.0 for agent structured-output calls.
qianl-nv Jun 5, 2026
c4a3fb5
Use assert instead of raise for agent API invariant checks.
qianl-nv Jun 5, 2026
497f810
Clarify tasks field description to discourage invented tasks.
qianl-nv Jun 5, 2026
1a36bf7
Rename ArenaEnvGraphSpatialConstraintSpec to ArenaEnvGraphSpatialRela…
qianl-nv Jun 5, 2026
d096110
Rename SpatialRelationSpec parent to reference
qianl-nv Jun 5, 2026
98364c2
Remove ItemRole from IntentSpec
qianl-nv Jun 5, 2026
f5da4fe
Attempt CI fix 1: make isaaclab_arena/tests/test_arena_env_graph_conv…
qianl-nv Jun 5, 2026
8224914
Revert "Attempt CI fix 1: make isaaclab_arena/tests/test_arena_env_gr…
qianl-nv Jun 8, 2026
43454f3
Merge remote-tracking branch 'origin/main' into qianl/dev/agentic_llm
qianl-nv Jun 8, 2026
89342c5
build_asset_catalogue uses tag instead of subclass to avoid early pxr…
qianl-nv Jun 8, 2026
8c0d1f2
Avoid registry import in test
qianl-nv Jun 8, 2026
d676457
More test fix
qianl-nv Jun 8, 2026
c557a21
Fix empty string error
qianl-nv Jun 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,9 @@ jobs:
runs-on: [self-hosted, gpu-arena]
timeout-minutes: 80
needs: [pre_commit]
env:
# NV_API_KEY is consumed by the live-endpoint tests for agentic env gen.
NV_API_KEY: ${{ secrets.ARENA_NV_API_KEY }}

container:
image: nvcr.io/nvstaging/isaac-amr/isaaclab_arena:latest
Expand Down
6 changes: 6 additions & 0 deletions docker/run_docker.sh
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,12 @@ else
fi
fi

# pass through API keys used by the agentic env-gen prototype; values are
# inherited from the host shell so the key never lives in the repo.
if [ -n "$NV_API_KEY" ]; then
DOCKER_RUN_ARGS+=("--env" "NV_API_KEY")
fi

# if gr00t is installed, mount the gr00t directory in case anything needs to change there
if [ "$INSTALL_GROOT" = "true" ]; then
DOCKER_RUN_ARGS+=("-v" "./submodules/Isaac-GR00T:${WORKDIR}/submodules/Isaac-GR00T")
Expand Down
4 changes: 4 additions & 0 deletions isaaclab_arena/agentic_environment_generation/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).
Comment thread
qianl-nv marked this conversation as resolved.
# All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0
84 changes: 84 additions & 0 deletions isaaclab_arena/agentic_environment_generation/agent_utils.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0

"""Utilities for the environment-generation agent (OpenAI-compatible API)."""

from __future__ import annotations

import copy
from typing import Any

from pydantic import BaseModel


def ping(client: Any, model: str) -> str:
"""Smoke-test the endpoint + API key + model with a minimal request.

Args:
client: An OpenAI-compatible client (typically
``openai.OpenAI`` or a compatible mock).
model: Model identifier forwarded to
``client.chat.completions.create(model=...)``.

Returns:
The model's response text.
"""
# TODO(qianl): wrap with transient-error retry.
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Respond with exactly: OK"}],
temperature=0,
max_tokens=8,
)
choices = getattr(resp, "choices", None) or []
assert choices, (
f"ping to model {model!r} returned HTTP 200 with no choices "
"(content filter / guardrail / rate-limit response with empty body)."
)
return choices[0].message.content or ""


def build_strict_schema(model_cls: type[BaseModel]) -> dict[str, Any]:
"""Return ``model_cls``'s JSON schema munged for OpenAI strict mode."""
schema = copy.deepcopy(model_cls.model_json_schema())
apply_strict_constraints(schema)
return schema


def apply_strict_constraints(node: Any) -> None:
"""Recursively apply OpenAI strict-mode constraints to a JSON-schema node."""
if isinstance(node, dict):
if node.get("type") == "object" and "properties" in node:
node["additionalProperties"] = False
node["required"] = list(node["properties"].keys())
# Strict mode forbids ``default`` keys (every field is required, so
# defaults can never apply). Drop them defensively at every level.
node.pop("default", None)
for v in node.values():
apply_strict_constraints(v)
elif isinstance(node, list):
for v in node:
apply_strict_constraints(v)


def extract_response_text(message: Any) -> tuple[str, str]:
"""Pull the agent's structured-output text from the chat-completion message.

Returns ``(text, route)`` where ``route`` is one of:

* ``"content"`` — the standard OpenAI-compatible channel.
* ``"reasoning_content"`` — NVIDIA DeepSeek's provider-specific
channel; the model emits structured outputs here instead of
``content``. We treat it as equivalent.
* ``"empty"`` — both channels were empty / missing; the caller
should surface a clear error.
"""
content = getattr(message, "content", None)
if content:
return content, "content"
reasoning = getattr(message, "reasoning_content", None)
if reasoning:
return reasoning, "reasoning_content"
return "", "empty"
Loading
Loading