-
Notifications
You must be signed in to change notification settings - Fork 62
v0.3 Add EnvGenAgent for agentic env gen #718
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
87 commits
Select commit
Hold shift + click to select a range
0edda59
Add LLMAgent code
qianl-nv d4bdaa6
Add asset resolver
qianl-nv 3162272
Move assert_unique_ids + assert_references_exist to post_init so it's…
qianl-nv 06d4b0e
Update schema to better support unary relation
qianl-nv 087c498
Simplify relation types in resolver
qianl-nv cf38ec3
Fix copyright years
qianl-nv 7e5f58e
Add unit test
qianl-nv ae99097
Add pydantic to runtime deps
qianl-nv ef8f694
Use explicit ValueError for missing LLM API key
qianl-nv 1a07ba6
Rewrite unary relation subjects in background override
qianl-nv 7fc2ba0
Expose at_pose as an LLM-emittable relation kind
qianl-nv c941d64
Derive prompt kind enumerations from schema literals
qianl-nv 2d27374
Replace _extract_json asserts with LLMResponseParseError
qianl-nv c6f2cfa
Document __init__ and generate_spec arguments on LLMAgent
qianl-nv 9188c1b
Rename schema.py to llm_schema.py and SceneSpec to LLMEnvSpec
qianl-nv decd878
Surface resolution failures via has_resolution_errors
qianl-nv 0ad51da
Add ping() health-check method to LLMAgent
qianl-nv 16f8224
Add unit + remote-e2e tests for LLMAgent
qianl-nv 85a7f83
Add llm_remote_e2e job to CI
qianl-nv 472a48b
Remove Resolver into a new branch
qianl-nv d5cadd7
Do ont skip test_generate_spec_against_live_endpoint
qianl-nv f3a36c7
Revert "Move assert_unique_ids + assert_references_exist to post_init…
qianl-nv 421a282
Revert ArenaEnvGraphSpec YAML-serialization additions
qianl-nv f04ba3e
Sanity-check LLM endpoint on LLMAgent construction
qianl-nv 0409075
Fix llm_remote_e2e test
qianl-nv e409553
Use "atomic task" consistently in env-gen prompt and schema
qianl-nv 0cde094
Drop implementation details from Item.scale comment
qianl-nv 58dbc13
Drop Resolver cross-references from LLMEnvSpec schema
qianl-nv 1989c8f
Move per-field LLM prompt rules into pydantic Field descriptions
qianl-nv 3de65f8
Add required reasoning field to LLMEnvSpec for forced CoT
qianl-nv d67fd12
Move llm_env_gen package under environments/agentic_env_gen
qianl-nv 1d1f8a4
Rename LLMEnvSpec to EnvIntentSpec
qianl-nv 3336f26
Rename LLMAgent to EnvGenAgent and sweep LLM naming across env-gen pa…
qianl-nv fb42596
Adopt structured outputs and wire-validate at construction time
qianl-nv 5cc6269
Guard empty choices list in check_structured_output_support
qianl-nv f92fa73
Allow empty tasks list in EnvIntentSpec
qianl-nv 9cd53f4
Simplify check_structured_output_support to bool-or-raise
qianl-nv bd4293b
Mark inference e2e tests flacky; add TODO in code to handle retries l…
qianl-nv 11fe53e
Guard empty choices list in ping
qianl-nv 0a3c30a
Add todo for light/hdr images
qianl-nv dc1ed0d
Add TODO for relation params incompatibility with strict mode models
qianl-nv 6dbeda4
Move e2e agentic env gen test back to default group
qianl-nv ab090cd
Simplify test comments and test cases
qianl-nv 82708e2
Simplify comments
qianl-nv 1f1f92d
Rename to initial_state_graph
qianl-nv 0ddc52b
Further simplifies comments
qianl-nv 3264581
Improve EnvIntentSpec descriptions
qianl-nv f9ecfdf
Test for env_intent relation is subset of all Relations
qianl-nv 89ad100
Merge remote-tracking branch 'origin/main' into qianl/dev/agentic_llm
qianl-nv e41e465
Remove ci Verify ARENA_NV_API_KEY step
qianl-nv 633f94a
Minor fixes from review comments
qianl-nv 7367e8f
Update build_catalog_text to use baseclass
qianl-nv 8d8edfd
More cleanups
qianl-nv 8c8f9cd
Move folder
qianl-nv ada141d
Remove scale from spec
qianl-nv 08fb51b
Merge remote-tracking branch 'origin/main' into qianl/dev/agentic_llm
qianl-nv 92efc23
Agentic env gen pulls relations from registry
qianl-nv 8d9bb93
Agent pulls task from registry; add decorate for agent-ready tasks
qianl-nv a184f25
Update IntentSpec Task to pull required params from runtime class
qianl-nv 0d506e4
Rename generate_spec param catalog to asset_catalog
qianl-nv 24bc9bb
Improves EnvironmentGenerationAgent docs
qianl-nv e5feae7
Rename to agent utils and remove check_structured_output helper
qianl-nv e58b1db
Merge origin/main into qianl/dev/agentic_llm
qianl-nv 21aa0d2
Refactor RelationSpec/TaskSpec from ArenaEnvGraph types to reuse for …
qianl-nv e0f0ba6
Self review
qianl-nv eb8f802
Simplify validtor in IntentSpec
qianl-nv c2c3a52
Add doc explaining necessity of Item class
qianl-nv cf9b313
Drop unusage task_description
qianl-nv 9dbd77f
Change taskspec from nesting to inheritance
qianl-nv 7fb97c8
Merge remote-tracking branch 'origin/main' into qianl/dev/agentic_llm
qianl-nv 54c16f6
Rename to SpatialRelationSpec
qianl-nv 12079a0
Remove stale scale field from agent test fixture.
qianl-nv 11b7421
Guard empty choices in generate_spec.
qianl-nv 1536ce8
Restore inline comments in maple table graph test YAML.
qianl-nv d11cbe8
Require openai>=2.0 for agent structured-output calls.
qianl-nv c4a3fb5
Use assert instead of raise for agent API invariant checks.
qianl-nv 497f810
Clarify tasks field description to discourage invented tasks.
qianl-nv 1a36bf7
Rename ArenaEnvGraphSpatialConstraintSpec to ArenaEnvGraphSpatialRela…
qianl-nv d096110
Rename SpatialRelationSpec parent to reference
qianl-nv 98364c2
Remove ItemRole from IntentSpec
qianl-nv f5da4fe
Attempt CI fix 1: make isaaclab_arena/tests/test_arena_env_graph_conv…
qianl-nv 8224914
Revert "Attempt CI fix 1: make isaaclab_arena/tests/test_arena_env_gr…
qianl-nv 43454f3
Merge remote-tracking branch 'origin/main' into qianl/dev/agentic_llm
qianl-nv 89342c5
build_asset_catalogue uses tag instead of subclass to avoid early pxr…
qianl-nv 8c0d1f2
Avoid registry import in test
qianl-nv d676457
More test fix
qianl-nv c557a21
Fix empty string error
qianl-nv File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| # Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). | ||
| # All rights reserved. | ||
| # | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
84 changes: 84 additions & 0 deletions
84
isaaclab_arena/agentic_environment_generation/agent_utils.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| # Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). | ||
| # All rights reserved. | ||
| # | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """Utilities for the environment-generation agent (OpenAI-compatible API).""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import copy | ||
| from typing import Any | ||
|
|
||
| from pydantic import BaseModel | ||
|
|
||
|
|
||
| def ping(client: Any, model: str) -> str: | ||
| """Smoke-test the endpoint + API key + model with a minimal request. | ||
|
|
||
| Args: | ||
| client: An OpenAI-compatible client (typically | ||
| ``openai.OpenAI`` or a compatible mock). | ||
| model: Model identifier forwarded to | ||
| ``client.chat.completions.create(model=...)``. | ||
|
|
||
| Returns: | ||
| The model's response text. | ||
| """ | ||
| # TODO(qianl): wrap with transient-error retry. | ||
| resp = client.chat.completions.create( | ||
| model=model, | ||
| messages=[{"role": "user", "content": "Respond with exactly: OK"}], | ||
| temperature=0, | ||
| max_tokens=8, | ||
| ) | ||
| choices = getattr(resp, "choices", None) or [] | ||
| assert choices, ( | ||
| f"ping to model {model!r} returned HTTP 200 with no choices " | ||
| "(content filter / guardrail / rate-limit response with empty body)." | ||
| ) | ||
| return choices[0].message.content or "" | ||
|
|
||
|
|
||
| def build_strict_schema(model_cls: type[BaseModel]) -> dict[str, Any]: | ||
| """Return ``model_cls``'s JSON schema munged for OpenAI strict mode.""" | ||
| schema = copy.deepcopy(model_cls.model_json_schema()) | ||
| apply_strict_constraints(schema) | ||
| return schema | ||
|
|
||
|
|
||
| def apply_strict_constraints(node: Any) -> None: | ||
| """Recursively apply OpenAI strict-mode constraints to a JSON-schema node.""" | ||
| if isinstance(node, dict): | ||
| if node.get("type") == "object" and "properties" in node: | ||
| node["additionalProperties"] = False | ||
| node["required"] = list(node["properties"].keys()) | ||
| # Strict mode forbids ``default`` keys (every field is required, so | ||
| # defaults can never apply). Drop them defensively at every level. | ||
| node.pop("default", None) | ||
| for v in node.values(): | ||
| apply_strict_constraints(v) | ||
| elif isinstance(node, list): | ||
| for v in node: | ||
| apply_strict_constraints(v) | ||
|
|
||
|
|
||
| def extract_response_text(message: Any) -> tuple[str, str]: | ||
| """Pull the agent's structured-output text from the chat-completion message. | ||
|
|
||
| Returns ``(text, route)`` where ``route`` is one of: | ||
|
|
||
| * ``"content"`` — the standard OpenAI-compatible channel. | ||
| * ``"reasoning_content"`` — NVIDIA DeepSeek's provider-specific | ||
| channel; the model emits structured outputs here instead of | ||
| ``content``. We treat it as equivalent. | ||
| * ``"empty"`` — both channels were empty / missing; the caller | ||
| should surface a clear error. | ||
| """ | ||
| content = getattr(message, "content", None) | ||
| if content: | ||
| return content, "content" | ||
| reasoning = getattr(message, "reasoning_content", None) | ||
| if reasoning: | ||
| return reasoning, "reasoning_content" | ||
| return "", "empty" |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.