Skip to content

Fix LEAPP projected gravity export#6007

Merged
lgulich merged 8 commits into
isaac-sim:developfrom
lgulich:lgulich/leapp-root-quat-gravity
Jun 8, 2026
Merged

Fix LEAPP projected gravity export#6007
lgulich merged 8 commits into
isaac-sim:developfrom
lgulich:lgulich/leapp-root-quat-gravity

Conversation

@lgulich

@lgulich lgulich commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

Summary

The current LEAPP export exports projected_gravity as an input instead of root_quat_w.
The reason is that with the change to use warp for observation terms LEAPP can no longer automatically trace projected gravity back to root_quat_w. For new we fix this manually, once we have warp support in LEAPP this will no longer be needed.

Changes

  • Export mdp.projected_gravity through root_quat_w for LEAPP graphs.
  • Keep the policy observation as projected gravity by computing it inside the exported graph.
  • Add regression coverage and a changelog fragment.

Test Plan

  • uv run --with pytest --with leapp python -m pytest source/isaaclab_rl/test/export/test_leapp_proxy.py -q
  • uv run --with ruff --with leapp ruff check source/isaaclab/isaaclab/utils/leapp/export_annotator.py source/isaaclab/isaaclab/utils/leapp/proxy.py source/isaaclab_rl/test/export/test_leapp_proxy.py
  • ./isaaclab.sh -f
  • Exported Isaac-Velocity-Flat-G1-v0 checkpoint model_1499.pt and verified YAML uses state/body/rotation for robot_root_quat_w.

Expose root orientation as the LEAPP graph input for projected gravity observations and compute the projected gravity tensor inside the exported graph. This keeps the policy observation contract unchanged while matching deployment inputs.
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Jun 6, 2026
@lgulich lgulich requested a review from frlai June 6, 2026 15:09

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

This PR correctly fixes the LEAPP export of projected_gravity by exposing the root quaternion orientation (root_quat_w) at the graph boundary instead of the pre-computed projected gravity vector. This is the right architectural decision since deployment backends (robot hardware/simulation) typically provide body orientation from IMUs rather than pre-computed projected gravity.

✅ Strengths

  1. Sound Architecture: Exposing root_quat_w as a BODY_ROTATION input instead of projected_gravity as VECTOR3D aligns with what deployment backends actually provide. The projection computation happens inside the exported graph where it belongs.

  2. Correct Quaternion Math: The inline quaternion inverse rotation implementation correctly matches the existing math_utils.quat_apply_inverse formula and properly handles Isaac Sim's (x, y, z, w) scalar-last quaternion convention.

  3. Good Test Coverage: The new tests verify both the direct proxy path (preserving VECTOR3D semantics) and the observation export path (using root_quat_w with BODY_ROTATION semantics), with mathematical correctness validated against the reference implementation.

  4. Proper Changelog Entry: Follows the project's changelog conventions.

📝 Follow-up Review (d30f2b7)

The new commits since the initial review (30d66c0d30f2b7) contain only housekeeping changes:

  • Documentation updates to use the new task naming convention (Isaac-Reach-UR10 instead of Isaac-Reach-UR10-v0)
  • Test file rename and task name updates to match the codebase-wide rename
  • Changelog .skip marker added

No changes to the core LEAPP projected gravity fix logic. The suggestions from the initial review remain optional enhancements and do not block merge.

📋 Overall Assessment

This is a well-implemented fix that addresses a real deployment concern. The approach is architecturally sound, the math is correct, and the test coverage validates the key behaviors. Ready to merge. ✅


Update (e290070): New commits are a merge of develop into the feature branch (cloner refactor #5770, OvPhysX runtime fix #5995, CI changelog bump). No changes to the LEAPP projected gravity fix. Previous assessment stands — ready to merge.


Update (2bd4a4b): New commit fixes hidden_statehidden_states attribute name in two places in export.py (lines 202 and 332). Straightforward correctness fix aligning with the actual API. No new issues introduced. Previous assessment unchanged — ready to merge. ✅


Update (ae5481a): Merge of develop into the feature branch — brings in docs command updates (#6023) and PPISP optional-dependency refactor (#5986). No changes to LEAPP export logic or tests. No new issues. ✅


Update (6f18d1c): This commit reverts the previous hidden_states back to hidden_state (singular) in export.py (lines 202 and 332). The commit message explains the rationale well: rsl_rl 5.0.1 (which Isaac Lab pins) uses hidden_state (singular), while the plural form (hidden_states) only exists in older/vendored forks like AGILE's 2.3.3.

This is a good catch — the earlier commit (2bd4a4b) was incorrect for the pinned rsl_rl version. The revert restores the correct attribute name. Net effect: the export.py file is unchanged from its state before the 2bd4a4b commit, which is correct for rsl_rl 5.0.1.

No new issues. Core LEAPP projected gravity fix remains intact. ✅

@frlai

frlai commented Jun 6, 2026

Copy link
Copy Markdown
Collaborator

can you move the tests to: IsaacLab/source/isaaclab_rl/test/export. that's where all the leapp export tests are.

@lgulich lgulich marked this pull request as ready for review June 8, 2026 07:17
@greptile-apps

greptile-apps Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes the LEAPP graph export for the projected_gravity observation by surfacing root_quat_w (body orientation) as the graph input instead of an already-projected vector, then re-deriving the projected gravity inside the exported graph.

  • ExportPatcher._wrap_projected_gravity is added: when the observation manager is patched, any projected_gravity term is replaced by a wrapper that reads root_quat_w through the LEAPP proxy (annotating it as a BODY_ROTATION input) and applies the inverse quaternion rotation to [0, 0, -1] to reconstruct the body-frame gravity vector.
  • Two regression tests are added in source/isaaclab_rl/test/export/test_leapp_proxy.py: one verifying the direct _DataProxy path retains its VECTOR3D semantic, and one verifying the export-patched path emits the correct BODY_ROTATION semantic and produces numerically correct gravity projections.

Confidence Score: 4/5

Safe to merge; the quaternion convention and rotation formula are correct, and regression tests validate both the annotated and unannotated paths.

The core math (xyzw convention, Rodrigues inverse-rotation formula) matches the existing quat_apply_inverse helper exactly, the asset-name fallback to robot handles the default-config case correctly, and the tests assert numerical correctness against the reference implementation. The only things worth noting are a minor duplication of the rotation formula and a stale file path in the PR description test-plan commands that could confuse contributors running the tests manually.

The new test file at source/isaaclab_rl/test/export/test_leapp_proxy.py — verify the CI configuration picks it up from that location rather than the path quoted in the PR description.

Important Files Changed

Filename Overview
source/isaaclab/isaaclab/utils/leapp/export_annotator.py Adds _wrap_projected_gravity which exports root orientation as root_quat_w and re-derives the projected gravity in-graph using a hand-rolled quat_apply_inverse; formula and quaternion convention (x,y,z,w) are correct.
source/isaaclab_rl/test/export/test_leapp_proxy.py New regression test validating both the direct _DataProxy.projected_gravity_b path (VECTOR3D semantic) and the export-patched path (BODY_ROTATION semantic via root_quat_w); test logic is correct but file is at a different path than the PR description's test-plan commands.
source/isaaclab/changelog.d/lgulich-leapp-root-quat-gravity.rst Changelog fragment documenting the fix; content is accurate.
source/isaaclab_rl/changelog.d/lgulich-leapp-root-quat-gravity.skip Empty skip file indicating no changelog entry is needed for the isaaclab_rl package.

Sequence Diagram

sequenceDiagram
    participant ObsManager as Observation Manager
    participant Patcher as ExportPatcher
    participant Wrapper as _wrap_projected_gravity
    participant ProxyEnv as _EnvProxy / _DataProxy
    participant LEAPP as LEAPP annotate

    ObsManager->>Patcher: _patch_observation_manager(obs_manager, proxy_env)
    Patcher->>Patcher: "detect func_name == projected_gravity"
    Patcher->>Wrapper: _wrap_projected_gravity(original_func, proxy_env)
    Patcher-->>ObsManager: "replace term_cfg.func, set noise=None"

    Note over ObsManager: At policy rollout / export trace
    ObsManager->>Wrapper: term_cfg.func(env, [asset_cfg])
    Wrapper->>ProxyEnv: proxy_env.scene[asset_name].data.root_quat_w.torch
    ProxyEnv->>LEAPP: annotate.input_tensors(task_name, BODY_ROTATION semantics)
    LEAPP-->>ProxyEnv: annotated root_quat_w tensor
    ProxyEnv-->>Wrapper: root_quat_w (Nx4, x,y,z,w)
    Wrapper->>Wrapper: "gravity_w = [0,0,-1] broadcast to (N,3)"
    Wrapper->>Wrapper: "projected_gravity_b = quat_apply_inverse(root_quat_w, gravity_w)"
    Wrapper-->>ObsManager: projected_gravity_b (Nx3)
Loading

Reviews (1): Last reviewed commit: "Merge branch 'develop' into lgulich/leap..." | Re-trigger Greptile

Comment thread source/isaaclab_rl/test/export/test_leapp_proxy.py
Comment thread source/isaaclab/isaaclab/utils/leapp/export_annotator.py
The export wrote the traced recurrent state to `memory.hidden_state`, but rsl_rl's
Memory module reads/writes `hidden_states` (plural). The registered state tensors were
therefore never fed into the LSTM: LEAPP flagged the `actor_state_*` inputs as unused and
dropped them, producing a stateless ONNX (the LSTM reset to zeros every call). Write to
`hidden_states` so the hidden state is threaded as graph I/O and the recurrent policy
exports as stateful.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@lgulich lgulich requested a review from ooctipus as a code owner June 8, 2026 14:53
lgulich and others added 2 commits June 8, 2026 18:05
My earlier commit changed the actor memory write to `hidden_states` (plural), but
rsl_rl 5.0.1 (which Isaac Lab pins) stores the RNN state in `hidden_state` (singular,
rsl_rl/modules/rnn.py). The plural form only exists in older/vendored rsl_rl (e.g. AGILE's
2.3.3 fork) and as the algorithm-level (actor, critic) tuple. The original `hidden_state`
is correct for 5.0.1, so reverting; the AGILE recurrent-export issue is a vendored-rsl_rl
version mismatch to be fixed by upgrading AGILE to rsl_rl 5.0.1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@lgulich lgulich merged commit 74f9e8e into isaac-sim:develop Jun 8, 2026
37 checks passed
AntoineRichard pushed a commit that referenced this pull request Jun 10, 2026
## Summary

Backports #6007 to `release/3.0.0-beta2`.

This fixes LEAPP export for `mdp.projected_gravity` so the exported
graph takes `root_quat_w` as the input and computes projected gravity
inside the graph, instead of exporting projected gravity directly as an
input.

## Changes

- Export `mdp.projected_gravity` through `root_quat_w` for LEAPP graphs.
- Keep the policy observation as projected gravity by computing it
inside the exported graph.
- Add regression coverage and changelog fragments.

## Testing

- Red check: temporarily removed the projected-gravity dispatch and ran
`uv run --with pytest --with leapp python -m pytest
source/isaaclab_rl/test/export/test_leapp_proxy.py -q` -> failed as
expected in
`test_projected_gravity_observation_exports_root_quat_w_input`.
- Green check: restored the fix and reran `uv run --with pytest --with
leapp python -m pytest
source/isaaclab_rl/test/export/test_leapp_proxy.py -q` -> `2 passed, 37
warnings`.
- `uv run --with ruff --with leapp ruff check
source/isaaclab/isaaclab/utils/leapp/export_annotator.py
source/isaaclab/isaaclab/utils/leapp/proxy.py
source/isaaclab_rl/test/export/test_leapp_proxy.py` -> passed.
- `./isaaclab.sh -f` -> passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants