Add Newton inverse kinematics action

[maxkra15]

Description

Add Newton-backed inverse kinematics support to Isaac Lab.

This PR introduces a NewtonIKManager wrapper around Newton's objective-list IK solver API. The manager builds Newton pose objectives, optional joint-limit regularization, sampler settings, and custom objective passthrough into a batched Isaac Lab interface. Targets are updated from Torch tensors and converted to Warp arrays before calling Newton's IKSolver.

The PR also adds a manager-based NewtonInverseKinematicsAction that plugs this solver into Isaac Lab MDP action terms. Because Newton IK needs a single-articulation model while Isaac Lab runs replicated scenes, the Newton cloner now registers prototype builders during replication. NewtonManager.get_prototype_model() lazily finalizes the matching prototype on the active device, and the action resolves Isaac Lab joint/body names into Newton prototype joint-coordinate and link indices before solving.

Finally, this adds a Franka reach task variant using the new Newton IK action:

• Isaac-Reach-Franka-Newton-IK-Rel-v0
• Isaac-Reach-Franka-Newton-IK-Rel-Play-v0

The implementation includes tests for prototype-model lookup and cached prototype finalization behavior.

Fixes # (no issue filed)

Type of change

• New feature (non-breaking change which adds functionality)

Screenshots

Not applicable.

Test plan

• Added source/isaaclab_newton/test/physics/test_newton_prototype_models.py.
• Manually launched the Franka Newton IK task and verified environment setup:

  ./isaaclab.sh -p scripts/environments/random_agent.py \
    --task Isaac-Reach-Franka-Newton-IK-Rel-Play-v0 \
    --num_envs 1 \
    --visualizer newton

[isaaclab-review-bot]

🤖 Isaac Lab Review Bot - PR #5756

Summary

This PR adds Newton-backed inverse kinematics support to Isaac Lab, including a NewtonIKManager, NewtonInverseKinematicsAction, and Franka reach task variants. Overall, this is a well-structured feature with clean separation of concerns.

---

🏗️ Architecture & API Design

Strengths:

• ✅ Clean separation between NewtonIKManager (solver wrapper) and NewtonInverseKinematicsAction (MDP action term)
• ✅ Follows established Isaac Lab patterns for action terms (process_actions/apply_actions lifecycle)
• ✅ Good use of configclass for configuration with sensible defaults
• ✅ Prototype model registration integrates well with the existing cloner infrastructure
• ✅ Named pose objectives design allows flexible multi-target IK scenarios

Suggestions:

1. Document fixed-base limitation prominently - The fixed-base check is in the action init, but users should see this constraint in the class/config docstrings before they hit the runtime error
2. Consider adding get_cost() method - Currently costs returns raw Warp array; a convenience method returning per-env convergence status would aid debugging

---

🔍 Potential Silent Failure Modes

1. No convergence validation (newton_ik_manager.py:161-167)

   • solve() returns joint positions without checking if the IK actually converged
   • Consider adding an optional warn_on_high_cost parameter or returning a (joint_pos, converged_mask) tuple

2. Unreachable target handling (newton_ik_actions.py:213-218)

   • When targets lie outside the robot's workspace, Newton may return the last iteration's result with high residual cost
   • Silent failures here could cause unexpected robot behavior in RL training

3. Device state caching (newton_manager.py:792-795)

   if info.model is None or info.model_device != device:
       info.model = info.builder.finalize(device=device)

   • If the Newton model device changes mid-session, older cached prototype models could become stale
   • Consider invalidating all cached models when NewtonManager._model.device changes

4. Jacobian singularities - No explicit handling or damping adjustments when the manipulator approaches singular configurations. The lambda_initial parameter helps, but consider documenting expected behavior near singularities.

---

🧪 Test Coverage Assessment

Current tests (test_newton_prototype_models.py):

• ✅ Prototype model lookup with env regex paths
• ✅ Cached model reuse on same device

Missing test coverage:

• ❌ NewtonIKManager.solve() with various target poses and seeds
• ❌ NewtonInverseKinematicsAction integration with mock environment
• ❌ Edge cases: unreachable targets, near-singularity configurations
• ❌ Error conditions: invalid joint names, shape mismatches
• ❌ Multi-objective IK scenarios (multiple pose objectives)
• ❌ Device migration scenarios

Recommendation: Add at least basic unit tests for NewtonIKManager.set_target_pose() and solve() to ensure the Torch↔Warp conversion is correct.

---

📝 Minor Issues

1. CI Status: The "Check for Broken Links" check is failing - may need to add API docs links for the new classes

2. Docstring completeness (newton_ik_manager.py):

   • NewtonIKPoseObjective fields could use docstrings explaining units (radians vs degrees, etc.)

3. Magic numbers (newton_ik_manager_cfg.py):

   • Default iterations=24 and lambda_initial=0.1 work well for typical cases but the rationale for these defaults would be helpful in comments

4. Unused import (newton_ik_actions.py:12):

   • Sequence is imported from collections.abc but the reset method uses slice(None) pattern instead

---

✅ What Looks Good

• Changelog fragments present for both isaaclab_newton and isaaclab_tasks
• Type hints throughout
• Proper lazy export pattern for new subpackages
• The Franka task variant configuration is minimal and delegates properly to base configs
• Good logging in the action init for debugging joint/body resolution

---

This review was generated by the Isaac Lab Review Bot using ensemble analysis.

---

📝 Update (2ce0a4b)

New changes reviewed: This commit addresses several suggestions from the original review and adds significant new functionality.

✅ Improvements Made

1. Persistent IK seeds - New use_persistent_seed config option and set_joint_seed() / step() methods allow the solver to warm-start from previous solutions. This should improve convergence in continuous control scenarios.

2. Body transform helpers - set_target_pose_from_body_q() and set_pose_targets_from_body_q() make it easy to initialize pose objectives from live Newton body transforms, with proper handling of env_origins.

3. Test coverage added - New test_newton_ik_manager.py with tests for:

   • Persistent seed reuse between solves ✅
   • Pose target initialization from body transforms ✅
   • This addresses the "missing test coverage" concern from the original review 🎉

4. Improved reset() API - Now accepts optional env_ids and joint_pos parameters for selective reset with seed initialization.

🔍 New Code Review Notes

1. Type hints - Good use of Sequence[int] | torch.Tensor | slice | None for flexible env_ids handling

2. Helper functions - _as_torch() and _env_ids_to_tensor() are clean utility functions. Consider whether these should be module-private or potentially reusable elsewhere.

3. Changelog updated - Properly documents the new persistent seed and body transform helpers

📋 Remaining Suggestions (from original review)

• Convergence validation still not addressed (returning cost threshold / converged mask)
• Docstring for default hyperparameters rationale

Overall: Solid iteration that improves usability and addresses test coverage gaps. The persistent seed feature is particularly useful for RL training loops. 👍

---

📝 Update (ef4dae4)

New changes reviewed: Merge of upstream/develop into the feature branch.

Merge Conflict Resolution

This commit merges ~60+ upstream PRs from develop into the Newton IK branch. Four conflicts were resolved:

1. source/isaaclab_newton/isaaclab_newton/cloner/newton_replicate.py — Adapted to the new clone-plan source path resolution introduced by #5743 and #5887. ✅
2. source/isaaclab_newton/setup.py — Resolved packaging conflict from setup.py removal (#5785). ✅
3. source/isaaclab_tasks/isaaclab_tasks/core/reach/config/franka/__init__.py — Adapted to the core/contrib task split (#5891) while preserving the Newton IK task registrations. ✅
4. source/isaaclab_tasks/isaaclab_tasks/core/reach/config/franka/ik_newton_env_cfg.py — Same task restructuring, IK config preserved correctly. ✅

Impact on PR-Specific Code

The merge brings in several changes that the Newton IK code now benefits from:

• Clone-plan source resolution (#5887): The articulation init now uses resolve_matching_prims_from_source() which aligns with how NewtonIKManager resolves prototype models via the clone plan
• Gravity fix (#5833): GRAVITY_VEC_W now binds directly to model.gravity — the IK solver's prototype model lookups remain unaffected since IK operates in joint space
• Passive tendons (#5522): num_fixed_tendons property now works on Newton; no impact on the IK path but validates the broader articulation data layer the IK action builds on
• setup.py → pyproject.toml (#5785): Package metadata migration; the IK subpackage integrates cleanly

Assessment

The conflict resolutions are clean and correct. No IK-specific logic was altered by the merge. The PR remains in good shape for landing.

---

Updated by the Isaac Lab Review Bot.

---

📝 Update (0ccb62c)

New changes reviewed: Merge of upstream/develop (17 commits) into the feature branch, bringing the IK implementation onto the latest codebase.

Upstream Integration

This update merges the following notable upstream changes into the PR branch:

• EventTermCfg.resample_interval_on_reset (#5894) — New event manager feature
• ProxyArray from FrameView.get_scales (#5712) — API contract change affecting NewtonSiteFrameView
• NewtonSDFCollisionPropertiesCfg (#5926) — New collision schema by the same author
• Torch tensor support in reshape_data_to_view_3d (#5897) — Collection helper improvement
• Lazy export refactoring (#5916, #5920) — Warp-first experimental envs and cloner lazy exports
• Per-env omni:scenePartition authoring (#5445) — RTX scene partitioning
• Clone-plan source resolution fix (#5929) — Nested template resolution
• Cartpole task restructuring (#5904, #5930) — Task rename/merge

Impact on Newton IK Code

The Newton IK files (newton_ik_manager.py, newton_ik_actions.py, newton_ik_actions_cfg.py, newton_ik_manager_cfg.py) remain functionally unchanged from the previous review. The merge correctly integrates:

1. ✅ Lazy export pattern — The new isaaclab_newton/envs/, isaaclab_newton/envs/mdp/, and isaaclab_newton/envs/mdp/actions/ packages use lazy_export() with .pyi stubs, consistent with the upstream refactoring (#5916)
2. ✅ Prototype model registration — newton_replicate.py updated to pass protos dict through register_prototype_builders(), and instantiate_builder_from_stage() also registers prototypes for the non-replicated path
3. ✅ get_scales() returns ProxyArray — NewtonSiteFrameView.get_scales() now wraps its return in ProxyArray(), consistent with the upstream contract change (#5712)
4. ✅ Changelog fragment present (max-newton-ik-manager.rst)
5. ✅ Tests — Both test_newton_ik_manager.py and test_newton_prototype_models.py are included

📋 Previously Noted Suggestions (Status)

Suggestion	Status
Document fixed-base limitation	✅ Present in class and config docstrings
Test coverage for IKManager	✅ Fixed — tests added for persistent seed, target pose, body transforms
Convergence validation / warn-on-high-cost	⏸️ Not addressed (acceptable for initial merge, can be a follow-up)
Default hyperparameter rationale in comments	⏸️ Not addressed (minor, docs improvement)

Assessment

Clean merge with no conflicts. The Newton IK code integrates correctly with all upstream changes. No new issues introduced. Ready to proceed when the author marks it non-draft.

---

Updated by the Isaac Lab Review Bot.

---

📝 Update (bbc8e51)

New changes reviewed: Rebase onto latest develop with commit squashing.

Rebase Summary

This update squashes all previous commits (including merge commits from upstream/develop) into a single clean commit for landing. The Newton IK implementation is functionally unchanged from the previous review.

Code Status

The following files are present and unchanged in functionality:

• ✅ isaaclab_newton/ik/newton_ik_manager.py — batched IK wrapper with persistent seeds
• ✅ isaaclab_newton/ik/newton_ik_manager_cfg.py — configuration with sensible defaults
• ✅ isaaclab_newton/envs/mdp/actions/newton_ik_actions.py — MDP action term
• ✅ isaaclab_newton/envs/mdp/actions/newton_ik_actions_cfg.py — action configuration
• ✅ isaaclab_newton/physics/newton_manager.py — prototype model registration
• ✅ isaaclab_newton/cloner/newton_replicate.py — prototype builder passthrough
• ✅ test_newton_ik_manager.py and test_newton_prototype_models.py — unit tests
• ✅ Franka reach task variant (ik_newton_env_cfg.py)
• ✅ Changelog fragments for isaaclab_newton and isaaclab_tasks

📋 Outstanding Suggestions (Status Unchanged)

Suggestion	Status
Convergence validation / warn-on-high-cost	⏸️ Not addressed (follow-up opportunity)
Default hyperparameter rationale in comments	⏸️ Not addressed (minor, docs improvement)

Assessment

Clean rebase with no functional changes to the Newton IK feature. Ready for merge when the author removes draft status.

---

Updated by the Isaac Lab Review Bot.

---

📝 Update (01d05fa)

New changes reviewed: Amended commit with targeted bug fixes and safety improvements.

Changes

1. ✅ Bug fix: .clone() on IK solve output (newton_ik_manager.py)

   • solve() now returns .clone() of the Warp output buffer, preventing stale/overwritten results when the caller holds a reference across multiple solves
   • Corresponding test added: test_solve_result_is_independent_from_next_solve ✅

2. ✅ New safety guard: root orientation validation (newton_ik_actions.py)

   • _validate_matching_root_orientations() ensures all env root orientations match env 0 before solving IK on the shared prototype
   • Clear error message with affected env IDs if the assumption is violated
   • Handles quaternion sign ambiguity correctly (q and -q are the same rotation)

3. Prototype model persistence across clear() (newton_manager.py)

   • Removed _prototype_models = {} from NewtonManager.clear() — prototypes now persist across scene resets
   • ⚠️ Minor concern: if a scene is rebuilt with different assets, stale prototype entries could linger. Should be fine for typical usage (scene cleared → session destroyed) but worth noting.

4. New tests added:

   • test_solve_result_is_independent_from_next_solve — validates the .clone() fix
   • test_register_prototype_builders_accumulates_across_calls — validates multi-asset registration

Assessment

Good iteration — the .clone() fix prevents a subtle memory aliasing bug, and the root orientation guard is a smart runtime safety check. Both are well-tested. No new concerns beyond the minor note on prototype persistence.

---

Updated by the Isaac Lab Review Bot.

[greptile-apps]

Greptile Summary

This PR adds Newton-backed inverse kinematics support to Isaac Lab: a NewtonIKManager wrapper, a NewtonInverseKinematicsAction manager-based action term, and a Franka reach task variant (Isaac-Reach-Franka-Newton-IK-Rel-v0). The IK solves on a single-env prototype model built during Newton physics replication, and applies the resulting joint solutions back to the batched live articulation.

• Newton IK pipeline: body-frame targets → converted to the prototype (env 0) world frame → batched LM/L-BFGS solve on the prototype model → joint positions applied per env. For fixed-base robots with identical orientations this is provably equivalent to per-env solving, and the _validate_matching_root_orientations guard enforces the invariant at every step.
• Prototype registry: register_prototype_builders now accumulates entries across calls (previous version reset on each call), verified by the new test_register_prototype_builders_accumulates_across_calls test.
• solve() safety: now returns wp.to_torch(self.joint_q_out).clone() instead of a view, so callers can safely hold results across steps.

Confidence Score: 5/5

Safe to merge for fixed-base replicated scenes; the prototype-frame IK design is correct and the previously flagged registry-reset and solver-buffer issues are both addressed.

The core pipeline is architecturally sound: body-frame targets are correctly re-expressed in the prototype world frame, solve() now clones its output buffer, and register_prototype_builders accumulates rather than resets across calls. The two comments are both non-blocking style or minor quality improvements.

newton_ik_actions.py — specifically the absolute-pose quaternion branch and the reset() method. All other files are straightforward.

Important Files Changed

Filename	Overview
source/isaaclab_newton/isaaclab_newton/envs/mdp/actions/newton_ik_actions.py	Core action term — logic sound for fixed-base replicated scenes; absolute-pose quaternion mode writes policy output directly to _target_quat_b without normalization, which can cause incorrect IK targets when RL policies produce non-unit quaternions.
source/isaaclab_newton/isaaclab_newton/ik/newton_ik_manager.py	Well-structured batched IK wrapper; solve() now clones the output buffer, set_joint_seed validates shapes, and objective naming uniqueness is enforced. No issues found.
source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py	Prototype registry accumulates correctly via dict update (no reset inside register_prototype_builders); get_prototype_model handles lazy finalization and device changes; ambiguity detection via longest-path match is solid.
source/isaaclab_newton/isaaclab_newton/cloner/newton_replicate.py	Return type updated to carry prototype builders dict; register_prototype_builders called after replication; visualizer path correctly discards protos. No issues.
source/isaaclab_newton/isaaclab_newton/ik/newton_ik_manager_cfg.py	Configs well-documented with sensible defaults; joint_limit_weight optional; persistent seed off by default.
source/isaaclab_tasks/isaaclab_tasks/core/reach/config/franka/ik_newton_env_cfg.py	Correctly wires Newton physics backend, removes table, and configures NewtonInverseKinematicsActionCfg for Franka reach.
source/isaaclab_newton/test/ik/test_newton_ik_manager.py	Tests cover persistent seed reuse, result independence across solves, named-objective updates, and body-transform initialization; good coverage of the new IK manager.
source/isaaclab_newton/test/physics/test_newton_prototype_models.py	Verifies env-regex path matching, cached model reuse on same device, and accumulation across multiple register_prototype_builders calls. Addresses previous registry-reset concern directly.

Sequence Diagram

sequenceDiagram
    participant RL as RL Policy
    participant Act as NewtonInverseKinematicsAction
    participant IKM as NewtonIKManager
    participant NM as NewtonManager
    participant NR as newton_replicate

    NR->>NM: register_prototype_builders(sources, destinations, protos)
    Note over NM: _prototype_models[source_path] = NewtonPrototypeModelInfo

    Act->>NM: get_prototype_model(prim_path)
    NM-->>Act: info (builder.finalize(device) if needed)
    Act->>Act: _resolve_prototype_view(prototype_model)
    Act->>Act: _resolve_prototype_joint_coord_ids / _resolve_prototype_link_index
    Act->>IKM: "NewtonIKManager(cfg, model=prototype_model, num_envs, pose_objectives)"

    loop Each control step
        RL->>Act: process_actions(actions)
        Act->>Act: scale + clip actions
        Act->>Act: "_compute_frame_pose() -> ee_pos_b, ee_quat_b"
        Act->>Act: compute _target_pos_b, _target_quat_b

        Act->>Act: apply_actions()
        Act->>Act: _validate_matching_root_orientations()
        Act->>Act: "convert body-frame targets -> prototype world-frame"
        Act->>IKM: set_target_pose(name, target_pos_w, target_quat_w)
        Act->>Act: build joint_seed from _prototype_joint_seed + live joint_pos
        Act->>IKM: solve(joint_seed)
        IKM->>IKM: solver.step(joint_q_in, joint_q_out)
        IKM-->>Act: joint_pos_des_all (cloned)
        Act->>Act: extract controlled coord ids
        Act->>Act: set_joint_position_target_index(joint_pos_des, joint_ids)
    end

Loading

_{Reviews (2): Last reviewed commit: "Add Newton inverse kinematics action" | Re-trigger Greptile}