Problems in Deployment on a DROID-like setup using Franka Robot and OnRobot Gripper

[NeumannJulius]

Thank you for the opportunity to deploy Pi0 in real-world applications. Below is an overview of my setup and the challenges encountered during the deployment of the pretrained DROID Fast Policy.

---

Setup Details

• Hardware Components:

  • Robot: Franka Robot
  • Gripper: OnRobot Gripper
  • Cameras: Two Intel RealSense cameras (one wrist-mounted and one external)

• System Performance:

  • Inference Rate: Approximately 3Hz
  • Low-Level Controller Rate: Approximately 30Hz

• Controller Architecture:

  • Primary Controller: PID velocity/effort controller
  • Safety Controller: Enforces safe joint efforts, but tends to override action velocities

• Image Preprocessing:

  • Method: Images are zero-padded and resized
  • Inputs:

    • Base Image:
      

    • Wrist Image:
      

---

Observed Issues

1. Drift and Task Execution:

   • The robot drifts from its dedicated working area.
   • It does not display clear intent toward completing the desired task (e.g., "Pick up the orange cube").

2. Safety Controller Impact:

   • The safety controller’s limits are frequently exceeded on each joint.
   • This results in joint actions not reaching the intended velocities.

---

Discussion Points

Problem Source 1: Deployment Concerns

• Setup Adequacy:
  • Has anyone experienced similar issues with a comparable deployment?
  • Could the current hardware setup be insufficient for the DROID deployment?
• Model Fine-tuning:
  • Would fine-tuning the model on domain-specific data help address these issues?
• Gripper Usage:
  • Given that a gripper was not used during training, could this lead to decreasing performance?

Problem Source 2: Safety Controller and Velocity Handling

• Safety Controller Limits:
  • How can we minimize the impact of the safety controller, aside from completely disabling it?
  • Is proportional scaling of all joint actions a viable solution?
• Controller Alternatives:
  • Could issues arise from the current velocity control approach?
  • Would it be preferable to bypass the velocity controller and instead integrate velocities to drive a joint position controller?

---

Conclusion

Any insights or suggestions regarding these points—ranging from hardware setup adequacy to controller strategies—would be greatly appreciated. Looking forward to the discussion!