Numpy on macOS installing incorrect version

[gftabor]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

The build requires numpy 2.x but there isn't a runtime compatibility requirement for np 2.x so it installs 1.x. Then building new packages fails because the files are in incorrect locations
"numpy/_core/include" instead of "numpy/core/include"

(mqp) gftabor@x86_64-apple-darwin13 pixi_test % colcon build --symlink-install  --cmake-args -DBUILD_TESTING=OFF --packages-skip ros2_control trajectory_msgs

[0.275s] colcon.colcon_core.package_selection WARNING ignoring unknown package 'trajectory_msgs' in --packages-skip
Starting >>> joint_limits
Starting >>> ros2_control_test_assets
Starting >>> controller_manager_msgs
Finished <<< ros2_control_test_assets [4.37s]
--- stderr: joint_limits
CMake Warning (dev) at /Users/gftabor/Documents/pixi_test/.pixi/envs/default/share/realtime_tools/cmake/ament_cmake_export_dependencies-extras.cmake:21 (find_package):
  Policy CMP0167 is not set: The FindBoost module is removed.  Run "cmake
  --help-policy CMP0167" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

Call Stack (most recent call first):
  /Users/gftabor/Documents/pixi_test/.pixi/envs/default/share/realtime_tools/cmake/realtime_toolsConfig.cmake:41 (include)
  CMakeLists.txt:22 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Error in CMakeLists.txt:
  Imported target "trajectory_msgs::trajectory_msgs__rosidl_generator_py"
  includes non-existent path

    "/Users/gftabor/Documents/pixi_test/.pixi/envs/default/lib/python3.12/site-packages/numpy/_core/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.



CMake Error in CMakeLists.txt:
  Imported target "trajectory_msgs::trajectory_msgs__rosidl_generator_py"
  includes non-existent path

    "/Users/gftabor/Documents/pixi_test/.pixi/envs/default/lib/python3.12/site-packages/numpy/_core/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.



CMake Generate step failed.  Build files cannot be regenerated correctly.
---
Failed   <<< joint_limits [9.46s, exited with code 1]
Aborted  <<< controller_manager_msgs [13.1s]

Summary: 1 package finished [13.2s]
  1 package failed: joint_limits
  1 package aborted: controller_manager_msgs
  2 packages had stderr output: controller_manager_msgs joint_limits
  10 packages not processed
(mqp) gftabor@x86_64-apple-darwin13 pixi_test % pixi list | grep numpy
 WARN Encountered 1 warning while parsing the manifest:
  ⚠ The `project` field is deprecated. Use `workspace` instead.
   ╭─[/Users/gftabor/Documents/pixi_test/pixi.toml:1:1]
 1 │ ╭─▶ [project]
 2 │ │   name = "mqp"
 3 │ │   version = "0.1.0"
 4 │ │   description = "Development environment for VEX MQP"
 5 │ │   authors = ["Your Name <your.email@example.com>"]
 6 │ │   channels = ["https://prefix.dev/conda-forge","https://prefix.dev/robostack-jazzy"]
 7 │ ├─▶ platforms = ["linux-64", "osx-64", "win-64","linux-aarch64"]
   · ╰──── replace this with 'workspace'
 8 │     
   ╰────

numpy                                             1.26.4              py312he3a82b2_0           6.67 MiB  conda  https://prefix.dev/conda-forge

Installed packages

[project]
name = "mqp"
version = "0.1.0"
description = "Development environment for VEX MQP"
authors = ["Your Name <your.email@example.com>"]
channels = ["https://prefix.dev/conda-forge","https://prefix.dev/robostack-jazzy"]
platforms = ["linux-64", "osx-64", "win-64","linux-aarch64"]


[dependencies]
compilers = "*"
cmake = "*"
pkg-config = "*"
make = "*"
ninja = "*"
pip = ">=25.1.1,<26"

ros-jazzy-desktop = "*"
python = "3.12.*"
setuptools = "<80"
colcon-common-extensions = "*"
rosdep = "*"
ros-jazzy-ament-cmake-google-benchmark = "*"
ros-jazzy-performance-test-fixture = "*"
ros-jazzy-mimick-vendor = "*"
ros-jazzy-ament-cmake-vendor-package = "*"
ros-jazzy-ament-cmake-clang-format = "*"
ros-jazzy-test-msgs = "*"
ros-jazzy-navigation2 = "*"
ros-jazzy-ros-gz-sim = "*"
#ros-jazzy-ros-gz-bridge = "*"
#ros-jazzy-gz-ros2-control = "*"
ros-jazzy-ackermann-msgs = "*"
filelock = "*"
#ros-jazzy-ros2-controllers = "*"
#ros-jazzy-gz-ros2-control-demos = ">=1.2.17,<2"
tinyxml2 = ">=11.0.0,<12"
ros-jazzy-realtime-tools = "*"
ros-jazzy-control-msgs = ">=5.8.0,<6"
ros-jazzy-pal-statistics = "*"
ros-jazzy-rsl = "*"
ros-jazzy-generate-parameter-library = "*"
ros-jazzy-xacro = "*"
ros2-distro-mutex  = "0.14.*"
ros-jazzy-ros2-control = "*"
ros-jazzy-joint-limits = "*"
ros-jazzy-ros2-control-cmake = "*"
ros-jazzy-control-toolbox = "*"
ros-jazzy-ros-gz-bridge = "*"
ros-jazzy-gz-ros2-control-demos ="*"
ros-jazzy-hardware-interface-testing = ">=4.43.0,<5"


[target.linux-aarch64.dependencies]
ros-jazzy-ros2-control = "*"



[target.linux-64.dependencies]
ros-jazzy-ros2-control = "*"


[target.win-64.dependencies]
vs2022_win-64 = "*"  # Uncomment if using Visual Studio 2022



[activation]
scripts = ["setup.sh"]

[target.win-64.activation]
# Overwrite activation scripts only for windows
scripts = ["setup.bat"]

Environment info

[project]
name = "mqp"
version = "0.1.0"
description = "Development environment for VEX MQP"
authors = ["Your Name <your.email@example.com>"]
channels = ["https://prefix.dev/conda-forge","https://prefix.dev/robostack-jazzy"]
platforms = ["linux-64", "osx-64", "win-64","linux-aarch64"]


[dependencies]
compilers = "*"
cmake = "*"
pkg-config = "*"
make = "*"
ninja = "*"
pip = ">=25.1.1,<26"

ros-jazzy-desktop = "*"
python = "3.12.*"
setuptools = "<80"
colcon-common-extensions = "*"
rosdep = "*"
ros-jazzy-ament-cmake-google-benchmark = "*"
ros-jazzy-performance-test-fixture = "*"
ros-jazzy-mimick-vendor = "*"
ros-jazzy-ament-cmake-vendor-package = "*"
ros-jazzy-ament-cmake-clang-format = "*"
ros-jazzy-test-msgs = "*"
ros-jazzy-navigation2 = "*"
ros-jazzy-ros-gz-sim = "*"
#ros-jazzy-ros-gz-bridge = "*"
#ros-jazzy-gz-ros2-control = "*"
ros-jazzy-ackermann-msgs = "*"
filelock = "*"
#ros-jazzy-ros2-controllers = "*"
#ros-jazzy-gz-ros2-control-demos = ">=1.2.17,<2"
tinyxml2 = ">=11.0.0,<12"
ros-jazzy-realtime-tools = "*"
ros-jazzy-control-msgs = ">=5.8.0,<6"
ros-jazzy-pal-statistics = "*"
ros-jazzy-rsl = "*"
ros-jazzy-generate-parameter-library = "*"
ros-jazzy-xacro = "*"
ros2-distro-mutex  = "0.14.*"
ros-jazzy-ros2-control = "*"
ros-jazzy-joint-limits = "*"
ros-jazzy-ros2-control-cmake = "*"
ros-jazzy-control-toolbox = "*"
ros-jazzy-ros-gz-bridge = "*"
ros-jazzy-gz-ros2-control-demos ="*"
ros-jazzy-hardware-interface-testing = ">=4.43.0,<5"


[target.linux-aarch64.dependencies]
ros-jazzy-ros2-control = "*"



[target.linux-64.dependencies]
ros-jazzy-ros2-control = "*"


[target.win-64.dependencies]
vs2022_win-64 = "*"  # Uncomment if using Visual Studio 2022



[activation]
scripts = ["setup.sh"]

[target.win-64.activation]
# Overwrite activation scripts only for windows
scripts = ["setup.bat"]

[traversaro]

Interesting. From the ABI point of view, if I recall correctly packages built with numpy 2 are indeed compatibile with numpy 1, if I am not wrong. The problem here is that the concrete numpy include dirs are leaked in the *__rosidl_generator_py, that instead if should find and use Numpy::numpy. Just a curiosity, do you have any idea why numpy 1 is installed when numpy 2 is available? What happens if in the same env if you force numpy>=2?

[gftabor]

Interesting. From the ABI point of view, if I recall correctly packages built with numpy 2 are indeed compatibile with numpy 1, if I am not wrong. The problem here is that the concrete numpy include dirs are leaked in the *__rosidl_generator_py, that instead if should find and use Numpy::numpy. Just a curiosity, do you have any idea why numpy 1 is installed when numpy 2 is available? What happens if in the same env if you force numpy>=2?

It installed fine ( i did numpy 2.* but should be the same). From there that include error seemed to have gone away.

[Tobias-Fischer]

Quite likely because of our patch, maybe we shouldn't make the interfaces public and keep them private? https://github.com/RoboStack/ros-jazzy/blob/main/patch/ros-jazzy-rosidl-generator-py.osx.patch

[traversaro]

Quite likely because of our patch, maybe we shouldn't make the interfaces public and keep them private? https://github.com/RoboStack/ros-jazzy/blob/main/patch/ros-jazzy-rosidl-generator-py.osx.patch

xref on a related issue on the change that we did to that patch in kilted: RoboStack/ros-kilted#76 .

[traversaro]

Quite likely because of our patch, maybe we shouldn't make the interfaces public and keep them private? https://github.com/RoboStack/ros-jazzy/blob/main/patch/ros-jazzy-rosidl-generator-py.osx.patch

Yes, I think marking the include directory as private should work fine. That will fail if somebody actually use those targets, but I think in that case you need anyhow to pass along the Python targets to link, so I think that the least bad situation.