Port bindings to pybind11?

[musicinmybrain]

It was suggested by @hobu in #220 (comment) that the libspatialindex bindings should ideally be ported from ctypes to pybind11.

This could reduce the need for manual alignment of function prototypes and manual type hinting, and it could hopefully prevent future subtle bugs due to prototype mismatches (cf. #220).

It would, however, require new machinery to build and publish binary wheels for popular platforms.

I am filing this issue to provide a forum for discussion and investigation, and to track eventual progress, on this topic.

[musicinmybrain]

Users seem to expect binary wheels, not just sdist-only releases, for PyPI packages with compiled code. I’ve seen a few options for building binary wheels for assorted platforms. I haven’t deployed any of them myself. It seems like cibuildwheel is a popular and relatively polished option. As is typical in Python packaging, there are almost as many workflows, processes, and combinations of tools as there are projects.

Integrating something like this into the release process will probably take some effort and decision-making by the project owners.

[musicinmybrain]

Comparison with selected other tools for wrapping C APIs:

• cffi is pleasantly simple and lightweight, but manual definition of function prototypes is still required, and mismatches are still possible. It is only suitable for wrapping C APIs; in this case, that would be fine. It’s mostly aimed at low-level bindings, akin to the current ctypes binding, and it’s usually possible to implement these with the interface definitions and little or no additional C code.
• Cython offers a powerful general-purpose almost-Python language that is transpiled to C or C++. It can be particularly useful for high-level bindings that need to implement significant functionality in compiled code for performance reasons, especially when a goal is to minimize the amount of C or C++ code needed. Unlike cffi, it can wrap C++. Manual definition of function prototypes is still required, and mismatches are still possible. Updates to Cython aren’t always entirely backwards-compatible, and its language isn’t formally specified as C or C++ are.
• Boost.Python is pretty similar to pybind11 in concept, but it’s a heavier-weight dependency, is not header-only, and seems to have a much smaller user base today than other options in this list.
• SWIG is still around, but—without going into detail—I think newer tools provide a better experience and produce better bindings. One mostly only sees it used for projects that have been around since SWIG and ctypes were the only mature choices and also have huge APIs.

For this particular project, in a quick up-front comparison, pybind11 still looks like a good choice.

[adamjstewart]

Does pybind11 have mypy-compatible type hints? That was another problem I encountered with the current ctypes code.

[musicinmybrain]

Does pybind11 have mypy-compatible type hints? That was another problem I encountered with the current ctypes code.

It’s an important question. I can’t answer from experience. I’ve use ctypes, Cython, and CFFI in the past, but not pybind11—and all before the widespread use of typing in Python.

A quick search turns up mypy.stubgen for generating draft stubs for arbitrary C extensions, with the expectation that they will be edited and maintained by hand. There is also a third-party pybind11-stubgen (discussion).

I don’t see an obvious way to avoid some manual type hinting as @hobu suggested, but I could easily be missing something.

[musicinmybrain]

An open question is, if type hints indeed need to be manually maintained, is it worth the switch, and particularly, worth introducing compiled code to the package?

Using pybind11 would still have the advantage that most API mismatches with libspatialite should produce compiler warnings or errors. (It might still be possible to have silent mismatches in some by-value parameters—I’m not sure.)

On the other hand, publishing binary wheels is a lot more annoying.

[hobu]

We are already in the business of publishing binary wheels. Doing this with ctypes+libspatialindex is just as annoying as it would be with pybind11+libspatialindex.

I would have expected some type hinting stuff coming out of pybind11 because the point of it is to have type checking when compiling wrappers 😄, but I can understand if it hasn't matured that far at this time.

The case for pybind11 is indeed to do away with the manually maintained and matched API calls of ctypes <-> libspatialindex to avoid the discrepancies of things like @musicinmybrain found in #222.

[adamjstewart]

Yes, manual type hints are fine. It's just that with ctypes, it didn't seem to be possible to use manual type hints. Mypy forbids assignment of methods to override other instance methods. Also, the typeshed hints for ctypes seem to have the wrong number of arguments, so there were all kinds of issues. As long as pybind11 type hints are slightly simpler, I'll be happy with the change!

[musicinmybrain]

We are already in the business of publishing binary wheels. Doing this with ctypes+libspatialindex is just as annoying as it would be with pybind11+libspatialindex.

That’s right, I forgot about the bundled libspatialite in the PyPI wheels. (Since Fedora’s RPM packaging uses the system copy of the library, it applies a small downstream patch that makes its python-Rtree package pure-Python/noarch.)

Then the conversion is probably worth attempting even if the gain is somewhat less.

If pybind11 isn’t offering automatic type hinting, it might mean that cffi could also be a good solution, since when used in “API mode” it still includes the library’s C header and explicitly attempts to check for function signature mismatches.

It might just come down to which tool’s style you like better.

[hobu]

We don't have to migrate. Having done both pybind11 and ctypes, I like pybind11 better nowadays for the reasons I've stated. This codebase has been quite stable over the years, so I do not wish to upset that with a big subsystem change that has the potential for breakage for very little external benefit.

A case for pybind11 is the opportunity to ditch the C API usage of libspatialindex for direct consumption of its C++ API. That has the potential to speed things up and make code simpler and easier to follow.

The plan for now is to make the 1.0.0 release be based on ctypes to include all of the recent activity, and then we will evaluate pybind11 options if you or someone else has time to put in the effort.