Python packaging references

[mdpiper]

The best way to share Python code with others is to package it. Here are two references for Python packaging.

1. The Python Packaging Authority is the standard. They have a user guide with a helpful tutorial. The tutorial files, hosted on GitHub, show the directory structure of a Python package.
2. The pyOpenSci Python Package Guide. This guide is created by Leah Wasser, a very experienced developer and educator, and a friend of CSDMS.

Here are two real-world examples: one from Eric, and one from me. Eric tends to do the right thing. I tend to do the least work possible.

If you'd like help packaging a Python project, please visit us at CSDMS Office Hours!

[samharrison7]

Do you guys ever use Poetry? For pure Python packages it's pretty neat and does away with the need for a separate setup.cfg file - everything goes in pyproject.toml (here's an example). It's scarily easy to build packages with it!

[mdpiper]

We've stuck with setuptools so far, though I'd have to defer to @mcflugen for future plans.

Currently, the only reason we have setup.cfg files is for flake8--it doesn't support configuration through pyproject.toml. We're working away from even this by using .flake8, or just setting up pre-commit rules on a repo.

[BSchilperoort]

does away with the need for a separate setup.cfg file - everything goes in pyproject.toml

this is now standard in Python, across setuptools/hatchling/poetry/etc.

We often use hatch/hatchling as our build system. It tries to use common standards as much as possible to keep it easy to switch to different systems, allows for defining scripts and test matrices.

[hrajagers]

Hi Sam,

Yes, I'm using Poetry as well for two Python-based rapid assessment tools (D-FAST Morphological Impact and D-FAST Bank Erosion) to make sure that all developers (and build/test agents) are using the same environment. Since I'm compiling the tools before distribution using Nuitka, I don't care too much about end users, but for advanced users (semi-developers) that want to run the tools inside Python it's equally useful.

[samharrison7]

I've never come across Nuitka before, that looks pretty useful!

[mcflugen]

@samharrison7 I tried poetry a while ago but decided it was a solution to a problem I didn't have. My current workflow with pip seems to work just fine.

Then when I discovered that poetry uses a random number generator to decide if it's going to succeed when running in CI (it's set to fail 5% of the time), I bailed. Maybe that's not the case anymore, I don't know, but it was (is) an odd decision by their developers.

I've also heard that poetry is slow when run in CI environments but I can neither confirm nor deny that claim.

Another option, which people these days seem to like, is uv. It's developed by Astral (the folks behind ruff), who I am not a fan of, so I'll probably continue to stick with my current workflow.

[samharrison7]

Thanks for the reflection. I've never used Poetry in CI so haven't come across these issues. uv looks interesting, they like to boast about their speed!

[hrajagers]

We are using poetry in a CI pipeline running on TeamCity and I've not encountered such random behaviour. I can't comment on whether it's the fastest or best solution; it's been good (enough?) for us.