Replace licensecheck with scancode-toolkit for dependency license checks

[w-martin]

Replaces licensecheck with scancode-toolkit for the verify-licences task.

Why

Two reasons:

1. Real license detection. licensecheck only reads declared PyPI metadata — the License field and trove classifiers — which is self-reported and frequently absent, stale, or too vague to act on. scancode matches the actual license text a package ships against a database of ~2500 known licenses, the same approach GitHub's own license detector takes.
2. Dependency pinning. licensecheck's Python 3.11-compatible release (2025.1.0) pulls in fhconfparser, which pins attrs<24. The release that fixes it (2026.0.8) requires Python >=3.12, above this project's 3.11 floor.

What changed

• licensecheck and [tool.licensecheck] removed; scancode-toolkit-mini>=32.5.0 added to the dev group.
• New scripts/check_licenses.py holds the policy; tasks.py's verify_licences calls it. Still a single uv run inv verify-licences, still part of uv run inv all.
• scripts/* added to the coverage omit list, alongside the existing tasks.py / benchmarks/* / examples/* entries. No new lint suppressions — scripts/__init__.py handles INP001, and printing stays in tasks.py, which already ignores T201.
• DEVELOPING.md updated (task usage, the policy, and the libmagic prerequisite).

How the check works

The check now runs over every distribution installed in the dev environment (136 packages), not just the 25 direct dev dependencies. licensecheck also resolved transitively — that's why its ignore list contained mypy-extensions, a transitive dep of mypy.

Per distribution:

1. Text-match every license file bundled in its .dist-info (LICENSE, COPYING, NOTICE, …).
2. Only when a package bundles no license file at all, fall back to declared metadata (License-Expression / License / License :: classifiers), resolved via scancode's SPDX symbol table and then its text matcher.

A package passes when at least one detected license is in an allowed scancode category: Permissive, Public Domain, or Copyleft Limited (LGPL/MPL/EPL). This mirrors licensecheck's effective policy here — I read its license_matrix/matrix.csv MIT row rather than assuming: for an MIT project it passed public domain, permissive, LGPL, MPL and EPL, and failed GPL, AGPL, EUPL, proprietary, unknown and no-license.

Why "at least one allowed" rather than "no disallowed": packages routinely concatenate the license texts of their vendored dependencies into their own LICENSE file. pandas's bundled LICENSE text-matches gpl-1.0-plus among a dozen others, because it ships the notices of everything it vendors — but the grant pandas makes to us is BSD-3-Clause. Failing on any disallowed match would reject pandas, which is plainly wrong. Requiring a positive allowed match still fails a package that offers only a disallowed license, or none at all. This is the one genuine behavioural trade-off of moving from metadata to text detection, and it's documented in both the module docstring and DEVELOPING.md.

What I verified

ARM install (the documented blocker is stale, but there's a different one). The claim that pip install scancode-toolkit fails on Apple Silicon because py-ahocorasick and intbitset lack arm64 wheels is no longer true — both now publish macosx_11_0_arm64 wheels and installed cleanly in a throwaway venv with no build step. The real arm64 problem is elsewhere: typecode-libmagic has never shipped a working arm64 build. Its macosx_11_0_arm64 wheel is a 1,285-byte placeholder (the manylinux_2_28_aarch64 one is 1,912 bytes), and the latest release (5.39.210531, 2021) dropped arm64 entirely. Without a system libmagic, importing scancode.api — or even licensedcode — dies with NoMagicLibError.

brew install libmagic fixes it; scancode's typecode explicitly globs /opt/homebrew/Cellar/libmagic/*/lib/libmagic.dylib. This is a new local prerequisite and the main thing worth a second opinion — it's documented in DEVELOPING.md's Prerequisites and the Verify Licenses section. CI doesn't run this check (publish.yml runs format-check, lint, test only), so CI is unaffected.

Python version support: scancode-toolkit 32.5.0 declares requires-python >=3.10 and publishes cp310–cp314 wheels, so it covers this project's full 3.11–3.14 range. Verified against PyPI metadata, and exercised on both 3.11 (scratch venv) and 3.14 (the repo's .python-version).

-mini rather than the full package: uv add scancode-toolkit fails to resolve on Apple Silicon — extractcode-7z==16.5.210531 has wheels only for manylinux1_x86_64, macosx_10_14_x86_64 and win_amd64. scancode-toolkit-mini drops the prebuilt binary plugins entirely and requires only typecode, which is all that license detection needs.

Deliberate-failure test. Temporarily added pylint (GPL-2.0) to the dev group. The check failed as intended, and detected the GPL from pylint's bundled LICENSE text, not its metadata:

Checking dependency licenses...
Disallowed or undetectable licenses in 1 of 142 packages:
  pylint: detected gpl-2.0 (categories: Copyleft) -- from licence files: LICENSE

EXIT=1. pylint was then removed and pyproject.toml / uv.lock diffed byte-identical against their pre-test backups.

Full suite: uv run inv all passes — 301 Python tests, 185 Rust tests, 100% line/branch coverage threshold, and 136 packages license-checked. (One unrelated flake on the first run: tests/integration/test_regression.py hit sqlite3.OperationalError: database is locked on mypy's cache under xdist. It passes in isolation and on rerun; that test has prior history, see #65.)

A tool bug worth knowing: scancode's own dependency-manifest route is not usable here. packagedcode.pypi.PythonInstalledWheelMetadataFile — its dedicated installed-wheel handler — crashes on pandas 3.0.5's METADATA with AttributeError: 'str' object has no attribute 'get' (pypi.py:1663), and it returns declared_license_expression=None for packages using the modern License-Expression core-metadata field (invoke, maturin, mypy). Hence the hand-rolled scan over .dist-info.

Cross-cutting: this may unblock #69

Removing licensecheck also removes its transitive fhconfparser → attrs<24 pin. That was the constraint blocking pygls>=2 for the LSP server in #69. attrs now resolves to 26.1.0 in uv.lock. I have not touched that branch or its files — flagging it only so it can be picked up separately.

Less certain / worth review

• The libmagic prerequisite is the biggest judgement call. It's a non-Python system dependency for local dev on Apple Silicon. It's one brew install and CI is unaffected, but it is genuinely new friction and arguably trades one kind of churn for another.
• Install size. scancode ships a 413 MB prebuilt license index plus ~2500 license texts and ~36k rules; the dev venv goes from roughly 700 MB to ~1.3 GB. The check itself is fast (~20 s wall, most of it loading that index) but peaks around 3.5 GB RSS.
• ply added to IGNORED_PACKAGES. It declares a bare "BSD" — not an SPDX identifier — with no trove classifier and no bundled license file, so nothing in the installed artifact pins down the variant. It's only in the tree because scancode depends on it. licensecheck passed it via substring matching on "BSD". Upstream ply is BSD-3-Clause.
• The other three ignores were kept for continuity but may now be dead config. mypy-extensions and pyrefly both pass on their own merits under the new checker (both ship a detectable LICENSE), and typedframes would pass via the metadata fallback. Happy to trim them if you'd prefer.
• examples/sql_connectors/*/uv.lock still reference licensecheck>=2025.1.0 in their recorded copies of typedframes' dev group. Those sub-projects are outside the uv workspace (exclude = ["examples/*/*"]), aren't part of inv all, and were already stale before this change (they record complexipy>=5.2.0 against the root's current >=6.2.0). Regenerating all ten would mean resolving pyspark/databricks/snowflake trees, so I left them — say the word if you'd rather they were refreshed.

https://claude.ai/code/session_013nRj6HvRbwkzAoj3QPbiSj