Parallel walkers, abi3 wheels, Python 3.9 CLI fix, limit/threads options - #1
Merged
Merged
Conversation
…output writes PEP 604 annotations in cli.py made the console script crash at import on Python 3.9 even though the package claims to support it. Add from __future__ import annotations. Exit cleanly on BrokenPipeError (e.g. piping to head) and on KeyboardInterrupt instead of printing an error or a traceback. Write output lines in a single buffered call instead of one print per line, and only pretty-print JSON when stdout is a terminal.
The release workflows only passed an interpreter list to the Linux builds, so Windows wheels were published for a single Python version (cp312 for 0.5.0) and macOS for whatever the runner had (cp314). Everyone else silently fell back to the sdist and needed a Rust toolchain. Build with pyo3 abi3-py39 instead: one wheel per platform covers Python 3.9 and up, including future versions. Also drop panic=abort from the release profile. In a cdylib loaded into a host interpreter it turned any Rust panic into a hard abort of the whole Python process; without it PyO3 converts panics into a catchable PanicException. Bump the trove classifier from Alpha to Beta.
Move walk, find, glob and index onto the ignore crate's parallel walker, the same engine disk_usage already uses, and drop the jwalk dependency. Filtering and metadata reads now run on N worker threads, and metadata comes from the directory read itself where the platform provides it (free on Windows) instead of a second stat per file. On a 442k-file tree on Windows: walk 19.1s -> 0.38s, find by size 17.3s -> 0.31s, find by name 1.9s -> 0.31s. New options: find(limit=N) stops the search as soon as N matches are found (a single-file lookup on the same tree returns in ~1.5ms), and every recursive function takes threads=N to tune walker parallelism, which helps on network drives. Also in this change: - glob starts walking at the pattern's literal directory prefix (src/**/*.py no longer scans the whole tree) - walk returns only matching files when any filter is active instead of interleaving every directory with the matches - index resolves stem collisions deterministically by keeping the lexicographically smallest path - files whose metadata cannot be read are excluded consistently when a size or time filter is active - list_dir results are sorted by name, and skip_hidden honors the Windows hidden attribute everywhere - name and extension filters no longer allocate per file when unset, which speeds up unfiltered disk_usage
README: document limit and threads, the behavior notes (hidden files, creation time on Linux, unreadable metadata, result ordering, index collisions), update the performance section, and drop the downloads badge (it permanently rendered as rate limited). CI now runs the installed console script and python -m pyofiles across every matrix cell, which would have caught the Python 3.9 CLI import crash.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full fix-and-optimize pass covering the issues found in review plus the performance work.
Fixes
from __future__ import annotations; CI now runs the CLI in every matrix cell so this class of bug cannot recur.indexstem collisions are deterministic (smallest path wins);walkwith filters returns only matching files instead of interleaving every directory;list_diris sorted;skip_hiddenhonors the Windows hidden attribute; CLI exits cleanly on| headand Ctrl+C.Performance
walk/find/glob/indexmove onto the ignore crate's parallel walker (same engine asdisk_usagesince 0.5.0), dropping jwalk. Metadata reads run on N threads and come from the directory read itself where the platform provides it.globstarts at the pattern's literal prefix. Name/ext filters no longer allocate when unset.Benchmark on a 442k-file tree (Windows, warm cache):
New API
find(..., limit=N): stop as soon as N matches are found (CLI--limit)threads=Non all recursive functions (CLI--threads), useful on network drivesTesting