Detect DataFrames inside class methods and control flow; trace external packages by default - #58
Merged
Conversation
…al packages by default The main lint pass never recursed into class bodies (only BaseSchema/ORM subclasses got their methods visited) or into if/for/while/with/try bodies, so a load call or a Protocol-typed method dispatch anywhere inside either was invisible to schema coverage. Both are now visited, which also makes Protocol/structural-typing call sites resolve for free (function resolution is already name-based, not type-based). Also fixes the column-contract heuristic always tainting a method's literal first parameter (self/cls) instead of skipping to the real DataFrame parameter. External package tracing (trace_external_packages) is now on by default: when first-party code calls an unresolved external function and uses the result like a DataFrame (subscript access or a pandas/polars-shaped method call), that package is auto-traced. trace_external_packages remains a forced, uncapped override; excluded_external_packages is the new opt-out. Auto-discovered candidates are capped at 20 packages per project.
4 tasks
Patch bump to 0.4.1 (pyproject.toml, __init__.py, Cargo.toml) alongside the fixes and feature in this PR. Bumped dev-dependency floors (ruff, ty, mypy, pyarrow) and re-locked every examples/comparisons/* subproject against current PyPI releases: great_expectations 1.19.1 -> 1.20.0, static-frame 5.0.0 -> 5.1.1 (everything else already at latest). strictly_typed_pandas stays pinned to pandas<=2.2.3 per its own upstream constraint. Re-ran benchmarks/benchmark_checkers.py --update-readme against the bumped toolchain, and bumped GE_COMMIT (1.9.3 -> 1.20.0) for the large-corpus timing benchmark per the script's own "bump deliberately on a new release" convention. Updated the README Feature Matrix's "Version tested" row and tool-description version parentheticals to match.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Investigated a report that typedframes found "no dataframes" in a real repo (catboost/pymc3 alongside pandas, DataFrames collected via an internal SQL package, dataframes passed through Protocol-typed instance methods). Root causes, all fixed here:
Stmt::ClassDefonly recursed into a class's body forBaseSchema/ORM subclasses — any ordinary class's methods (the overwhelming majority of real code, including every Protocol implementation) were completely invisible to the checker, no control flow required. Confirmed with a one-line repro.if/for/while/with/tryfell into a bare_ => {}— a load call wrapped in awith connection:ortry/except(virtually all real DB code) was invisible.self/clsinstead of the actual DataFrame parameter.Fixing class-body recursion turned out to be sufficient for Protocol/structural-typing support too — no Protocol-specific code needed, since function resolution here is already name-based rather than type-based.
Also makes external package tracing default-on: when first-party code calls an unresolved function from an installed package and uses the result like a DataFrame (subscript access, or a pandas/polars-shaped method call), that package is now auto-traced — no config needed for the common "internal SQL wrapper package" case.
trace_external_packagesremains available as a forced, uncapped override;excluded_external_packagesis the new opt-out. Auto-discovered candidates are capped at 20 packages/project, and the discovery step is proven not to sweep in pandas/polars' own (often huge) internals, since their calls are always recognized directly and never reach the "unresolved" fallback.Also bundled in this PR (version bump + comparison/benchmark refresh, folded in from a separate branch since it belongs alongside the change that motivates it):
pyproject.toml,src/typedframes/__init__.py,rust/Cargo.toml).ruff0.16.2→0.16.3,ty0.0.69→0.0.72,mypy2.3.0→2.3.1,pyarrow25.0.0→25.0.1) and re-locked everyexamples/comparisons/*subproject against current PyPI releases:great-expectations1.19.1→1.20.0,static-frame5.0.0→5.1.1 — everything else in the comparison set was already at latest.strictly_typed_pandasstays pinned topandas<=2.2.3per that package's own upstream constraint.benchmarks/benchmark_checkers.py --update-readmeagainst the bumped toolchain (including a fresh Great Expectations 1.20.0 clone for the large-corpus benchmark, bumpingGE_COMMITper the script's own "bump deliberately on a new release" convention), and updated the README Feature Matrix's "Version tested" row and tool-description version parentheticals to match.Test plan
cargo test(162 unit + 13 integration tests, including new coverage for every fix above)uv run inv test— 301 Python tests, 100% branch coverageuv run inv lint— ruff/ty/bandit/complexipy/clippy/fmt all cleanif/with/try, inside a plain class method, through a Protocol-typed call site, and via an unconfigured external package) — each now resolves correctly.venv: ~11.2ms in all cases, no measurable regression when nothing external is actually useduv lockclean in all 10examples/comparisons/*subprojects, no resolution conflicts