Add experimental RAPIDS cuDF backend support - #71
Open
w-martin wants to merge 1 commit into
Open
Conversation
Recognize `cudf` as a DataFrame module so column sets are inferred from cuDF
read calls, and document how far that coverage actually goes.
cuDF uses pandas' string-subscript column access, so the tracked structural
operations (rename/drop/assign/pop/insert/del/subscript assignment/merge/concat),
the row-passthrough methods, and `Annotated[cudf.DataFrame, Schema]` contracts all
work through the existing pandas handling with no cuDF-specific code — those paths
dispatch on method name, not on the receiving module.
The reader surface does need narrowing. `LOAD_FUNCTIONS` is the union across every
supported library, so adding `cudf` to `LOAD_MODULES` alone would have resolved
`cudf.read_sql(...)` and `cudf.scan_csv(...)` — calls that cannot exist, since
cuDF exports neither. `CUDF_LOAD_FUNCTIONS` restricts cuDF to the seven readers it
actually exports; pandas and polars keep the full union unchanged.
Every cuDF API shape asserted here was verified against cuDF's own published
source (`cudf/__init__.py`'s `__all__` and `cudf/core/{dataframe,indexed_frame,
frame}.py`), not inferred from cuDF's "drop-in pandas replacement" billing — which
holds for column access but not for the reader surface, and not for
`DataFrame.filter`, which cuDF does not implement.
Claude-Session: https://claude.ai/code/session_013nRj6HvRbwkzAoj3QPbiSj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
cudfas a recognized DataFrame module, aiming for real parity with the existing pandas/polars tracked-operation matrix rather than a read-call-only slice.Verification status — read this first
cuDF could not be executed while building this. It requires an NVIDIA GPU with a CUDA runtime and ships Linux-only wheels;
uv pip install --dry-run cudf-cu12fails on this machine (nvidia-smi command not found, Darwin/arm64). So no claim below is backed by running real cuDF.Instead, every cuDF API shape was checked against cuDF's own published source at
rapidsai/cudf@branch-25.10. That matters more than usual here, because cuDF's "drop-in pandas replacement" billing is true for column access and false for the reader surface — assuming parity would have produced exactly the confident-sounding wrong claims this repo's norms warn about.read_avro/csv/feather/hdf/json/orc/parquet/textand noread_sql*/read_excel/read_html/read_clipboard/scan_*cudf/__init__.py's__all__read_csvtakesusecols=anddtype=;read_parquet/read_orc/read_avrotakecolumns=;read_jsontakesdtype=cudf/io/{csv,parquet,orc,avro,json}.pyread_feather/read_hdfacceptcolumns=*args/**kwargstopyarrow.feather.read_table/pd.read_hdf, which do takecolumns=. Not directly verified end-to-endread_textreturnsSeries, notDataFrame(so excluded)cudf/io/text.pyrename/drop/assign/pop/insert/mergecarry pandas' signaturescudf/core/{dataframe,indexed_frame}.pyDataFrame.filter,select,sort, orwith_columnsfilter/sortare the two that don't)collect()/compute()/lazyoncudf.DataFramedataframe.py/indexed_frame.py/frame.py)import cudf, no short aliasWhat's covered
Most of the matrix needed no cuDF-specific code: the structural-op and row-passthrough dispatch keys on method name, not on the receiving module, and
Annotated[<anything>.DataFrame, Schema]recognition was already module-agnostic. Verified working end-to-end:usecols=/columns=/dtype=across all seven cuDF readersrename,drop,assign,pop,insert,del, subscript assignment,merge,concatAnnotated[cudf.DataFrame, Schema]on variables, parameters and return types; bare-> cudf.DataFrameregisters an open schema rather than a false positiveThe one real code change is a narrowing.
LOAD_FUNCTIONSis the union across every supported library, so addingcudftoLOAD_MODULESalone would have madecudf.read_sql(...)andcudf.scan_csv(...)resolve to column sets — calls that cannot exist.CUDF_LOAD_FUNCTIONSrestricts cuDF to its seven real readers; pandas and polars keep the full union, with a regression test pinning that.No new
pl.col()-style expression recognition was needed: cuDF's column-access idiom is pandas-identical, verified rather than assumed.Deliberately not done
CudfFramewrapper class.PandasFrame/PolarsFrameare deprecated perAGENTS.md; new backends use theAnnotated[...]form only. Zero Python-side changes, so the 100% branch-coverage gate is untouched.cudf.pandasaccelerator mode needs nothing. Source written for it still saysimport pandas as pdand is already covered by the pandas path.dask-cudfis out of scope — it has a.compute()materialization step that plain cuDF does not.Notes for review
examples/features/annotated_cudf_example.pyis written, checked, and never executed. Running it needs a GPU. It catches 5 intentional errors via the real CLI, and its inline comments were corrected against actual checker output rather than written from expectation — an earlier draft claimed merge tracking onload_orders().merge(load_customers(), ...), which the checker does not resolve (both operands must be tracked variables); the example now says so explicitly.tyalready ignores unresolved imports underexamples/**/*.py, soimport cudfneeded no new exemption.⚠️ cuDF (experimental), not✅— the row is "DuckDB, cuDF, etc." and only the cuDF half is addressed.unused import: crate::config::load_linter_configwarning in the Rust test profile is present onmaintoo and was left alone.Checks
uv run inv allpasses (ruff format/check, ty, bandit, complexipy, cargo fmt, clippy-D warnings, 301 Python tests at 100% branch coverage, 181 + 13 Rust tests, licensecheck). On the very first run in a cold worktree,test_should_accept_polarsframe_with_type_argumentfailed once underpytest-xdistand passed on every subsequent run and in isolation; this PR makes no Python changes, so it appears to be cold-mypy-cache flakiness rather than a regression. Flagging rather than silently re-running.https://claude.ai/code/session_013nRj6HvRbwkzAoj3QPbiSj