Skip to content

Add experimental RAPIDS cuDF backend support - #71

Open
w-martin wants to merge 1 commit into
mainfrom
experimental-cudf-backend-support
Open

Add experimental RAPIDS cuDF backend support#71
w-martin wants to merge 1 commit into
mainfrom
experimental-cudf-backend-support

Conversation

@w-martin

Copy link
Copy Markdown
Owner

Adds cudf as a recognized DataFrame module, aiming for real parity with the existing pandas/polars tracked-operation matrix rather than a read-call-only slice.

Verification status — read this first

cuDF could not be executed while building this. It requires an NVIDIA GPU with a CUDA runtime and ships Linux-only wheels; uv pip install --dry-run cudf-cu12 fails on this machine (nvidia-smi command not found, Darwin/arm64). So no claim below is backed by running real cuDF.

Instead, every cuDF API shape was checked against cuDF's own published source at rapidsai/cudf@branch-25.10. That matters more than usual here, because cuDF's "drop-in pandas replacement" billing is true for column access and false for the reader surface — assuming parity would have produced exactly the confident-sounding wrong claims this repo's norms warn about.

Claim How it was verified
cuDF exports exactly read_avro/csv/feather/hdf/json/orc/parquet/text and no read_sql*/read_excel/read_html/read_clipboard/scan_* Source-verifiedcudf/__init__.py's __all__
read_csv takes usecols= and dtype=; read_parquet/read_orc/read_avro take columns=; read_json takes dtype= Source-verified — signatures in cudf/io/{csv,parquet,orc,avro,json}.py
read_feather/read_hdf accept columns= Weakest claim here. Source-verified only that both forward *args/**kwargs to pyarrow.feather.read_table / pd.read_hdf, which do take columns=. Not directly verified end-to-end
read_text returns Series, not DataFrame (so excluded) Source-verified — return annotation in cudf/io/text.py
rename/drop/assign/pop/insert/merge carry pandas' signatures Source-verifiedcudf/core/{dataframe,indexed_frame}.py
cuDF has no DataFrame.filter, select, sort, or with_columns Source-verified + docs-cross-checked (absent from source; absent from the RAPIDS DataFrame API page)
12 of the 14 row-passthrough methods exist on cuDF (filter/sort are the two that don't) Source-verified
No lazy/eager split — no collect()/compute()/lazy on cudf.DataFrame Source-verified (grep across dataframe.py/indexed_frame.py/frame.py)
Conventional import is bare import cudf, no short alias Docs-verified — cuDF's own README and the "10 minutes to cuDF" notebook
The checker behaves as described on cuDF-shaped source Executed locally — this part needs no GPU, since the checker is pure AST analysis. 9 new Rust tests + the example run through the real CLI

What's covered

Most of the matrix needed no cuDF-specific code: the structural-op and row-passthrough dispatch keys on method name, not on the receiving module, and Annotated[<anything>.DataFrame, Schema] recognition was already module-agnostic. Verified working end-to-end:

  • Read-call inference: usecols=/columns=/dtype= across all seven cuDF readers
  • Structural ops: rename, drop, assign, pop, insert, del, subscript assignment, merge, concat
  • All 12 row-passthrough methods cuDF implements
  • Annotated[cudf.DataFrame, Schema] on variables, parameters and return types; bare -> cudf.DataFrame registers an open schema rather than a false positive

The one real code change is a narrowing. LOAD_FUNCTIONS is the union across every supported library, so adding cudf to LOAD_MODULES alone would have made cudf.read_sql(...) and cudf.scan_csv(...) resolve to column sets — calls that cannot exist. CUDF_LOAD_FUNCTIONS restricts cuDF to its seven real readers; pandas and polars keep the full union, with a regression test pinning that.

No new pl.col()-style expression recognition was needed: cuDF's column-access idiom is pandas-identical, verified rather than assumed.

Deliberately not done

  • No CudfFrame wrapper class. PandasFrame/PolarsFrame are deprecated per AGENTS.md; new backends use the Annotated[...] form only. Zero Python-side changes, so the 100% branch-coverage gate is untouched.
  • cudf.pandas accelerator mode needs nothing. Source written for it still says import pandas as pd and is already covered by the pandas path.
  • dask-cudf is out of scope — it has a .compute() materialization step that plain cuDF does not.

Notes for review

  • examples/features/annotated_cudf_example.py is written, checked, and never executed. Running it needs a GPU. It catches 5 intentional errors via the real CLI, and its inline comments were corrected against actual checker output rather than written from expectation — an earlier draft claimed merge tracking on load_orders().merge(load_customers(), ...), which the checker does not resolve (both operands must be tracked variables); the example now says so explicitly.
  • No lint-ignore rules added and no bandit checks skipped. ty already ignores unresolved imports under examples/**/*.py, so import cudf needed no new exemption.
  • The README comparison row is marked ⚠️ cuDF (experimental), not — the row is "DuckDB, cuDF, etc." and only the cuDF half is addressed.
  • A pre-existing unused import: crate::config::load_linter_config warning in the Rust test profile is present on main too and was left alone.

Checks

uv run inv all passes (ruff format/check, ty, bandit, complexipy, cargo fmt, clippy -D warnings, 301 Python tests at 100% branch coverage, 181 + 13 Rust tests, licensecheck). On the very first run in a cold worktree, test_should_accept_polarsframe_with_type_argument failed once under pytest-xdist and passed on every subsequent run and in isolation; this PR makes no Python changes, so it appears to be cold-mypy-cache flakiness rather than a regression. Flagging rather than silently re-running.

https://claude.ai/code/session_013nRj6HvRbwkzAoj3QPbiSj

Recognize `cudf` as a DataFrame module so column sets are inferred from cuDF
read calls, and document how far that coverage actually goes.

cuDF uses pandas' string-subscript column access, so the tracked structural
operations (rename/drop/assign/pop/insert/del/subscript assignment/merge/concat),
the row-passthrough methods, and `Annotated[cudf.DataFrame, Schema]` contracts all
work through the existing pandas handling with no cuDF-specific code — those paths
dispatch on method name, not on the receiving module.

The reader surface does need narrowing. `LOAD_FUNCTIONS` is the union across every
supported library, so adding `cudf` to `LOAD_MODULES` alone would have resolved
`cudf.read_sql(...)` and `cudf.scan_csv(...)` — calls that cannot exist, since
cuDF exports neither. `CUDF_LOAD_FUNCTIONS` restricts cuDF to the seven readers it
actually exports; pandas and polars keep the full union unchanged.

Every cuDF API shape asserted here was verified against cuDF's own published
source (`cudf/__init__.py`'s `__all__` and `cudf/core/{dataframe,indexed_frame,
frame}.py`), not inferred from cuDF's "drop-in pandas replacement" billing — which
holds for column access but not for the reader surface, and not for
`DataFrame.filter`, which cuDF does not implement.

Claude-Session: https://claude.ai/code/session_013nRj6HvRbwkzAoj3QPbiSj
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant