Skip to content

Add experimental dask.dataframe backend support - #72

Open
w-martin wants to merge 1 commit into
mainfrom
experimental-dask-backend-support
Open

Add experimental dask.dataframe backend support#72
w-martin wants to merge 1 commit into
mainfrom
experimental-dask-backend-support

Conversation

@w-martin

Copy link
Copy Markdown
Owner

Adds dask.dataframe as an experimental third backend, aiming for parity with the existing tracked-operation matrix rather than a read-call-only slice.

What's covered

Read calls. dd.read_csv(usecols=...), dd.read_parquet(columns=...), dd.read_json, dd.read_hdf, dd.read_orc, and the read_sql/read_sql_query/read_sql_table family (including SELECT-list inference for the SQL-shaped ones). No new LOAD_FUNCTIONS entries were needed — dask is a deliberate near-drop-in for pandas, so every read function it implements already appears under its pandas spelling with the same usecols=/columns= kwargs.

Both import spellings. import dask.dataframe as dd and a plain import dask.dataframe (dask.dataframe.read_csv(...)). The second works because the call receiver is now flattened by a new ast_extract::dotted_module_path instead of being destructured as a bare Expr::Name. A bare name flattens to itself, so every pandas/polars/Schema.from_pandas path is byte-for-byte unchanged.

Structural operations. drop, rename (mapping and str.lower/str.upper fold), assign, column-list subscript (df[["a","b"]]), subscript assignment, del, pop, left.merge(right, ...), dd.concat([...]). These needed no code changes — they were already keyed on method name plus a tracked receiver, so they are backend-agnostic. The new tests pin that down rather than leaving it to coincidence.

Row-passthrough. query, head, tail, sample, sort_values, reset_index, nlargest, nsmallest, fillna, dropna, ffill, bfill — every member of the existing list that dask actually implements.

Materialization. compute, persist, and repartition added to ROW_PASSTHROUGH_METHODS, so a lazy dask pipeline's column set survives the lazy→eager boundary instead of being dropped there.

dd.from_pandas(pdf, npartitions=n). New FRAME_WRAP_FUNCTIONS constant: carries the source frame's schema over unchanged, and counts as a DataFrame origin either way (emitting untracked-dataframe when the source itself has no known columns), which is the invariant every other LOAD_MODULES call already upholds.

Annotated[dd.DataFrame, Schema]. Already worked — the annotation parser keys on the DataFrame attribute name, not the module it hangs off. Pinned with a cross-file contract test so a future narrowing of that check can't silently drop dask.

Incidental fixes outside dask

Two things surfaced while verifying parity. Both are called out here rather than slipped in quietly:

  1. polars .collect() had no handling at all. A pl.scan_csv(...) column set was lost the moment it was collected, so every post-.collect() access went unchecked. collect is added alongside dask's .compute() — they are the same shape, and building the mechanism for one without the other made no sense. There's a regression test for it.

  2. Two inaccuracies in docs/method-matrix.md. The schema-modifying table listed pd.merge(left, right, …) as tracked (it isn't — only the left.merge(right, …) instance form is), and the untracked table listed df.merge(other, …) as untracked (it is tracked, as a union of both schemas). Both corrected. Verified with:

    a = pd.read_csv("a.csv", usecols=["id", "x"])
    b = pd.read_csv("b.csv", usecols=["id", "y"])
    print(pd.merge(a, b, on="id")["nope"])   # no diagnostic
    print(a.merge(b, on="id")["nope"])       # unknown-column in {id, x, y}

pl.from_pandas(pdf) also now resolves via the same FRAME_WRAP_FUNCTIONS path (it previously did nothing and left pl as an unresolved external-package candidate). Note this is a small behaviour change for existing polars users: pl.from_pandas(x) with an untracked x now counts as a DataFrame origin and emits untracked-dataframe, which can move a --coverage-fail-under number.

Deliberately deferred

  • dd.from_dict({...}), from_delayed, from_array, from_map. from_dict takes a literal rather than a tracked frame — the same shape as pd.DataFrame({...}), which this checker also leaves untracked, so tracking it here would be an inconsistency, not parity. The rest produce columns that only exist once the graph runs.
  • Module-level dd.merge(left, right). Not tracked, exactly matching pd.merge today. Making it work is a separate change that would apply to all three backends; doing it only for dask would be worse than not doing it.
  • Dask names in RESERVED_METHODS (compute, persist, npartitions, categorize, …). Adding them would start flagging schema columns with those names for existing pandas/polars users who don't use dask, since that check is not backend-scoped. Net-negative noise, so left alone.
  • filter, sort, insert, select. Confirmed absent from dask.dataframe (they are pandas/polars-only spellings). They stay in the tables because a dask user never writes them, and they only fire on a tracked receiver.
  • Multi-call chaining in one expression (df.query(...).sort_values(...)) is still not followed — the receiver must be a named variable. This is pre-existing and backend-independent (pandas behaves identically); now documented in the method matrix rather than silently assumed.

How this was tested

Real dask execution, not docs. Every column-set claim in the new code comments and tests was verified by running the operation against dask 2026.8.0 and reading back .columns. That covers: read_csv(usecols=), read_parquet(columns=), all twelve row-passthrough methods, drop/rename/rename(str.upper)/assign/subscript slice/subscript assignment/del/pop, compute/persist/repartition, concat, instance and module merge, and from_pandas. It also confirmed that filter/sort/select/insert genuinely do not exist on a dask DataFrame, and that dd.read_csv(dtype={...}) does not narrow columns (same as pandas — pre-existing behaviour, unchanged here).

Docs-only verification: none of the behavioural claims. The only non-executed checks were the comparison-matrix cells for other tools: narwhals' Dask support was read from its docs (lazy-only backend list), Great Expectations' absence of Dask support from its data-sources page. Pandera's Dask support was verified by execution (ddf.pandera.add_schema(...) after importing pandera.accessors.dask_accessor).

Suites. uv run inv all green — ruff, ty, bandit, complexipy, cargo fmt --check, cargo clippy -- -D warnings, 301 Python tests at 100% branch coverage, licensecheck (dask is BSD-3-Clause). cargo test green: 201 Rust tests, 15 of them new.

No lint-ignore rules were added and no bandit checks skipped.

Also in this PR

  • examples/features/annotated_dask_example.py (four real caught errors + one warning), fitted alongside the existing annotated_pandas_example.py/annotated_polars_example.py rather than a new examples/backends/ tree. Its README entry notes it runs mypy without --strict: dask ships py.typed but its annotations are incomplete (dd.read_csv returns Any, dd.from_pandas is untyped), so --strict reports against dask's own API, not this file.
  • README: a "Use With Dask (Experimental)" quick-start section, a Dask row in the comparison matrix marked ⚠️ Experimental for typedframes, plus the operations/examples lists.
  • docs/usage.md and docs/method-matrix.md.
  • dask[dataframe] added to the dev dependency group only — the shipped package gains no runtime dependency, and the checker never imports dask (it is a pure AST pass).

https://claude.ai/code/session_013nRj6HvRbwkzAoj3QPbiSj

Recognizes `dask.dataframe` as a third backend alongside pandas and polars,
aiming for parity with the existing tracked-operation matrix rather than a
read-call-only slice.

- `LOAD_MODULES` gains `dd` and `dask.dataframe`. The dotted spelling works
  because the call receiver is now flattened by a new
  `ast_extract::dotted_module_path` instead of being destructured as a bare
  `Expr::Name`; a bare name flattens to itself, so every pd/pl path is
  unchanged.
- `ROW_PASSTHROUGH_METHODS` gains `compute`/`persist`/`repartition` (dask) and
  `collect` (polars), so a lazy frame's column set survives materialization.
  polars' `.collect()` had no handling at all before this.
- New `FRAME_WRAP_FUNCTIONS` handles `dd.from_pandas(pdf, npartitions=n)` and
  `pl.from_pandas(pdf)`, carrying the source frame's schema over and counting
  the result as a DataFrame origin either way.

No new `LOAD_FUNCTIONS` entries were needed: every read function dask
implements already appears under its pandas spelling with the same
`usecols=`/`columns=` kwargs. The structural operations (drop, rename, assign,
subscript slice, subscript assignment, del, pop, instance-method merge,
module-level concat) and `Annotated[dd.DataFrame, Schema]` were already keyed
on method/attribute name rather than backend, so they work unchanged — the new
tests pin that down.

Every column-set claim was verified by executing the operation against dask
2026.8.0 and reading back `.columns`, not from docs alone. dask is added as a
dev dependency for that purpose only; the shipped package gains no runtime
dependency, and the checker never imports dask.

Also corrects two pre-existing inaccuracies in docs/method-matrix.md found
while verifying: `pd.merge(left, right)` is listed as schema-modifying but is
not tracked, while `left.merge(right)` is listed as untracked but is.

Claude-Session: https://claude.ai/code/session_013nRj6HvRbwkzAoj3QPbiSj
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant