Add experimental dask.dataframe backend support - #72
Open
w-martin wants to merge 1 commit into
Open
Conversation
Recognizes `dask.dataframe` as a third backend alongside pandas and polars, aiming for parity with the existing tracked-operation matrix rather than a read-call-only slice. - `LOAD_MODULES` gains `dd` and `dask.dataframe`. The dotted spelling works because the call receiver is now flattened by a new `ast_extract::dotted_module_path` instead of being destructured as a bare `Expr::Name`; a bare name flattens to itself, so every pd/pl path is unchanged. - `ROW_PASSTHROUGH_METHODS` gains `compute`/`persist`/`repartition` (dask) and `collect` (polars), so a lazy frame's column set survives materialization. polars' `.collect()` had no handling at all before this. - New `FRAME_WRAP_FUNCTIONS` handles `dd.from_pandas(pdf, npartitions=n)` and `pl.from_pandas(pdf)`, carrying the source frame's schema over and counting the result as a DataFrame origin either way. No new `LOAD_FUNCTIONS` entries were needed: every read function dask implements already appears under its pandas spelling with the same `usecols=`/`columns=` kwargs. The structural operations (drop, rename, assign, subscript slice, subscript assignment, del, pop, instance-method merge, module-level concat) and `Annotated[dd.DataFrame, Schema]` were already keyed on method/attribute name rather than backend, so they work unchanged — the new tests pin that down. Every column-set claim was verified by executing the operation against dask 2026.8.0 and reading back `.columns`, not from docs alone. dask is added as a dev dependency for that purpose only; the shipped package gains no runtime dependency, and the checker never imports dask. Also corrects two pre-existing inaccuracies in docs/method-matrix.md found while verifying: `pd.merge(left, right)` is listed as schema-modifying but is not tracked, while `left.merge(right)` is listed as untracked but is. Claude-Session: https://claude.ai/code/session_013nRj6HvRbwkzAoj3QPbiSj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
dask.dataframeas an experimental third backend, aiming for parity with the existing tracked-operation matrix rather than a read-call-only slice.What's covered
Read calls.
dd.read_csv(usecols=...),dd.read_parquet(columns=...),dd.read_json,dd.read_hdf,dd.read_orc, and theread_sql/read_sql_query/read_sql_tablefamily (including SELECT-list inference for the SQL-shaped ones). No newLOAD_FUNCTIONSentries were needed — dask is a deliberate near-drop-in for pandas, so every read function it implements already appears under its pandas spelling with the sameusecols=/columns=kwargs.Both import spellings.
import dask.dataframe as ddand a plainimport dask.dataframe(dask.dataframe.read_csv(...)). The second works because the call receiver is now flattened by a newast_extract::dotted_module_pathinstead of being destructured as a bareExpr::Name. A bare name flattens to itself, so every pandas/polars/Schema.from_pandaspath is byte-for-byte unchanged.Structural operations.
drop,rename(mapping andstr.lower/str.upperfold),assign, column-list subscript (df[["a","b"]]), subscript assignment,del,pop,left.merge(right, ...),dd.concat([...]). These needed no code changes — they were already keyed on method name plus a tracked receiver, so they are backend-agnostic. The new tests pin that down rather than leaving it to coincidence.Row-passthrough.
query,head,tail,sample,sort_values,reset_index,nlargest,nsmallest,fillna,dropna,ffill,bfill— every member of the existing list that dask actually implements.Materialization.
compute,persist, andrepartitionadded toROW_PASSTHROUGH_METHODS, so a lazy dask pipeline's column set survives the lazy→eager boundary instead of being dropped there.dd.from_pandas(pdf, npartitions=n). NewFRAME_WRAP_FUNCTIONSconstant: carries the source frame's schema over unchanged, and counts as a DataFrame origin either way (emittinguntracked-dataframewhen the source itself has no known columns), which is the invariant every otherLOAD_MODULEScall already upholds.Annotated[dd.DataFrame, Schema]. Already worked — the annotation parser keys on theDataFrameattribute name, not the module it hangs off. Pinned with a cross-file contract test so a future narrowing of that check can't silently drop dask.Incidental fixes outside dask
Two things surfaced while verifying parity. Both are called out here rather than slipped in quietly:
polars
.collect()had no handling at all. Apl.scan_csv(...)column set was lost the moment it was collected, so every post-.collect()access went unchecked.collectis added alongside dask's.compute()— they are the same shape, and building the mechanism for one without the other made no sense. There's a regression test for it.Two inaccuracies in
docs/method-matrix.md. The schema-modifying table listedpd.merge(left, right, …)as tracked (it isn't — only theleft.merge(right, …)instance form is), and the untracked table listeddf.merge(other, …)as untracked (it is tracked, as a union of both schemas). Both corrected. Verified with:pl.from_pandas(pdf)also now resolves via the sameFRAME_WRAP_FUNCTIONSpath (it previously did nothing and leftplas an unresolved external-package candidate). Note this is a small behaviour change for existing polars users:pl.from_pandas(x)with an untrackedxnow counts as a DataFrame origin and emitsuntracked-dataframe, which can move a--coverage-fail-undernumber.Deliberately deferred
dd.from_dict({...}),from_delayed,from_array,from_map.from_dicttakes a literal rather than a tracked frame — the same shape aspd.DataFrame({...}), which this checker also leaves untracked, so tracking it here would be an inconsistency, not parity. The rest produce columns that only exist once the graph runs.dd.merge(left, right). Not tracked, exactly matchingpd.mergetoday. Making it work is a separate change that would apply to all three backends; doing it only for dask would be worse than not doing it.RESERVED_METHODS(compute,persist,npartitions,categorize, …). Adding them would start flagging schema columns with those names for existing pandas/polars users who don't use dask, since that check is not backend-scoped. Net-negative noise, so left alone.filter,sort,insert,select. Confirmed absent fromdask.dataframe(they are pandas/polars-only spellings). They stay in the tables because a dask user never writes them, and they only fire on a tracked receiver.df.query(...).sort_values(...)) is still not followed — the receiver must be a named variable. This is pre-existing and backend-independent (pandas behaves identically); now documented in the method matrix rather than silently assumed.How this was tested
Real dask execution, not docs. Every column-set claim in the new code comments and tests was verified by running the operation against dask 2026.8.0 and reading back
.columns. That covers:read_csv(usecols=),read_parquet(columns=), all twelve row-passthrough methods,drop/rename/rename(str.upper)/assign/subscript slice/subscript assignment/del/pop,compute/persist/repartition,concat, instance and modulemerge, andfrom_pandas. It also confirmed thatfilter/sort/select/insertgenuinely do not exist on a dask DataFrame, and thatdd.read_csv(dtype={...})does not narrow columns (same as pandas — pre-existing behaviour, unchanged here).Docs-only verification: none of the behavioural claims. The only non-executed checks were the comparison-matrix cells for other tools: narwhals' Dask support was read from its docs (lazy-only backend list), Great Expectations' absence of Dask support from its data-sources page. Pandera's Dask support was verified by execution (
ddf.pandera.add_schema(...)after importingpandera.accessors.dask_accessor).Suites.
uv run inv allgreen — ruff, ty, bandit, complexipy,cargo fmt --check,cargo clippy -- -D warnings, 301 Python tests at 100% branch coverage, licensecheck (dask is BSD-3-Clause).cargo testgreen: 201 Rust tests, 15 of them new.No lint-ignore rules were added and no bandit checks skipped.
Also in this PR
examples/features/annotated_dask_example.py(four real caught errors + one warning), fitted alongside the existingannotated_pandas_example.py/annotated_polars_example.pyrather than a newexamples/backends/tree. Its README entry notes it runs mypy without--strict: dask shipspy.typedbut its annotations are incomplete (dd.read_csvreturnsAny,dd.from_pandasis untyped), so--strictreports against dask's own API, not this file.Daskrow in the comparison matrix marked⚠️ Experimentalfor typedframes, plus the operations/examples lists.docs/usage.mdanddocs/method-matrix.md.dask[dataframe]added to the dev dependency group only — the shipped package gains no runtime dependency, and the checker never imports dask (it is a pure AST pass).https://claude.ai/code/session_013nRj6HvRbwkzAoj3QPbiSj