Skip to content

fix: don't use SELECT DISTINCT for group (take 1) when a later step reads another column - #6547

Merged
max-sixty merged 3 commits into
mainfrom
fix/distinct-reads-dropped-column
Nov 15, 2026
Merged

max-sixty merged 3 commits into
mainfrom
fix/distinct-reads-dropped-column

Conversation

@prql-bot

Copy link
Copy Markdown
Collaborator

group {a} (take 1) compiled to SELECT DISTINCT whenever the pipeline's final select was exactly the partition columns, even if a step between the take and that select still read another column. That column then has to be output by the relation applying DISTINCT, so DISTINCT compares more columns than the partition and returns more than one row per group. This PR keeps DISTINCT only when nothing after the take reads a non-partition column of its input; otherwise the existing ROW_NUMBER / DISTINCT ON path is used.

from t
select {a, b}
group {a} (take 1)
filter b > 0
select {a}

On main this compiles to a CTE of SELECT DISTINCT a, b FROM t filtered on b > 0, so an a with two positive b values comes back twice. With the fix it compiles to ROW_NUMBER() OVER (PARTITION BY a) with WHERE _expr_0 <= 1 AND b > 0, i.e. one arbitrary row per a, then the filter. The same happened with sort b in place of the filter.

The cause is in distinct() in sql/pq/preprocess.rs: it compared the partition against determine_select_columns of the whole pipeline, which only sees the final select. The new reads_non_partition_column check collects the cids read by transforms after the take and rejects DISTINCT if any of them is an input column (from the from/joins or a compute before the take) outside the partition.

test_distinct_10 in tests/integration/sql.rs is the regression test; it pinned the SELECT DISTINCT a, b output before the fix. All 805 tests in prqlc-parser, prqlc and mdbook-prql pass with no other snapshot changes, so existing DISTINCT outputs (test_distinct_02/03/05, the mssql FETCH cases, test_distinct_on_*) are unchanged.

Why not compare against the frame at the take instead

The first attempt computed the columns from the transforms before the take rather than the whole pipeline. It broke two existing tests: by that stage prune_inputs has already dropped table columns that aren't read later, so from tab1 | group col1 (take 1) | derive foo = 1 | select foo saw a frame of just col1 and emitted SELECT DISTINCT NULL, and the mssql group {d = this.District} (take 1) case lost its DISTINCT. Keeping the whole-pipeline check and adding the later-reads condition avoids both.

…eads another column

A `group {a} (take 1)` whose final select is just `a` compiled to
`SELECT DISTINCT` even when a step between the take and the final select
(a filter, sort or derive) still read another column. That column then
had to be in the relation applying DISTINCT, so DISTINCT compared
`(a, b)` rather than `a` and returned more than one row per group.

Fall back to ROW_NUMBER (or DISTINCT ON) when a transform after the take
reads a column of its input outside the partition.
@max-sixty
max-sixty merged commit 413ce6c into main Nov 15, 2026
36 checks passed
@max-sixty
max-sixty deleted the fix/distinct-reads-dropped-column branch November 15, 2026 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants