Repository navigation
fix: don't use SELECT DISTINCT for group (take 1) when a later step reads another column - #6547
Merged
Merged
Conversation
…eads another column
A `group {a} (take 1)` whose final select is just `a` compiled to
`SELECT DISTINCT` even when a step between the take and the final select
(a filter, sort or derive) still read another column. That column then
had to be in the relation applying DISTINCT, so DISTINCT compared
`(a, b)` rather than `a` and returned more than one row per group.
Fall back to ROW_NUMBER (or DISTINCT ON) when a transform after the take
reads a column of its input outside the partition.
This was referenced Nov 12, 2026
# Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
group {a} (take 1)compiled toSELECT DISTINCTwhenever the pipeline's final select was exactly the partition columns, even if a step between thetakeand that select still read another column. That column then has to be output by the relation applying DISTINCT, so DISTINCT compares more columns than the partition and returns more than one row per group. This PR keeps DISTINCT only when nothing after thetakereads a non-partition column of its input; otherwise the existingROW_NUMBER/DISTINCT ONpath is used.On
mainthis compiles to a CTE ofSELECT DISTINCT a, b FROM tfiltered onb > 0, so anawith two positivebvalues comes back twice. With the fix it compiles toROW_NUMBER() OVER (PARTITION BY a)withWHERE _expr_0 <= 1 AND b > 0, i.e. one arbitrary row pera, then the filter. The same happened withsort bin place of the filter.The cause is in
distinct()insql/pq/preprocess.rs: it compared the partition againstdetermine_select_columnsof the whole pipeline, which only sees the final select. The newreads_non_partition_columncheck collects the cids read by transforms after thetakeand rejects DISTINCT if any of them is an input column (from thefrom/joins or a compute before thetake) outside the partition.test_distinct_10intests/integration/sql.rsis the regression test; it pinned theSELECT DISTINCT a, boutput before the fix. All 805 tests inprqlc-parser,prqlcandmdbook-prqlpass with no other snapshot changes, so existingDISTINCToutputs (test_distinct_02/03/05, the mssqlFETCHcases,test_distinct_on_*) are unchanged.Why not compare against the frame at the take instead
The first attempt computed the columns from the transforms before the
takerather than the whole pipeline. It broke two existing tests: by that stageprune_inputshas already dropped table columns that aren't read later, sofrom tab1 | group col1 (take 1) | derive foo = 1 | select foosaw a frame of justcol1and emittedSELECT DISTINCT NULL, and the mssqlgroup {d = this.District} (take 1)case lost its DISTINCT. Keeping the whole-pipeline check and adding the later-reads condition avoids both.