Skip to content

fix: keep a join against a table with unknown columns as a join instead of INTERSECT or EXCEPT - #6556

Merged
max-sixty merged 2 commits into
mainfrom
fix/semi-join-intersect
Nov 15, 2026
Merged

max-sixty merged 2 commits into
mainfrom
fix/semi-join-intersect

Conversation

@prql-bot

Copy link
Copy Markdown
Collaborator

A plain semi-join against a table whose columns prqlc doesn't know compiled to a set operation with mismatched columns. This keeps it a join.

from a
join b (==id)
select {a.id}

compiled to SELECT id FROM a INTERSECT ALL SELECT * FROM b. The database either rejects it (different column counts) or, when b happens to have one column, returns a different result. The left-join + filter b.id == null anti-join had the same problem with EXCEPT ALL. With this change both compile to the INNER JOIN / LEFT OUTER JOIN ... WHERE b.id IS NULL the user wrote.

Cause. intersect and except in prqlc/prqlc/src/sql/pq/preprocess.rs check the join condition against bottom, the bottom relation's columns after prune_inputs has dropped the unused ones. Here that left just b.id, so the condition looked like an equality over every column. But the generated set operation reads the bottom as SELECT *. The rewrite now only fires when pruning kept every column of the bottom relation instance (table_ref.columns is the same length as original_cids). #6443 described this gap as a separate follow-up.

What stays the same. std.intersect and std.remove still compile to INTERSECT / EXCEPT, including on wildcard tables (from album | intersect artist). They join on every column, so nothing gets pruned. A bottom relation whose declared or CTE columns are exactly the joined ones also still gets rewritten. All the existing test_intersect_* / test_remove_* snapshots and the book snapshots are unchanged.

Tests. test_intersect_10 and test_remove_08 in tests/integration/sql.rs captured the INTERSECT ALL / EXCEPT ALL output before the fix and show the joins after it. task prqlc:test passes (796 tests) and cargo clippy -p prqlc --all-targets is clean.

This also covers the shape reported in #4279 (from foo | join side:inner bar (==idx) | aggregate {count this}), which now compiles to an INNER JOIN.

…ad of INTERSECT or EXCEPT

The INTERSECT/EXCEPT rewrite compared the join condition against the bottom
relation's columns after pruning, but the generated set operation reads the
bottom as SELECT *. A semi-join such as `from a | join b (==id) | select {a.id}`
compiled to `SELECT id FROM a INTERSECT ALL SELECT * FROM b`, which a database
rejects or answers differently whenever b has other columns.

Only rewrite when pruning kept every column of the bottom relation instance.
@max-sixty
max-sixty merged commit 79871f0 into main Nov 15, 2026
36 checks passed
@max-sixty
max-sixty deleted the fix/semi-join-intersect branch November 15, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants