Repository navigation
fix: keep a join against a table with unknown columns as a join instead of INTERSECT or EXCEPT - #6556
Merged
Merged
Conversation
…ad of INTERSECT or EXCEPT
The INTERSECT/EXCEPT rewrite compared the join condition against the bottom
relation's columns after pruning, but the generated set operation reads the
bottom as SELECT *. A semi-join such as `from a | join b (==id) | select {a.id}`
compiled to `SELECT id FROM a INTERSECT ALL SELECT * FROM b`, which a database
rejects or answers differently whenever b has other columns.
Only rewrite when pruning kept every column of the bottom relation instance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A plain semi-join against a table whose columns prqlc doesn't know compiled to a set operation with mismatched columns. This keeps it a join.
compiled to
SELECT id FROM a INTERSECT ALL SELECT * FROM b. The database either rejects it (different column counts) or, whenbhappens to have one column, returns a different result. The left-join +filter b.id == nullanti-join had the same problem withEXCEPT ALL. With this change both compile to theINNER JOIN/LEFT OUTER JOIN ... WHERE b.id IS NULLthe user wrote.Cause.
intersectandexceptinprqlc/prqlc/src/sql/pq/preprocess.rscheck the join condition againstbottom, the bottom relation's columns afterprune_inputshas dropped the unused ones. Here that left justb.id, so the condition looked like an equality over every column. But the generated set operation reads the bottom asSELECT *. The rewrite now only fires when pruning kept every column of the bottom relation instance (table_ref.columnsis the same length asoriginal_cids). #6443 described this gap as a separate follow-up.What stays the same.
std.intersectandstd.removestill compile toINTERSECT/EXCEPT, including on wildcard tables (from album | intersect artist). They join on every column, so nothing gets pruned. A bottom relation whose declared or CTE columns are exactly the joined ones also still gets rewritten. All the existingtest_intersect_*/test_remove_*snapshots and the book snapshots are unchanged.Tests.
test_intersect_10andtest_remove_08intests/integration/sql.rscaptured theINTERSECT ALL/EXCEPT ALLoutput before the fix and show the joins after it.task prqlc:testpasses (796 tests) andcargo clippy -p prqlc --all-targetsis clean.This also covers the shape reported in #4279 (
from foo | join side:inner bar (==idx) | aggregate {count this}), which now compiles to anINNER JOIN.