Repository navigation
branch-4.2: [fix](nereids) Fix deep copy of scans with virtual columns #69399 - #69560
Merged
Merged
Conversation
### What problem does this PR solve? Related PR: apache#69399 Problem Summary: Backport apache#69399 to branch-4.2. A CTE that orders by score() or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite, failed with "A expression contains slot not from children". LogicalPlanDeepCopier gave each virtual column two ids, copied them in list order, kept the old slot in the output of the copied scan and left the score and ANN order keys on the old slots. The copier now maps every virtual column to its new id before copying their expressions and remaps the order keys, and LogicalOlapScan.withVirtualColumns replaces the old virtual slots in the output. The branch has no table streams, so the change to LogicalOlapTableStreamScan is dropped. withVirtualColumns keeps the constructor call of the branch, and LogicalPlanDeepCopierTest keeps the tests of the branch and adds the two tests of apache#69399. ### Release note Fix "A expression contains slot not from children" for a CTE that orders by score() or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite. ### Check List (For Author) - Test: Includes the unit and regression tests of apache#69399; results are recorded in the pull request. - Behavior changed: No. - Does this need documentation: No. Source-PR: apache#69399 Source-Head: 5be9ff5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
4 of 5 tasks
Member
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
yiguolei
approved these changes
Oct 24, 2026
yiguolei
pushed a commit
that referenced
this pull request
Oct 24, 2026
…69292 (#69562) ### What problem does this PR solve? Related PR: #69292 Problem Summary: Backport #69292 to `branch-4.2`. It contains the commit of #69560, the backport of #69399, which it needs: binding runs before CTE inlining, which copies scans with virtual columns. Merge #69560 first. `score()` only worked as `ORDER BY score() LIMIT n` directly above the scan whose MATCH defines it. A minimum score filter without that TopN evaluated the unbound `score()` placeholder as zero and silently dropped every row: ```sql SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001; -- returned nothing ``` `BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()` to the scan of its SELECT at the start of rewrite and rejects the uses it cannot bind. The minimum score stays a predicate of the scan, and `PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score together with the minimum. In the BE, an expression on the score reads the score column instead of evaluating `Score()`, and MATCH projections do not add to the score. See #69292 for the supported and rejected uses. #### Branch adaptations - The `SegmentIterator` of the branch collects the columns of pushed-down expressions in `_extract_common_expr_columns` instead of per-column states, so the score column is marked there, and the new BE test checks `_common_expr_columns` and `_is_common_expr_column`. The branch reads the columns collected there before it evaluates the expressions, also virtual ones, and a virtual slot uses a column that is already read. - When `limit + offset` overflows, `PushDownScoreTopNIntoOlapScan` keeps the behavior of the branch and skips the push-down instead of failing. - `SearchDslParser` is imported from its package on the branch. - The branch does not have #68672, which keeps predicates on virtual columns out of the condition cache. The minimum score is now a predicate of the scan, and a score depends on every rowset, so a cached result goes stale when a rowset is added: without the change, `test_score_filter_matrix` lost the rows of the first rowset. `VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672 that this needs, with the `VirtualSlotRef` tests of #68672 that the branch can run. ### Release note `score()` works in the select list, window functions and ORDER BY of a SELECT on one table whose WHERE clause has MATCH or SEARCH, with an optional minimum score `score() > x` or `score() >= x`, without `ORDER BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set operation still needs `ORDER BY score() LIMIT`. Other uses of `score()` are rejected instead of evaluating it as 0. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - Behavior changed: - [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and unsupported uses of `score()` fail planning instead of evaluating it as 0. A minimum score without a TopN needs the BE change, so BEs must be upgraded before FEs. - Does this need documentation? - [x] Yes. The `score()` documentation should describe where `score()` can be used. ### Validation Validated together with the other two backports (#69399, #69419 and #69292 on `branch-4.2` at ecbc5fe), in a local ASAN build: - PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`, `InferPredicateByReplaceTest`, `InferPredicatesTest`, `UnequalPredicateInferTest`, `BindScoreToOlapScanTest`, `SearchExpressionTest`, `TopNRuntimeFilterTest`, `PullUpProjectExprUnderTopNTest` and `MaterializedViewUtilsTest` (280 tests), and FE Checkstyle. - PASS: `./build.sh --fe --be`. - PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`, `SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests); clang-format 16 on the changed BE files. - PASS: the 23 regression suites of the branch that use `score()`, including `test_cte_inline_virtual_column`, `infer_match_predicate`, `test_score_filter_usage` and `test_score_filter_matrix`, with unchanged `.out` files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #69399
Problem Summary: Backport #69399 to
branch-4.2.A CTE fails to plan with "A expression contains slot not from children" when the optimizer inlines it after one of its scans got a virtual column: a CTE that orders by
score()or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite.LogicalPlanDeepCopiergave each virtual column two ids, copied them in list order, kept the old slot in the output of the copied scan and left the score and ANN order keys on the old slots. The copier now maps every virtual column to its new id before copying their expressions and remaps the order keys, andLogicalOlapScan.withVirtualColumnsreplaces the old virtual slots in the output.Branch adaptations
The branch has no table streams, so the change to
LogicalOlapTableStreamScanis dropped.withVirtualColumnskeeps the constructor call of the branch,LogicalPlanDeepCopierimportsLogicalOlapScan, which master already imported, andLogicalPlanDeepCopierTestkeeps the tests of the branch and adds the two tests of #69399.Release note
Fix "A expression contains slot not from children" for a CTE that orders by
score()or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite.Check List (For Author)
Validation
Validated together with the other two backports (#69399, #69419 and #69292 on
branch-4.2at ecbc5fe), in a local ASAN build:LogicalPlanDeepCopierTest,CTEInlineTest,InferPredicateByReplaceTest,InferPredicatesTest,UnequalPredicateInferTest,BindScoreToOlapScanTest,SearchExpressionTest,TopNRuntimeFilterTest,PullUpProjectExprUnderTopNTestandMaterializedViewUtilsTest(280 tests), and FE Checkstyle../build.sh --fe --be.SegmentIteratorScoreCommonExprTest,SegmentIteratorApplyIndexExprTestandVirtualSlotRefTest(31 tests); clang-format 16 on the changed BE files.score(), includingtest_cte_inline_virtual_column,infer_match_predicate,test_score_filter_usageandtest_score_filter_matrix, with unchanged.outfiles.🤖 Generated with Claude Code