[fix](inverted index) Count only visible rows in BM25 statistics of merge-on-write tables by guptashivam04 · Pull Request #69438 · apache/doris

guptashivam04 · 2026-10-22T06:46:07Z

What problem does this PR solve?

Issue Number: None

Related PR: #69352 (BM25 statistics shared across the parallel scanners of a tablet; this PR is its follow-up for replicas), #69313 (BM25 token totals after CLucene compaction)

Problem Summary:

On a merge-on-write table, BM25 score() gives the same row a different score on each replica of its tablet. A query picks one replica per tablet, so with replication_num = 3 a row has up to three scores. Pagination over ORDER BY score() then repeats and skips rows from one page to the next. Each replica is stable on its own (use_fix_replica pins the scores), but the replicas disagree.

CollectionStatistics::process_segment() takes the document count, the token count and each term's document frequency straight from every segment's inverted index. Under merge-on-write, a row overwritten or deleted by a later load stays in its segment's index, marked only in the delete bitmap. A row written with the delete sign, which CDC deletes and DELETE produce, stays as a normal row that queries filter out by predicate. Both kinds are counted until compaction removes them, and each replica compacts on its own schedule. So each replica counts a different set of rows nobody can see.

Reproduction (3 BEs, replication_num = 3, 8 buckets, auto compaction off): 8 loads, 3 rounds of overwriting a quarter of the rows, a DELETE, and full rows inserted with __DORIS_DELETE_SIGN__ = 1. Then a full compaction (POST /api/compaction/run?compact_type=full) of the tablets on one BE only. On master with #69352, the query below changed the score of all 3,799 matching rows from run to run: 1,900 distinct scores instead of 950. With use_fix_replica = 0/1/2 each replica was stable, but the three disagreed.

SELECT id, score() FROM rt WHERE SEARCH('content:jobs OR content:try') ORDER BY score() DESC LIMIT 100000;

The token total has a second, replica-independent problem: CLucene index compaction (inverted_index_compaction_enable, on by default) copies the source token totals even when it drops rows, which #69313 addresses. On master, the same rows scored differently in a compacted table and in a table freshly loaded with its visible rows.

Fix:

When the scan reads a delete bitmap (merge-on-write), CollectionStatistics describes only the rows the scan returns:

OlapScanner passes the scan's delete bitmap and read version (MowHiddenRows) to collect(), or to TabletCollectionStatistics::copy_to() for a split tablet. Every scanner of a tablet reads with the same bitmap and version, so the statistics shared by [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 stay correct.
For each segment, the hidden rows are the rows the delete bitmap marks at the read version (get_agg, the same call the rowset reader uses), plus the rows whose hidden __DORIS_DELETE_SIGN__ column is set. Every unique-key table has that column, and DELETE on a merge-on-write table writes such rows unless enable_mow_light_delete is on. The delete-sign column is read only for segments whose zone map shows a non-zero sign (or cannot tell), so a segment without CDC deletes costs nothing extra. The read follows the query's enable_segment_cache and enable_page_cache.
Document count: the index's document count minus the hidden rows.
Term document frequency: IndexSource::scoring_doc_freq_among() counts how many hidden rows a term's postings list. When the hidden rows are much rarer than the term's postings, it leaps between them with seek_block. Otherwise it decodes the postings once and walks the hidden rows alongside them, with no allocation, and a term without postings costs nothing. SNII overrides it so that reserved-namespace scoring terms resolve as scoring_doc_freq resolves them.
Token count: the sum of the document lengths the norms of the returned non-NULL rows stand for, or the returned row count for an index without norms. The exact length of a hidden row is not stored anywhere: CLucene and SNII keep only the encoded norm (exact up to 40 tokens for CLucene, up to 254 for SNII). Subtracting decoded lengths from the exact sumTotalTermFreq would still leave replicas apart. A row's norm is the same in every segment it is written to, so summing norms gives every replica the same count. It is also the same quantity the BM25 tf normalization already divides, dl / avgdl, with dl taken from the same norm.
A NULL row has no tokens, but its norm stands for one (CLucene's default norm, SNII's encode_norm(0)). Its length is left out, using the index's null bitmap, and it still counts as a document, as before. So on CLucene a table whose texts all have at most 40 tokens, and none of which is an empty string, scores exactly as a duplicate-key table with the same rows.
The statistics read CLucene norms into a buffer of their own, one segment at a time, and keep none. Asking a CLucene reader for a field's norms makes it keep them for the searcher's lifetime, uncharged to the searcher cache, and the statistics visit every segment of the read set, including those the scan prunes and never scores. Reading per segment also avoids the copy of all segments' norms a multi-segment reader would keep.
What the statistics keep instead is small and kept with the cached searcher: the sum of a field's non-NULL norm lengths, and the sum of the hidden rows' lengths for the last few sets of hidden rows of a field (queries at one read version hide the same rows). So queries at an unchanged version read no norms; a segment's norms are read again only when its hidden rows change. The hidden and NULL row ID lists the statistics build while collecting are charged to the query's memory.

Why the token count has to change too: a variant that left out hidden rows only from the document count and the document frequencies, keeping today's token total, made the replicas agree only while every replica compacted through inverted index compaction, which carries dropped rows' token counts forward. With one replica compacted with inverted_index_compaction_enable = false (the index rebuilt from column data), every row again scored differently on that replica. Its scores were also 12–31 % off a fresh load of the visible rows on average, because its token total still counted the hidden rows.

A tablet whose rows are all hidden (every row deleted or overwritten, not yet compacted) has no document a query can match. Its statistics report zero documents instead of failing the query with "No data available for SimilarityCollector".

Tables without a delete bitmap (duplicate key, merge-on-read, or skip_delete_bitmap) collect the same statistics as before.

For every score() query, with or without a delete bitmap:

The statistics reads follow the query's disable_file_cache and inverted_index_snii_read_no_write_file_cache, as the scan's reads do. They run before the scan, so they are the reads that fill the file cache. CLucene reads norms without a per-read context, as scoring does.
A CLucene searcher the statistics open drops the query's I/O context once its meta is read, as the query path already does (create_index_searcher), and a CLucene error while opening it fails the query; before, it escaped the scanner thread uncaught and aborted the backend. Before, a searcher the statistics put into the searcher cache kept the opening query's I/O context, including its file cache statistics pointer, for later queries to read through.

Not covered:

Merge-on-read unique tables and duplicate tables with DELETE predicates also count rows that compaction later drops. They need merge or predicate evaluation to tell which rows are visible, and are left for a follow-up.
With skip_delete_sign = true or show_hidden_columns = true, rows written with the delete sign are returned but still left out of the statistics. Rows deleted by a delete predicate (enable_mow_light_delete) are still counted, as the delete bitmap does not mark them.
A row indexed without tokens that is not NULL keeps a norm that cannot be told from a one-token row's, so on a merge-on-write table it counts as one token: an analyzed empty string, which both formats index like a NULL but leave out of the null bitmap, and on SNII any text that yields no token (for example '!!!', or text without ASCII letters or digits under the english parser), whose norm SNII stores as that of one token. On CLucene such a text has a zero norm and counts as zero tokens. Telling these rows apart would mean reading the column.
On SNII, the sum of the norm lengths is not cached yet: each merge-on-write query reads the segment's norms.
Leaving NULL rows out relies on the index's null bitmap, which CLucene index compaction carries over; if reading a source bitmap fails there, the compacted index loses it (existing behaviour of the vendored CLucene compaction).
Statistics stay per tablet, as before.

Interaction with #69313: that PR makes the token total of a CLucene index built by compaction exact. With this PR, merge-on-write scans no longer read that total, so their scores do not depend on it. Scans of other tables still do. The two changes are independent.

Release note

Fix BM25 score() on merge-on-write tables differing between the replicas of a tablet, and before and after compaction. Statistics now count only rows visible to the query, and the average document length comes from the indexed norms.

Check List (For Author)

Test

Regression test: inverted_index_p0/test_bm25_score_replica_statistics. It loads overwrites, DELETEs and delete-sign rows, then expects every row to score exactly as in a table freshly loaded with the visible rows. It checks before compaction, with one replica compacted (≥ 3 BEs: 10 runs and use_fix_replica = 0/1/2), and after compacting all replicas (the compaction steps are skipped in cloud mode, which keeps one shared copy of each tablet). It also checks that a merge-on-write table with NULL rows scores exactly as a duplicate-key table with the same rows, and that a bucket whose rows are all deleted returns no rows instead of failing. Without this change it fails before any compaction (1352 rows score differently from a freshly loaded table); with only the first version of this change it failed the NULL check (250 rows score differently). With it, it passes on a 3-BE cluster.
Regression test: fault_injection_p0/test_bm25_score_statistics_searcher_io_context. With the CSIndexInput.readInternal debug point, which fails a postings read that still reads as index meta, a cold score() query on a merge-on-write table failed before the searcher reset ("The 'is_index_data' flag should be false for non-meta inverted index files") and passes with it.
Existing suites: all 23 suites that use score() (in inverted_index_p0, search, ann_index_p0, variant_p0 and nereids_rules_p0) pass on a 3-BE cluster. One expected value changes: in test_bm25_score, table t2 is a merge-on-write table whose row a = 3 was updated, so the old version of that row no longer counts. Two documents count instead of three, and the score of hello world goes from 1.0925692 to 0.8025915.
Unit Test: CollectionStatisticsTest.HiddenRowsAreCollectedLikeCompactedAway (real CLucene segments: a segment with a hidden 45-token row collects the same doc count, token count, df, avgdl and idf as the segment without the row), CollectionStatisticsTest.AllRowsHiddenCollectsEmptyStatistics, CollectionStatisticsTest.NullRowsCountAsDocumentsWithoutTokens (real CLucene segment with NULL rows: the same doc count and token count with and without the delete bitmap, and with a hidden NULL row), CollectionStatisticsHiddenRowsTest.* (counting hidden rows in postings by scanning and by leaping, over sparse and dense blocks; norm sums with and without NULL rows; rows beyond the index; indexes without norms), CluceneSearcherCacheValueTest.NonNullNormLengthSumIsComputedOncePerField and OlapScannerTest.BuildScoreRuntimeCollectionIoContextFollowsQueryFileCacheOptions. Run locally on macOS (ASAN build): all 243 BM25, collection-statistics, scanner, searcher cache, inverted index reader and null bitmap tests pass. A wider set of 2,732 inverted index, SNII, BM25, search and scan tests also passes, except 6 that fail on macOS in code this PR does not touch (5 SNII writer memory-accounting formulas and 1 ScoreRuntimeTest death-test message).

Manual test: master with [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352, with and without this change, macOS arm64, 3 BEs on one host, the reproduction above. use_fix_replica pins one replica; "random" lets the FE pick a replica per tablet per query.

State	without this change	with this change
replicas identical	0 of 3,799 rows change across 10 runs, 950 scores	0 rows change, 950 scores
full compaction on one replica, random replica, 10 runs	3,799 of 3,799 rows change, 1,900 scores	0 rows change, 950 scores
same, `use_fix_replica = 0/1/2` compared	every row differs between replicas	identical
all replicas compacted	stable, but differs from a fresh load of the same rows on every row	identical to the one-replica-compacted state and to a fresh load, bit for bit

Same result for a 2-replica table with BUCKETS AUTO, the unicode parser with lower_case, 4 indexed text columns queried by SEARCH('a:x OR b:x OR c:x OR d:x') plus a non-index filter, auto compaction on, updates by INSERT INTO ... SELECT and deletes by DELETE FROM. Before: every row differed between the replicas (e.g. 3.30 vs 2.69). After: identical on both and equal to a fresh load. The same holds with 1–10 MB text values.

Each of these also scored identically on every replica (one replica fully compacted) and matched a fresh load of its visible rows: partial-column updates, a sequence column, stream loads with __DORIS_DELETE_SIGN__ and with merge_type: DELETE, group commit, keyword and lowercase-normalizer indexes, a range-partitioned table, the SNII storage format, a variant column, MATCH_ANY / MATCH_ALL / MATCH_PHRASE, and a tablet split across scanners (enable_parallel_scan and optimize_index_scan_parallelism). A 90-second run of concurrent upserts, deletes, compactions and score queries finished without errors and left the replicas identical.

Overhead, measured from query profiles with a 4-field SEARCH(... OR ...) and a non-index filter. Each number is the median of 15 runs on the same data and query:

Table	without this change	with this change
11 k rows, 3 replicas	27 ms	15 ms (within noise)
1.75 M visible rows, 1.3 M hidden (43 % of stored rows), 8 buckets	35 ms	36 ms

On one 8 M-row tablet (20 % NULL, 80 k overwritten rows), StatisticsCollectTime went from 0.6 ms to 4.7 ms per query (median of 9), and the query from 27 ms to 22 ms (noise). The first query on a segment also sums its norms (15 ms here).

The cost grows with the rows compaction has not removed yet. To count the hidden rows that hold a term, the collector leaps from hidden row to hidden row when they are much rarer than the term's postings (32 times rarer). Otherwise it decodes the postings once and walks the hidden rows alongside. The norms of a segment's hidden rows are read once per set of hidden rows.

No need to test or manual test.

Behavior changed:

Yes. On merge-on-write tables, BM25 statistics leave out rows hidden by the delete bitmap or the delete sign, and avgdl is computed from the norms of the visible rows. Scores no longer depend on the replica or on compaction. They also shift once on tables without deletes, because norms store long lengths only approximately. Measured on freshly loaded tables:

Texts	Mean score change	Largest change	Row pairs that swap order	Top 100
short to medium (up to ~45 words)	0.27 %	0.59 %	0.021 %	unchanged
long (~380 words on average)	1.75 %	2.6 %	0.085 %, all near-ties	1 near-tie at the boundary
with 19 documents of 1–10 MB	1.47 %	6.2 %	0.11 %	unchanged

Does this need documentation?
- No.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…erge-on-write tables A merge-on-write segment keeps rows that a later load overwrote or deleted, marked only in the delete bitmap, and rows written with the delete sign, which queries filter out by predicate. BM25 statistics counted both until compaction removed them, and each replica compacts on its own schedule, so the replicas of a tablet scored the same row differently and a query that picked a different replica returned different scores and orders. When the scan reads a delete bitmap, collect statistics over the rows it returns only: per segment, leave the rows marked at the read version and the rows with the delete sign out of the document count and of each term's document frequency, counting the hidden rows a term's postings hold by leaping between them. Exact token counts of hidden rows are not stored, and norms hold lengths only approximately, so take the token count as the sum of the lengths the norms of the returned rows stand for, which every replica computes alike whether or not it has compacted.

hello-stephen · 2026-10-22T06:46:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

…eplica suite outside cloud mode Cloud mode keeps one shared copy of each tablet, so there are no replicas to compact apart, and its compaction is driven differently. The comparison with a freshly loaded table still runs in every mode.

When every row of a merge-on-write tablet is overwritten or deleted but not yet compacted, the visible-row statistics hold no documents. Scoring then treated them as missing and failed the query with "No data available for SimilarityCollector". Such a tablet has no row a query can match, so record that its segments were collected and report zero documents instead.

…ablet id The delete bitmap's aggregation cache is keyed by tablet, rowset, segment and version. The mock rowsets share their ids, so the second collection in HiddenRowsAreCollectedLikeCompactedAway read the first one's cached hidden rows.

…merge-on-write tables A NULL row has no tokens, but its norm stands for one, so summing the norms of the returned rows counted one token per NULL row. avgdl then grew with the share of NULL rows, and a merge-on-write table scored its rows differently from a duplicate table with the same data. Leave the rows of the index's null bitmap out of the token count; they still count as documents, as before. The sum of the non-NULL norm lengths of a CLucene index never changes, so keep it with the cached searcher instead of summing every row's norm on every query. Each query now reads only the norms of its hidden rows. Read the delete sign with the query's segment, page and file cache settings, and also read it when the zone map cannot vouch for its bounds or shows a non-zero minimum, since any non-zero sign deletes the row. Document the read modes whose returned rows differ from the counted ones. Follow the regression test conventions in the replica suite, and check it against a duplicate table with NULL rows.

…e options The statistics read postings, norms and null bitmaps before the scan does, so they are the reads that fill the file cache, yet their I/O context ignored disable_file_cache and inverted_index_snii_read_no_write_file_cache. Set both for a query's reads, as the rowset reader does; the delete sign read copies the context and follows them too. Document that a row indexed without tokens but not NULL keeps a norm that cannot be told from a one-token row's, and test that the non-NULL norm length sum is computed once per field of a cached searcher.

…stics searchers A CLucene searcher that BM25 statistics open is put into the searcher cache, but its index stream kept the opening query's I/O context, with that query's file cache statistics pointer, and stayed marked as reading index meta. Later reads without their own context went through the finished query's context, and the statistics' own reads ignored the query's file cache options. Reset the stream once the meta is read, as the query path's searcher opener does.

When a CLucene index holds several segments, its reader builds and keeps one more copy of a field's norms for all of them, filled without the segments' own norm caches, so the statistics read the norms a second time and held them twice for the cached searcher's lifetime, uncharged. Read the norms of each segment instead, as scoring does; a segment without the field's norms reads as zero lengths, as the combined norms do. Add a regression test for the statistics searcher's stream reset: with the CSIndexInput.readInternal debug point a cold score() query failed while the stream still read postings as index meta.

…a CLucene index The BM25 statistics opened a segment's CLucene index without catching CLuceneError, so an error reading the index, such as a failed remote read of its header, escaped the scanner thread and aborted the backend. Report it as the query's error, as the query path does. Keep the searcher reset test from passing on searchers the write path caches, test a hidden row whose length the norm holds only approximately (CLucene's norms are exact up to 40 tokens), and correct the norm comment.

yiguolei · 2026-10-24T03:17:31Z

/review

yiguolei · 2026-10-24T03:17:42Z

run buildall

github-actions

Request changes for two resource regressions: eager CLucene norms remain outside searcher-cache accounting (P1), and sparse hidden-row intersections repeatedly allocate segment-sized bitmaps (P2). Both are detailed inline. No existing inline findings were supplied.

Static review covered all 21 changed files, the scanner/read-source chain, both index formats, and the exact pinned CLucene norm implementation. The read flow is: captured rowsets + bitmap/version → scanner/shared tablet statistics → per-segment hidden IDs → document/term/norm totals → BM25 scoring.

Required checkpoint	Conclusion
Goal and proof	Bitmap/delete-sign-hidden MoW rows are removed consistently from document counts and term frequencies; norm-based lengths address replica compaction differences. New unit/regression sources exercise this goal; execution was not performed.
Scope, clarity and reuse	Changes stay within BM25 statistics and reuse common IndexSource helpers with backend-specific overrides.
Concurrency and locks	Scanner-local results and shared first-collector locking are consistent. The new sum mutex serializes its initial I/O; CLucene reader/norm locks have no identified reverse acquisition path. No new thread or deadlock found.
Lifecycle and static initialization	Cached searchers retain readers and captured rowsets remain protected during collection. Borrowed opening I/O context is cleared. Retained norm accounting is the P1 finding; no new static-initialization dependency found.
Configuration and dynamic changes	No configuration added. Existing query file-cache, segment-cache and page-cache choices are forwarded on the changed paths; CLucene norms retain the existing context-free API behavior.
Symbol/storage compatibility	Internal signatures/default virtual methods change; no persisted index or wire format changes. Existing missing-norm rejection and no-bitmap behavior remain.
Thrift/protobuf IDs	Not applicable: no generated definitions changed.
Parallel and special paths	CLucene/SNII, split scanners, physical segment IDs, reserved scoring terms, NULLs, no-norm indexes and all-hidden tablets checked. No additional correctness issue confirmed.
Conditional checks	Empty inputs, invalid row IDs, zone-map sign pruning and all-hidden handling follow concrete cases and propagate failures.
Functional/negative/unit coverage	New tests cover compaction equivalence, NULL/hidden/all-hidden rows, sparse/dense cursors, invalid IDs, cache computation errors, multiple readers and I/O-context reset. The new checked-in end-to-end suites do not explicitly select SNII; its hidden-row behavior was reviewed statically.
Changed test results	Updated BM25 score and new golden outputs are consistent with the input rows and BM25 arithmetic. Their generation provenance and runtime reproducibility were not verified.
Observability	Existing collection VLOG and error propagation remain available. Local delete-sign read counters are not merged into the parent profile; retained norms also escape cache metrics as described inline.
Transactions/persistence/failover	No transaction, EditLog or metadata-write changes. Read-version and aggregate delete-bitmap keys match the normal rowset reader.
Writes/atomicity/crashes	No new data writes. I/O/cursor failures clear partial statistics and the shared collector returns the same failure to all callers; memory exhaustion risk is P1.
FE/BE variable propagation	No new transmitted variables. FE selected-index schema includes hidden delete-sign metadata; existing cache options reach the new reader context.
CPU/memory/performance	One-time norm totals and tablet sharing avoid redundant collection; the two inline findings identify remaining unbounded retained memory and repeated sparse-intersection work.
Other issues	Final static recheck found no additional substantiated defect.

No additional user focus was supplied. Disclosed scope limits remain: light-delete predicates, merge-on-read/duplicate-key delete visibility, skip_delete_sign/show_hidden_columns, non-NULL zero-token encodings, and statistics remaining per tablet. These were considered and are not duplicate inline findings. No builds, tests, format checks or runtime experiments were run, as this invocation permits static review only.

Review converged after 2 round(s).

github-actions · 2026-10-24T03:30:54Z

+#ifdef BE_TEST
+                RETURN_IF_ERROR(sum_lengths(&non_null_length_sum));
+#else
+                RETURN_IF_ERROR(inverted_index_cache_handle.get<CluceneSearcherCacheValue>()


[P1] Account for the norms retained by statistics collection

This callback now loads norms for every segment in a MoW tablet's captured read set before normal key/zone-map pruning, even when its hidden-row set is empty. CLucene's SegmentReader::norms() allocates and retains one byte per document per scored field, but these searchers were inserted using only getTermInfosRAMUsed() and neither their cache charge nor tracked bytes are updated afterward. A selective query over a large, low-vocabulary tablet can therefore leave gigabytes of norms resident while the searcher cache accounts for only small dictionaries; repeated queries across tablets can exhaust BE memory despite the configured cache limit. Previously such pruned segments only contributed dictionary statistics and did not load these norms. Please reserve/track the added buffers and update the cache charge, including for searchers already in the cache, or compute the sum without retaining unaccounted norm buffers.

Fixed in 96a0b49: the statistics no longer ask CLucene readers for their retained norms. They read each segment's norms into their own buffer via IndexReader::norms(field, bytes) and free it after use. Only small sums are cached with the searcher: the per-field non-NULL norm-length sum, and the hidden rows' length sum for the last few sets of hidden rows per field (queries at one read version reuse it and read no norms). Measured on a 10M-row tablet with 4 scored fields and a key-pruned query: memory growth went from about +51 MB to +12–15 MB, the same as before the PR. Covered by HiddenNormLengthSumIsComputedOncePerRowSet and NormLengthSumsReadEachSegment.

github-actions · 2026-10-24T03:30:54Z

+// Decodes every posting once and tests it against a bitset of `docs`.
+Status count_listed_docs_by_scan(PostingsCursor& cursor, std::span<const uint32_t> docs,
+                                 uint64_t* out) {
+    std::vector<uint64_t> bits(docs.back() / 64 + 1, 0);


[P2] Avoid a segment-sized bitmap for sparse term intersections

The scan branch is selected from docs.size() and doc_freq(), but this allocation scales with the largest hidden row ID. For example, hidden_docs = {9999999} and a term with one posting select this branch and zero 1,250,000 bytes just to compare two IDs. CLucene also returns a non-null cursor for an absent term, so df = 0 incurs the same allocation before discovering EOF. Collection statistics repeat this for every query term and every segment with hidden rows on every query, making selective or many-term searches perform substantial segment-sized allocation/zeroing even with tiny or empty posting lists. Please use a sparse/sorted intersection for this case, account for the maximum ID when choosing a dense bitset, and short-circuit zero-frequency terms.

Fixed in 96a0b49: count_listed_docs no longer allocates a bitset. The scan path walks the hidden rows and the postings together (both ascending) and stops once every hidden row is passed, so the cost no longer depends on the largest row ID. A term with doc_freq() == 0 returns before opening anything. Covered by CountListedDocsScansFarDocsWithoutWalkingEveryBlock (IDs up to 9,999,999) and CountListedDocsOfNothing.

Asking a CLucene segment reader for a field's norms makes it keep them for the cached searcher's lifetime, uncharged to the searcher cache, and the merge-on-write statistics visit every segment of the read set, including the segments the scan prunes and never scores. Read the norms into a buffer of the statistics' own, one segment at a time, and keep only sums with the searcher: the field's non-NULL norm lengths, and the hidden rows' lengths for the last few sets of hidden rows, so queries at one read version read no norms. Count the hidden rows a term's postings list by walking both ascending lists together instead of zeroing a bitset as large as the largest hidden row, and skip terms without postings.

hello-stephen · 2026-10-24T04:08:39Z

TPC-H: Total hot run time: 28796 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7830cb129cb21c4d5ef045aef00dcb996c1db36a, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17621	5185	5206	5185
q2	2119	316	281	281
q3	10197	1366	828	828
q4	4680	461	337	337
q5	7430	810	525	525
q6	179	182	149	149
q7	749	793	615	615
q8	9259	1280	1028	1028
q9	6120	4244	4184	4184
q10	6834	1313	1014	1014
q11	417	247	238	238
q12	640	400	284	284
q13	18065	3572	2958	2958
q14	267	271	248	248
q15	q16	709	713	646	646
q17	1180	745	582	582
q18	7054	6180	6097	6097
q19	1107	944	615	615
q20	383	325	222	222
q21	4971	2487	2498	2487
q22	374	318	273	273
Total cold run time: 100355 ms
Total hot run time: 28796 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5972	5907	5924	5907
q2	744	605	544	544
q3	5540	5786	5256	5256
q4	2022	2132	1558	1558
q5	5977	5559	5710	5559
q6	249	194	145	145
q7	2289	1974	1775	1775
q8	3010	2621	2671	2621
q9	8051	7510	7527	7510
q10	3652	3579	3184	3184
q11	591	410	386	386
q12	655	699	497	497
q13	3209	3644	3050	3050
q14	283	288	264	264
q15	q16	688	686	608	608
q17	7719	7018	6898	6898
q18	12989	12152	12843	12152
q19	881	787	795	787
q20	2151	2151	1931	1931
q21	5617	4594	4736	4594
q22	485	454	407	407
Total cold run time: 72774 ms
Total hot run time: 65633 ms

hello-stephen · 2026-10-24T04:18:28Z

TPC-DS: Total hot run time: 150742 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7830cb129cb21c4d5ef045aef00dcb996c1db36a, data reload: false

query5	4306	596	435	435
query6	428	212	190	190
query7	4809	436	225	225
query8	318	164	158	158
query9	8760	3094	3094	3094
query10	481	308	260	260
query11	5933	3884	3571	3571
query12	147	90	87	87
query13	1233	463	330	330
query14	6110	3910	3624	3624
query14_1	3434	3452	3378	3378
query15	205	196	175	175
query16	955	403	423	403
query17	868	664	533	533
query18	2424	429	320	320
query19	193	169	140	140
query20	87	83	84	83
query21	207	126	111	111
query22	13046	13029	12946	12946
query23	13402	12935	12463	12463
query23_1	12555	12487	12489	12487
query24	7416	823	461	461
query24_1	496	481	471	471
query25	537	424	360	360
query26	1278	263	142	142
query27	2804	451	283	283
query28	5054	1971	1967	1967
query29	1591	600	469	469
query30	311	219	188	188
query31	1029	926	820	820
query32	145	96	94	94
query33	576	305	243	243
query34	942	845	489	489
query35	764	773	645	645
query36	811	813	735	735
query37	129	106	81	81
query38	1787	1734	1676	1676
query39	792	787	761	761
query39_1	735	740	736	736
query40	216	122	97	97
query41	72	70	68	68
query42	88	90	85	85
query43	342	367	309	309
query44	1309	679	691	679
query45	181	179	164	164
query46	823	974	564	564
query47	3013	2841	2973	2841
query48	336	313	210	210
query49	565	388	293	293
query50	670	278	212	212
query51	10559	10224	10632	10224
query52	79	77	69	69
query53	191	203	144	144
query54	251	190	174	174
query55	74	71	63	63
query56	231	204	208	204
query57	1666	1629	1578	1578
query58	284	266	259	259
query59	2082	2144	1946	1946
query60	287	244	222	222
query61	151	149	146	146
query62	384	336	286	286
query63	186	158	163	158
query64	2733	972	770	770
query65	3323	3275	3250	3250
query66	1773	419	293	293
query67	20219	19961	19971	19961
query68	2977	1022	601	601
query69	408	282	257	257
query70	850	769	773	769
query71	301	236	203	203
query72	2770	2604	2318	2318
query73	506	518	293	293
query74	4623	4495	4364	4364
query75	2316	2308	1917	1917
query76	2335	1006	583	583
query77	338	368	272	272
query78	9804	9679	9203	9203
query79	989	855	519	519
query80	1215	417	370	370
query81	580	321	270	270
query82	591	136	110	110
query83	305	192	179	179
query84	308	122	106	106
query85	854	415	369	369
query86	394	241	231	231
query87	2014	1919	1811	1811
query88	3679	2657	2620	2620
query89	327	277	238	238
query90	1868	170	167	167
query91	158	147	128	128
query92	100	90	86	86
query93	1121	997	567	567
query94	640	290	266	266
query95	575	331	374	331
query96	666	487	238	238
query97	2430	2420	2332	2332
query98	158	148	142	142
query99	768	761	644	644
Total cold run time: 236082 ms
Total hot run time: 150742 ms

hello-stephen · 2026-10-24T04:23:20Z

ClickBench: Total hot run time: 24.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7830cb129cb21c4d5ef045aef00dcb996c1db36a, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.04
query3	0.26	0.10	0.09
query4	1.61	0.10	0.11
query5	0.23	0.24	0.23
query6	1.18	0.68	0.68
query7	0.03	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.34	0.36
query10	0.58	0.58	0.57
query11	0.20	0.15	0.14
query12	0.19	0.14	0.14
query13	0.50	0.49	0.49
query14	1.02	1.02	0.99
query15	0.63	0.60	0.59
query16	0.36	0.34	0.35
query17	1.13	1.19	1.16
query18	0.20	0.19	0.20
query19	2.14	1.94	1.98
query20	0.02	0.01	0.01
query21	15.48	0.21	0.14
query22	4.82	0.05	0.05
query23	16.28	0.34	0.12
query24	3.06	0.55	0.47
query25	0.12	0.04	0.05
query26	1.13	0.20	0.15
query27	0.06	0.05	0.04
query28	3.96	0.78	0.40
query29	12.47	4.10	3.23
query30	0.28	0.13	0.10
query31	2.77	0.56	0.31
query32	3.23	0.60	0.50
query33	3.10	3.24	3.26
query34	15.38	4.00	3.37
query35	3.34	3.34	3.37
query36	0.54	0.43	0.43
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.15
query41	0.08	0.03	0.02
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 97.38 s
Total hot run time: 24.39 s

…w-statistics

…w BM25 statistics The merge-on-write cases overwrite and delete rows, which no longer count in the BM25 statistics, so their scores are those of the visible rows. The same rows are returned; only their scores change.

hello-stephen · 2026-10-24T06:32:58Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.15% (35927/46565)
Line Coverage	62.45% (405966/650088)
Region Coverage	58.89% (342979/582437)
Branch Coverage	59.88% (158715/265057)

airborne12 · 2026-10-24T08:42:14Z

run buildall

airborne12 · 2026-10-24T08:42:20Z

/review

github-actions

One new P2 finding: temporary hidden/NULL row-ID arrays bypass query memory accounting before scan pruning. It is detailed inline. No P0/P1 issue was independently confirmed for this head.

Both existing threads were revalidated: retained CLucene norms (4365847990) is fixed for the reviewed Doris writer/reader paths by using caller-owned norm buffers and bounded scalar caches; sparse intersection allocation (4365847994) is fixed by sorted intersection and zero-frequency exits. The new finding concerns temporary uint32 row-ID arrays, distinct from retained norm buffers. No existing blocker is carried forward.

Static coverage includes all 22 changed files and their scan, reader, writer, cache and scoring interactions at a690a60. The invocation chain is:

captured rowsets + bitmap/read version
  -> scanner/shared tablet statistics
  -> per-segment bitmap/delete-sign hidden IDs
  -> CLucene or SNII visible document/term/norm totals
  -> scanner-local BM25 statistics and matched-row scores

Required checkpoint	Conclusion
Goal and tests	Bitmap/delete-sign-hidden MoW rows are removed from document counts, term frequencies and decoded norm totals. Replica/fresh-table, NULL and all-hidden fixtures support compaction invariance statically; the new memory issue remains.
Scope, clarity and reuse	Changes stay within BM25 collection, source helpers, cache sums and associated tests; existing captured read sources and segment/column readers are reused.
Concurrency and locks	Scanner tasks and queries can collect concurrently. TabletCollectionStatistics locks once-only collection and copies results into scanner-local maps; one searcher mutex protects sum caches. Its initial I/O is serialized intentionally, with no verified reverse lock order or deadlock.
Lifecycle and static initialization	Rowset readers and cache handles preserve reader lifetimes. Synchronous callbacks retain no query references after return; local resources release on errors. No new static initialization dependency or ownership cycle was found. Caller-owned norms fix the historical retention path; untracked ID buffers are the inline finding.
Configuration and dynamic changes	No configuration is added. Existing segment/page/file-cache options are forwarded where the read APIs accept them; the existing context-free CLucene norm API remains a limitation.
Symbols, formats and rolling compatibility	Internal collection signatures and default virtual helpers have matching callers. No persisted index or wire format changes; old/default delete-sign columns resolve by UID, and existing missing-norm scoring rejection remains.
Thrift/protobuf field IDs	Not applicable: no generated definition changed.
Parallel and special paths	Serial/shared split scanners, CLucene/SNII, physical cloud segment IDs, reserved scoring terms, NULL/empty inputs, no-norm keyword indexes and all-hidden collections were traced. Explicit deletion/debug exceptions are disclosed below.
Conditional checks	Bitmap presence, zero frequency, row bounds, zone-map pruning and all-hidden handling have concrete scenarios. Roaring supplies sorted distinct IDs; bounds/frequency failures return errors instead of silently continuing.
Functional, negative and unit coverage	New tests cover hidden/compacted equivalence, NULL and hidden NULL, all-hidden/refill, sparse/dense postings, distant IDs, invalid row IDs, cache callback errors, multiple readers and I/O flags. Memory-limit and concurrent-cache coverage is absent; dedicated reserved-term and schema/segment-ID edge fixtures are limited, with those paths reviewed statically.
Changed test results	Every new/modified numeric expectation is consistent with the input corpus and BM25 arithmetic, including .8025915, .60996956/.8713851 and 1.1823694. Output-generation provenance and execution were not verified.
Observability	Existing collection timing, query/tablet/term VLOG and error propagation remain available. Temporary raw arrays escape query memory metrics as described inline; context-free CLucene norms also retain the existing I/O-accounting limitation.
Transactions, persistence and failover	Read-only collection uses the captured rowsets and the same versioned physical-segment bitmap keys as ordinary reads. No transaction, EditLog, persistence or failover protocol changes.
Writes, atomicity and crash behavior	No data-write path changes. Failed collection clears partial statistics and shares the error with other scanners; cache sums are inserted only after successful computation. The verified resource-accounting risk is inline.
FE/BE variable propagation	No transmitted variables are added. Existing cache options reach the new scanner context.
CPU, memory and performance	Tablet sharing, zone-map delete-sign pruning, bounded batches and cached norm sums reduce repeated work. Sparse posting intersection no longer allocates by maximum ID. Full-cardinality untracked ID buffers remain the new P2 issue.
Further issues	Final rechecks of cursor progress, norm ownership/overloads, NULL writer metadata, TopN masks, cached-reader lifetimes and parallel paths found no additional substantiated defect.

No additional user focus was supplied. Declared scope limits remain: enable_mow_light_delete predicates are still counted until compaction; skip_delete_sign/show_hidden_columns do not change the exclusion of delete-sign rows; skip_delete_bitmap and non-MoW tables retain their existing statistics behavior; non-NULL zero-token encodings remain approximate; statistics remain per tablet. CLucene caller-buffer behavior was checked against the pinned SegmentReader implementation, with Doris NULL-writer metadata traced to exclude a synthetic missing-field branch.

No builds, tests, format checks or Doris runtime experiments were run under this static-only review contract.

Review converged after 2 round(s).

hello-stephen · 2026-10-24T09:12:51Z

TPC-H: Total hot run time: 28525 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a690a608ebdbf0e16432a1d290ccfd8dbbbd31d5, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17591	5141	5082	5082
q2	2137	299	249	249
q3	10272	1376	821	821
q4	4678	453	336	336
q5	7433	809	533	533
q6	186	196	159	159
q7	758	787	595	595
q8	9251	1300	1102	1102
q9	5357	4201	4189	4189
q10	6838	1302	1015	1015
q11	427	251	233	233
q12	638	403	284	284
q13	18079	3550	3023	3023
q14	280	274	254	254
q15	q16	700	719	689	689
q17	1198	760	586	586
q18	7064	6163	6135	6135
q19	1101	955	622	622
q20	395	336	230	230
q21	4528	2140	2259	2140
q22	346	290	248	248
Total cold run time: 99257 ms
Total hot run time: 28525 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5645	5771	5547	5547
q2	666	567	545	545
q3	5213	5410	4939	4939
q4	1970	2175	1543	1543
q5	5290	5157	5499	5157
q6	253	204	169	169
q7	2411	2108	1761	1761
q8	3063	2743	2762	2743
q9	8095	7946	7965	7946
q10	3686	3658	3245	3245
q11	605	437	445	437
q12	856	719	543	543
q13	3510	3929	3200	3200
q14	319	333	297	297
q15	q16	708	711	679	679
q17	8445	7476	6997	6997
q18	13014	12203	12856	12203
q19	952	843	871	843
q20	2153	2142	1915	1915
q21	5813	4689	4827	4689
q22	486	459	413	413
Total cold run time: 73153 ms
Total hot run time: 65811 ms

hello-stephen · 2026-10-24T09:22:37Z

TPC-DS: Total hot run time: 150260 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a690a608ebdbf0e16432a1d290ccfd8dbbbd31d5, data reload: false

query5	4307	596	446	446
query6	432	203	191	191
query7	4800	435	225	225
query8	309	156	152	152
query9	8776	3135	3084	3084
query10	477	312	247	247
query11	6047	3876	3570	3570
query12	143	97	86	86
query13	1270	440	323	323
query14	6148	3958	3663	3663
query14_1	3411	3422	3422	3422
query15	201	198	181	181
query16	944	430	399	399
query17	853	663	528	528
query18	2417	433	313	313
query19	217	174	140	140
query20	85	84	85	84
query21	212	130	106	106
query22	12989	12993	12787	12787
query23	13394	12808	12351	12351
query23_1	12622	12484	12742	12484
query24	7360	816	455	455
query24_1	467	466	473	466
query25	531	423	359	359
query26	1259	263	140	140
query27	2888	444	276	276
query28	5266	1984	1997	1984
query29	1586	632	470	470
query30	299	229	186	186
query31	1038	936	821	821
query32	144	98	93	93
query33	525	307	248	248
query34	906	862	531	531
query35	755	772	660	660
query36	808	809	701	701
query37	127	104	85	85
query38	1842	1784	1677	1677
query39	801	817	758	758
query39_1	755	746	761	746
query40	221	120	98	98
query41	72	68	71	68
query42	91	85	84	84
query43	357	392	324	324
query44	1363	706	715	706
query45	184	183	164	164
query46	855	994	582	582
query47	2769	2877	2627	2627
query48	354	315	216	216
query49	566	402	292	292
query50	696	277	216	216
query51	10419	10570	10211	10211
query52	77	81	68	68
query53	204	209	152	152
query54	227	189	180	180
query55	74	67	63	63
query56	244	229	221	221
query57	1665	1613	1577	1577
query58	287	268	252	252
query59	2111	2118	1889	1889
query60	284	226	222	222
query61	147	144	143	143
query62	391	325	290	290
query63	191	155	169	155
query64	2743	953	749	749
query65	3319	3276	3268	3268
query66	1762	409	294	294
query67	20045	19920	19950	19920
query68	3052	1004	620	620
query69	373	288	245	245
query70	837	756	721	721
query71	292	228	218	218
query72	2675	2573	2282	2282
query73	506	519	298	298
query74	4622	4484	4360	4360
query75	2321	2282	1904	1904
query76	2307	1001	594	594
query77	336	374	273	273
query78	9863	9752	9151	9151
query79	975	892	527	527
query80	540	406	335	335
query81	549	313	264	264
query82	576	134	107	107
query83	302	216	183	183
query84	316	123	106	106
query85	793	413	366	366
query86	329	237	222	222
query87	1999	1956	1817	1817
query88	3649	2666	2652	2652
query89	327	285	242	242
query90	1878	175	173	173
query91	157	144	121	121
query92	87	92	92	92
query93	993	1002	568	568
query94	460	293	277	277
query95	572	330	361	330
query96	675	473	232	232
query97	2453	2414	2347	2347
query98	166	151	147	147
query99	780	758	635	635
Total cold run time: 234642 ms
Total hot run time: 150260 ms

hello-stephen · 2026-10-24T09:27:31Z

ClickBench: Total hot run time: 24.53 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a690a608ebdbf0e16432a1d290ccfd8dbbbd31d5, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.06	0.05
query3	0.26	0.08	0.09
query4	1.61	0.10	0.09
query5	0.24	0.24	0.25
query6	1.18	0.68	0.69
query7	0.03	0.01	0.01
query8	0.05	0.04	0.04
query9	0.39	0.33	0.35
query10	0.58	0.57	0.60
query11	0.22	0.14	0.14
query12	0.18	0.16	0.14
query13	0.50	0.49	0.50
query14	1.02	1.00	1.00
query15	0.62	0.60	0.62
query16	0.34	0.34	0.35
query17	1.13	1.16	1.16
query18	0.21	0.19	0.20
query19	2.13	2.02	1.98
query20	0.01	0.01	0.01
query21	15.49	0.21	0.15
query22	4.84	0.06	0.06
query23	16.29	0.33	0.13
query24	2.80	0.46	0.67
query25	0.10	0.08	0.06
query26	1.65	0.21	0.17
query27	0.05	0.03	0.05
query28	4.02	0.68	0.37
query29	12.48	4.20	3.32
query30	0.27	0.14	0.11
query31	2.78	0.55	0.32
query32	3.22	0.60	0.50
query33	3.17	3.26	3.22
query34	15.28	3.97	3.38
query35	3.34	3.38	3.35
query36	0.55	0.45	0.43
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.14	0.14
query41	0.09	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 97.64 s
Total hot run time: 24.53 s

hello-stephen · 2026-10-24T11:41:14Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.13% (35918/46568)
Line Coverage	62.42% (405819/650122)
Region Coverage	58.92% (343168/582441)
Branch Coverage	59.85% (158629/265058)

…mory The hidden and NULL row ID lists the merge-on-write statistics build per segment are as large as the segment's hidden or NULL rows, and were plain vectors that Doris memory tracking does not see. Use tracked vectors, charge the SNII reader's NULL ID vector while it lives, and sum the hidden rows as they are when the segment has no NULL rows instead of copying them.

guptashivam04 · 2026-10-25T02:33:39Z

run buildall

guptashivam04 · 2026-10-25T02:33:55Z

/review

hello-stephen · 2026-10-25T02:50:11Z

TPC-H: Total hot run time: 28362 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit da1c9d25f02aed03157e26f771415407997769d5, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17596	5163	5059	5059
q2	2140	314	247	247
q3	10257	1407	832	832
q4	4680	465	336	336
q5	7430	804	524	524
q6	184	183	152	152
q7	749	800	600	600
q8	9255	1347	1031	1031
q9	5501	4219	4212	4212
q10	6814	1288	1030	1030
q11	428	246	226	226
q12	632	398	277	277
q13	18097	3617	2978	2978
q14	274	288	260	260
q15	q16	723	723	674	674
q17	1192	744	559	559
q18	7086	6182	6122	6122
q19	1263	963	604	604
q20	400	332	224	224
q21	5857	2165	2320	2165
q22	348	306	250	250
Total cold run time: 100906 ms
Total hot run time: 28362 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5599	5636	5633	5633
q2	689	564	543	543
q3	5210	5411	4903	4903
q4	1995	2134	1546	1546
q5	5302	5191	5830	5191
q6	246	194	151	151
q7	2304	2079	1846	1846
q8	3088	2793	2688	2688
q9	8120	8061	7919	7919
q10	3736	3661	3288	3288
q11	610	442	430	430
q12	693	696	541	541
q13	3469	3870	3185	3185
q14	309	323	298	298
q15	q16	698	710	653	653
q17	8321	7611	7323	7323
q18	12987	12164	12958	12164
q19	907	787	789	787
q20	2128	2149	1913	1913
q21	5664	4694	4670	4670
q22	480	446	419	419
Total cold run time: 72555 ms
Total hot run time: 66091 ms

hello-stephen · 2026-10-25T02:59:57Z

TPC-DS: Total hot run time: 150552 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit da1c9d25f02aed03157e26f771415407997769d5, data reload: false

query5	4305	575	459	459
query6	422	212	195	195
query7	4814	436	222	222
query8	317	180	150	150
query9	8777	3090	3091	3090
query10	473	316	260	260
query11	5879	3832	3596	3596
query12	145	91	85	85
query13	1280	476	322	322
query14	6162	3974	3630	3630
query14_1	3425	3423	3425	3423
query15	201	201	183	183
query16	974	421	394	394
query17	866	651	536	536
query18	2426	429	312	312
query19	197	170	139	139
query20	84	85	84	84
query21	208	123	105	105
query22	12906	13064	12778	12778
query23	13478	12826	12552	12552
query23_1	12500	12524	12509	12509
query24	7350	813	464	464
query24_1	478	485	480	480
query25	546	424	368	368
query26	1267	265	142	142
query27	2764	439	275	275
query28	4634	1943	1943	1943
query29	1591	614	462	462
query30	308	224	187	187
query31	992	936	812	812
query32	150	94	94	94
query33	529	312	265	265
query34	907	855	501	501
query35	733	741	644	644
query36	814	802	737	737
query37	133	100	84	84
query38	1756	1748	1655	1655
query39	784	781	748	748
query39_1	730	724	740	724
query40	222	119	99	99
query41	72	71	68	68
query42	88	86	85	85
query43	340	361	306	306
query44	1344	696	693	693
query45	182	179	168	168
query46	834	973	600	600
query47	2936	3022	2677	2677
query48	288	298	216	216
query49	567	404	290	290
query50	680	286	222	222
query51	10455	10322	10204	10204
query52	76	80	69	69
query53	187	209	148	148
query54	245	206	185	185
query55	72	66	60	60
query56	234	220	214	214
query57	1637	1612	1558	1558
query58	284	263	257	257
query59	2118	2191	1983	1983
query60	281	240	225	225
query61	144	146	146	146
query62	386	341	283	283
query63	190	152	159	152
query64	2803	920	759	759
query65	3342	3268	3254	3254
query66	1782	412	307	307
query67	20256	20255	19903	19903
query68	3078	990	621	621
query69	382	286	247	247
query70	807	748	768	748
query71	298	235	210	210
query72	2656	2605	2315	2315
query73	515	529	290	290
query74	4635	4476	4360	4360
query75	2340	2302	1922	1922
query76	2326	1023	599	599
query77	345	359	271	271
query78	9748	9738	9147	9147
query79	970	901	519	519
query80	625	416	348	348
query81	552	309	273	273
query82	578	135	104	104
query83	261	196	184	184
query84	314	125	97	97
query85	872	431	375	375
query86	333	239	240	239
query87	2017	1918	1779	1779
query88	3639	2673	2665	2665
query89	328	284	236	236
query90	1806	182	168	168
query91	168	147	124	124
query92	94	92	105	92
query93	994	999	572	572
query94	465	302	272	272
query95	576	329	313	313
query96	662	488	230	230
query97	2472	2437	2324	2324
query98	166	167	149	149
query99	770	759	641	641
Total cold run time: 233854 ms
Total hot run time: 150552 ms

hello-stephen · 2026-10-25T03:04:50Z

ClickBench: Total hot run time: 24.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit da1c9d25f02aed03157e26f771415407997769d5, data reload: false

query1	0.01	0.01	0.00
query2	0.08	0.05	0.05
query3	0.26	0.10	0.09
query4	1.62	0.10	0.10
query5	0.23	0.25	0.26
query6	1.17	0.70	0.69
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.40	0.34	0.34
query10	0.60	0.58	0.56
query11	0.20	0.15	0.14
query12	0.18	0.14	0.14
query13	0.49	0.50	0.49
query14	1.02	1.02	1.01
query15	0.62	0.60	0.61
query16	0.34	0.35	0.30
query17	1.11	1.14	1.14
query18	0.21	0.20	0.19
query19	2.08	1.94	2.01
query20	0.02	0.02	0.01
query21	15.52	0.20	0.14
query22	4.76	0.06	0.05
query23	16.28	0.33	0.13
query24	2.96	0.67	0.44
query25	0.12	0.04	0.04
query26	1.63	0.21	0.17
query27	0.06	0.03	0.03
query28	3.68	0.83	0.39
query29	12.47	4.15	3.30
query30	0.28	0.14	0.09
query31	2.77	0.55	0.33
query32	3.22	0.59	0.51
query33	3.14	3.14	3.29
query34	15.61	4.00	3.39
query35	3.36	3.32	3.33
query36	0.54	0.44	0.44
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.04	0.04
Total cold run time: 97.64 s
Total hot run time: 24.35 s

…w-statistics # Conflicts: # be/test/storage/index/inverted/reader/collection_statistics_test.cpp

guptashivam04 · 2026-10-25T03:42:58Z

run buildall

guptashivam04 · 2026-10-25T03:43:09Z

/review

hello-stephen · 2026-10-25T04:30:43Z

TPC-H: Total hot run time: 28390 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0d993147352ec34120c93871c266be1e34cc22f, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17601	5270	5044	5044
q2	2164	316	266	266
q3	10222	1366	838	838
q4	4676	466	337	337
q5	7447	811	527	527
q6	182	189	150	150
q7	757	793	605	605
q8	9258	1324	1041	1041
q9	5464	4181	4322	4181
q10	6816	1290	1043	1043
q11	459	251	238	238
q12	639	397	278	278
q13	18089	3911	3016	3016
q14	276	274	241	241
q15	q16	725	715	672	672
q17	1184	754	584	584
q18	7042	6160	6138	6138
q19	1114	966	592	592
q20	383	343	229	229
q21	5028	2116	2321	2116
q22	347	295	254	254
Total cold run time: 99873 ms
Total hot run time: 28390 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5620	5635	5624	5624
q2	668	587	544	544
q3	5249	5367	4938	4938
q4	2005	2141	1533	1533
q5	5255	5166	5680	5166
q6	251	199	153	153
q7	2303	1997	1867	1867
q8	3030	2719	2643	2643
q9	8104	7898	7835	7835
q10	3707	3642	3267	3267
q11	632	433	519	433
q12	685	709	536	536
q13	3528	3903	3258	3258
q14	321	309	311	309
q15	q16	701	740	645	645
q17	8165	7493	6933	6933
q18	13036	12164	12905	12164
q19	890	776	782	776
q20	2135	2132	1915	1915
q21	5711	4635	4689	4635
q22	490	434	409	409
Total cold run time: 72486 ms
Total hot run time: 65583 ms

hello-stephen · 2026-10-25T04:40:30Z

TPC-DS: Total hot run time: 150312 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0d993147352ec34120c93871c266be1e34cc22f, data reload: false

query5	4311	576	432	432
query6	429	199	207	199
query7	4824	433	216	216
query8	319	164	153	153
query9	8815	3040	3072	3040
query10	470	311	241	241
query11	5981	3845	3560	3560
query12	142	93	86	86
query13	1260	454	327	327
query14	6098	3919	3641	3641
query14_1	3397	3396	3410	3396
query15	212	198	183	183
query16	944	403	425	403
query17	861	648	551	551
query18	2425	424	317	317
query19	195	167	134	134
query20	85	85	83	83
query21	210	123	106	106
query22	12898	13104	12836	12836
query23	13534	12870	12448	12448
query23_1	12605	12652	12485	12485
query24	7419	808	479	479
query24_1	467	471	473	471
query25	530	418	361	361
query26	1261	254	146	146
query27	2761	438	265	265
query28	4624	1962	1934	1934
query29	1571	596	454	454
query30	308	233	184	184
query31	1005	925	806	806
query32	148	90	90	90
query33	527	297	249	249
query34	878	864	498	498
query35	724	752	655	655
query36	810	834	720	720
query37	119	98	81	81
query38	1765	1753	1658	1658
query39	794	780	742	742
query39_1	730	724	739	724
query40	227	118	98	98
query41	73	70	69	69
query42	90	84	82	82
query43	342	361	311	311
query44	1289	677	681	677
query45	181	177	168	168
query46	828	955	551	551
query47	2949	2875	2755	2755
query48	285	302	218	218
query49	574	397	286	286
query50	667	273	214	214
query51	10302	10366	10273	10273
query52	77	85	68	68
query53	187	212	150	150
query54	242	187	178	178
query55	76	65	64	64
query56	242	249	213	213
query57	1642	1619	1558	1558
query58	274	259	255	255
query59	2170	2136	1999	1999
query60	274	238	223	223
query61	146	148	144	144
query62	392	343	295	295
query63	184	156	154	154
query64	2734	955	761	761
query65	3325	3268	3256	3256
query66	1779	412	294	294
query67	20352	20185	20072	20072
query68	2993	984	606	606
query69	390	280	240	240
query70	812	783	733	733
query71	301	220	202	202
query72	2724	2627	2286	2286
query73	513	519	293	293
query74	4613	4506	4328	4328
query75	2324	2282	1893	1893
query76	2306	1012	577	577
query77	351	365	271	271
query78	9794	9720	9166	9166
query79	976	832	504	504
query80	1224	406	353	353
query81	587	323	282	282
query82	613	128	125	125
query83	312	191	174	174
query84	314	115	99	99
query85	867	416	355	355
query86	401	243	221	221
query87	2004	1922	1807	1807
query88	3653	2643	2634	2634
query89	332	276	239	239
query90	1918	164	163	163
query91	163	157	125	125
query92	100	87	79	79
query93	993	977	566	566
query94	634	296	262	262
query95	559	380	289	289
query96	662	475	230	230
query97	2473	2443	2341	2341
query98	158	146	146	146
query99	769	748	652	652
Total cold run time: 234941 ms
Total hot run time: 150312 ms

hello-stephen · 2026-10-25T04:45:21Z

ClickBench: Total hot run time: 24.33 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0d993147352ec34120c93871c266be1e34cc22f, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.26	0.09	0.08
query4	1.61	0.11	0.11
query5	0.23	0.24	0.24
query6	1.17	0.69	0.67
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.40	0.37	0.36
query10	0.57	0.59	0.57
query11	0.19	0.14	0.14
query12	0.18	0.14	0.14
query13	0.48	0.51	0.49
query14	1.05	1.03	1.01
query15	0.63	0.60	0.60
query16	0.34	0.36	0.34
query17	1.10	1.15	1.14
query18	0.19	0.19	0.18
query19	2.13	1.99	1.95
query20	0.02	0.02	0.01
query21	15.49	0.19	0.14
query22	4.88	0.05	0.05
query23	16.26	0.34	0.12
query24	3.04	0.53	0.47
query25	0.09	0.06	0.06
query26	1.32	0.22	0.16
query27	0.04	0.03	0.05
query28	3.48	0.73	0.35
query29	12.52	4.10	3.27
query30	0.28	0.15	0.10
query31	2.77	0.55	0.32
query32	3.22	0.59	0.51
query33	3.16	3.15	3.24
query34	15.31	3.98	3.39
query35	3.33	3.31	3.34
query36	0.56	0.43	0.40
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.99 s
Total hot run time: 24.33 s

hello-stephen · 2026-10-25T07:13:58Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.18% (35878/46486)
Line Coverage	62.44% (405269/649073)
Region Coverage	58.93% (342685/581507)
Branch Coverage	59.91% (158535/264636)

airborne12 · 2026-10-25T11:33:47Z

MoW light deletes still include deleted rows in BM25 statistics (V2/V3/SNII). The collector does not evaluate delete predicates with enable_mow_light_delete=true. Replica compaction can therefore change scores. Code.
Delete-sign options cause inconsistent statistics (V2/V3/SNII). With skip_delete_sign=true or show_hidden_columns=true, queries can return delete-sign rows, but the collector excludes them. Code.
SNII repeats full norm scans. Each MoW scoring query recomputes lengths for fields with norms, even at the same read version. Code.

Add regression tests for both visibility gaps across V2/V3/SNII. Cache the SNII sums and measure repeated-query cost.

Code inspection at a0d9931; no new runtime tests.

airborne12 · 2026-10-25T11:55:21Z

/review

github-actions

Static review of all 22 changed files and related scan, storage, scoring and cache paths at a0d9931 found two new P2 issues: SNII selected-row norm summation allocates a segment-proportional, untracked output array, and bulk CLucene norm reads lose the query I/O context and its cache-write policy. Both are reported inline. No verified P0/P1 remains, and no additional user focus was supplied.

The existing retained-CLucene-norm P1 (4365847990) is fixed by caller-owned per-segment buffers; the sparse-intersection P2 (4365847994) is fixed by allocation-free sorted traversal. The input-row-ID P2 (4366633033) now has tracked inputs; its remaining SNII admission-after-expansion concern stays with that existing thread. The new finding concerns a different allocation storing norm codes, so none of those reports is reposted.

Critical checkpoint conclusions:

Goal and evidence: bitmap-overwritten rows and delete-sign rows are excluded from document counts, term frequencies and non-NULL norm-length totals. Replica/fresh-table and compaction regressions exercise the intended score consistency; runtime success was not independently verified.
Scope and reuse: the change stays within query statistics, index-source helpers, cache bookkeeping and tests, reusing existing bitmap, segment and NULL-index readers.
Concurrency: scanner threads share first collection under the tablet-statistics mutex; searcher sums have their own mutex. Norm reads occur under that mutex to avoid duplicate computation. The traced lock order has no verified reverse acquisition; first-use contention remains possible.
Lifecycle and initialization: rowsets/searchers stay pinned while observed, callbacks are synchronous, and buffers/handles unwind through RAII. Cached searchers clear query I/O context as the normal query path does. No new cross-translation-unit initialization or ownership cycle was found.
Configuration: no settings are added. Existing segment/page/file-cache choices are forwarded into collection, but the new CLucene norm reads discard that context (the second inline finding). The separate, pre-existing omission of enable_file_cache/read_file_cache propagation is unchanged.
Compatibility: no persisted format or wire representation changes. MoW scores intentionally change because avgdl now uses visible decoded norms, including quantization for long documents.
Thrift/protobuf IDs: no gensrc changes; release-branch field-ID comparison is inapplicable.
Parallel paths: CLucene and SNII, local/cloud physical segment identities, both scanner split builders, and unsplit scans were traced. The non-bitmap statistics path retains its prior corpus semantics.
Special conditions: sorted unique bitmap IDs justify subtraction; NULLs contribute documents without tokens; all-hidden collections avoid missing-statistics errors; missing norms and corrupt row bounds fail explicitly. New branches explain their concrete cases.
Test coverage: added tests cover visible counts/df/long norms, NULL/all-hidden cases, sparse/dense intersections, partitioned norm sums, cache reuse/error/eviction, I/O context, and replica compaction. Direct tests for SNII selected-row memory limits, concurrent cancellation and noncontiguous cloud IDs remain absent.
Expected results: changed score families and the fault-test row set were independently derived and agree with the checked-in outputs. Output generation provenance and execution were not verified.
Observability: existing StatisticsCollectTime, collection VLOG counters and propagated index/segment errors cover the changed path; no new critical tracing requirement was identified.
Transactions and persistence: read-set versions and bitmap aggregation match normal scanning; no EditLog, failover or persistent-write logic changes.
Data writes and crash atomicity: no data mutation or new commit protocol; captured ownership and failure cleanup were traced without a new leak or inconsistent publication finding.
FE/BE variable passing: no new transmitted variable; existing query options are consumed locally.
Performance and memory: sorted intersections remove the old dense-bitmap cost, and CLucene caches bounded scalar sums instead of retaining norms. The untracked SNII output and CLucene cache-policy findings remain. SNII still reads/sums norms for every MoW query, an explicitly documented cost.
Other issues and user concerns: no additional substantiated issue after the missed-case rechecks; there was no extra user-provided focus.

Limits: this was static review only, with no builds, runtime tests or source changes. Predicate deletes, debug reads exposing delete-sign rows, tokenless non-NULL ambiguity, missing NULL bitmaps inherited from old compaction behavior, and tablet-local statistics remain the PR's documented limits. Those paths were examined and dispositioned; they are not claims of runtime validation.

Review converged after 3 round(s).

github-actions · 2026-10-25T12:20:08Z

+Status IndexSource::sum_norm_lengths_of(std::span<const uint32_t> docs, uint64_t* out) {
+    *out = 0;
+    const std::span<const float> lengths = norm_lengths();
+    std::vector<uint32_t> norms;


[P2] Bound or track the SNII selected-row norm buffer

For an analyzed SNII MoW segment with 10 million NULL rows and one matching row, sum_non_null_norm_lengths passes the entire NULL list here, and SniiIndexSource::encoded_norms assigns one uint32_t per ID. This allocates another roughly 40 MB before scan pruning, without a MemTracker charge or reservation. The manually charged NULL-ID vector and DorisVector hidden IDs cover separate inputs; the 4096-row batching in sum_norm_lengths does not bound this direct call. Large hidden non-NULL sets reach the same path, so concurrent tablet collection can exceed query memory limits. Please batch selected rows here or sum them directly with tracked storage. This is a distinct norm-code output buffer from the previously reported input row-ID arrays.

github-actions · 2026-10-25T12:20:08Z

+void CluceneIndexSource::_read_norms(const ReaderPartition& partition,
+                                     DorisVector<uint8_t>* norms) const {
+    norms->assign(partition.end - partition.begin, 0);
+    partition.reader->norms(_field.c_str(), norms->data());


[P2] Preserve the query I/O context on statistics norm reads

For a cold remote MoW tablet, the new statistics pass reads norms from every captured segment, including segments a selective scan later prunes. This call never supplies the source's _io_ctx: the norm stream's CSIndexInput has no per-read context, and the cached base stream was reset to defaults. Consequently CacheContext sees is_disposable=false, no query ID and no remote-scan cache-write limiter. Even with disable_file_cache=true or a query cache-write budget, these new bulk reads populate normal-priority cache blocks and bypass that budget. Ordinary scoring had the same limitation, but previously did not read norms of all pruned segments. Please propagate the context per norm read without retaining query-owned pointers on the shared cached searcher.

guptashivam04 · 2026-10-25T12:40:50Z

Thanks for the detailed review, @airborne12!

Delete-sign options and SNII norm sums were introduced by this PR, so I'll fix both here:

The statistics will count delete-sign rows when the scan doesn't filter __DORIS_DELETE_SIGN__, so skip_delete_sign=true and show_hidden_columns=true score with statistics that match the returned rows.
SNII norm sums will be cached per segment (like CLucene's), so repeated queries at the same read version read no norms. I'll measure the repeated-query cost before and after. This also covers the bot's P2 on the SNII selected-row buffer: the remaining reads will be batched and tracked.
I'll also address the bot's other P2: the statistics' CLucene norm reads will carry the query's I/O context without keeping it on the cached searcher.
I'll add regression tests for the delete-sign options across V2/V3/SNII.

MoW light deletes were already counted in the statistics before this PR. Supporting them means evaluating delete predicates inside the statistics (reading the predicate columns of each older segment, with caching), which is a sizeable change of its own. Since light delete is off by default for MoW tables, could I handle it, with its V2/V3/SNII regression tests, in a follow-up PR right after this one?

This PR fixes a production issue for us (replicas returning different score() values with CDC upserts and deletes), so getting the default paths merged first would help a lot. Does this plan work for you?

guptashivam04 requested review from airborne12, csun5285 and eldenmoon as code owners October 22, 2026 06:46

Shivam Gupta added 5 commits October 22, 2026 13:30

guptashivam04 requested review from gavinchou and yiguolei as code owners October 23, 2026 14:44

Shivam Gupta added 3 commits October 23, 2026 21:02

github-actions Bot requested changes Oct 24, 2026

View reviewed changes

Shivam Gupta added 2 commits October 24, 2026 11:15

Merge remote-tracking branch 'origin/master' into fix/bm25-visible-ro…

8147a24

…w-statistics

github-actions Bot reviewed Oct 24, 2026

View reviewed changes

Comment thread be/src/storage/index/inverted/reader/collection_statistics.cpp Outdated

Merge remote-tracking branch 'origin/master' into fix/bm25-visible-ro…

a0d9931

…w-statistics # Conflicts: # be/test/storage/index/inverted/reader/collection_statistics_test.cpp

github-actions Bot reviewed Oct 25, 2026

View reviewed changes

Conversation

guptashivam04 commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Oct 22, 2026

Uh oh!

yiguolei commented Oct 24, 2026

Uh oh!

yiguolei commented Oct 24, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

guptashivam04 Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

guptashivam04 Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Oct 24, 2026

Uh oh!

hello-stephen commented Oct 24, 2026

Uh oh!

hello-stephen commented Oct 24, 2026

Uh oh!

hello-stephen commented Oct 24, 2026

BE Regression && UT Coverage Report

Uh oh!

airborne12 commented Oct 24, 2026

Uh oh!

airborne12 commented Oct 24, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hello-stephen commented Oct 24, 2026

Uh oh!

hello-stephen commented Oct 24, 2026

Uh oh!

hello-stephen commented Oct 24, 2026

Uh oh!

hello-stephen commented Oct 24, 2026

BE Regression && UT Coverage Report

Uh oh!

guptashivam04 commented Oct 25, 2026

Uh oh!

guptashivam04 commented Oct 25, 2026

Uh oh!

hello-stephen commented Oct 25, 2026

Uh oh!

hello-stephen commented Oct 25, 2026

Uh oh!

hello-stephen commented Oct 25, 2026

Uh oh!

guptashivam04 commented Oct 25, 2026

Uh oh!

guptashivam04 commented Oct 25, 2026

Uh oh!

hello-stephen commented Oct 25, 2026

Uh oh!

hello-stephen commented Oct 25, 2026

Uh oh!

hello-stephen commented Oct 25, 2026

Uh oh!

hello-stephen commented Oct 25, 2026

BE Regression && UT Coverage Report

Uh oh!

airborne12 commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

guptashivam04 commented Oct 22, 2026 •

edited

Loading

airborne12 commented Oct 25, 2026 •

edited

Loading