Realize only the requested chunk inside the classified read boundary - #128
Merged
Conversation
Adapter scans are ordered lazy sequences from the exclusive bound to the end of the endpoint. Since the physical layer was wired onto the routed engine (92c76be), classified-fetch-fn realized the WHOLE remainder with `vec` on every command, and only then did the reducer keep its physical chunk of 64. Every scan command therefore re-read the endpoint tail: walking an endpoint of degree n cost about n^2/128 realized values instead of n, and a 20-result first page realized the entire endpoint. The boundary now realizes exactly the descriptor's `:limit` (the reducer always sends its physical chunk size); descriptors without a limit keep realizing the complete scan. Released values, `more-physical?`, work counters, order and pages are unchanged by construction: the reducer already truncated to the same chunk. Atomic partial-output discard and retry semantics are unchanged because the chunk is still realized inside the try. Measured on the local million-server Datomic demo (recursive schema, warm second peer, single runs): count-resources for user-1 (24,024 results) realized 1,079,996 -> 96,188 scan values; a 30,000-limited super-user count 1,292,809 -> 114,337 (1,113 -> 730 ms); super-user first page of 20 realized 568 -> 128 values (15.7 -> 8.6 ms). On a synthetic DataScript endpoint with 8,000 direct subjects an exhaustive count realized 504,000 values before and at most 8,064 after. Regression test: routed-reads-realize-only-the-requested-chunk-test (fails 6 assertions under the previous behaviour). CI-equivalent battery 659 tests / 26,510 assertions green; DataScript CLJS 203 / 7,428 green; public-source-closure ledger regenerated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Adapter scans (
:subject->resources/:resource->subjects) return ordered lazy sequences from the exclusive bound to the end of the endpoint. Since the physical execution layer was wired onto the routed engine (#119,92c76be),eacl.engine.physical/classified-fetch-fnrealized the entire remainder withvecon every command; only afterwards didstable-reducer/fetch-valueskeep its physical chunk of 64.Consequences on the routed path (
can?, lookups, counts, checkpoint replay):nrealized aboutn²/128scan values instead ofn(measured 504,000 for a synthetic 8,000-subject endpoint);None of the demo/deployment measurements recorded on 2026-08-14 include this: they predate the wiring merge.
Change
classified-fetch-fnnow realizes exactly the descriptor's:limit(the reducer always sends its physical chunk size); descriptors without a positive integer limit keep realizing the complete scan (raw callers unchanged). No SPI change: adapters still receive the same scan options.Correctness argument: the reducer already truncated every fetch to the same chunk (
fetch-values(into [] (take physical-chunk-size) …)), so released values,more-physical?,:fetched-values/:max-valuesaccounting, first-discovery order, pages, cursors and checkpoints are byte-identical by construction. Atomic partial-output discard and retry semantics are unchanged: the chunk is still realized inside the classificationtry.Measurements (single runs, warm)
Local million-server Datomic demo database (recursive schema, second peer):
count-resourcesuser-1 (24,024 results, 25,632 commands)count-resourcessuper-user:count-limit 30000lookup-resourcessuper-user first 20The remaining count time here is per-command overhead (≈1 command per result on this schema), not realization; wide endpoints (10k–1M direct subjects/resources) are where the quadratic term dominates.
DataScript, one document with
ndirect viewers,count-subjects: realized 8,320 / 32,256 / 127,008 / 504,000 for n = 1k / 2k / 4k / 8k before; at mostn + 64after.Verification
routed-reads-realize-only-the-requested-chunk-test(per-command realization ≤ chunk, linear total, identical page/count, unlimited descriptors still realize fully) — fails 6 assertions under the previous behaviour.bin/formal source-closurepasses;formal/verification/public-source-closure.jsonregenerated.Follow-up (not in this PR): forward
:limitas a scan option so adapters may bound their own work (e.g. Datahike's temporal-wrapper sort fallback).