Skip to content

Realize only the requested chunk inside the classified read boundary - #128

Merged
theronic merged 1 commit into
mainfrom
agent/bound-physical-chunk-realization
Aug 17, 2026
Merged

Realize only the requested chunk inside the classified read boundary#128
theronic merged 1 commit into
mainfrom
agent/bound-physical-chunk-realization

Conversation

@theronic

Copy link
Copy Markdown
Owner

Problem

Adapter scans (:subject->resources / :resource->subjects) return ordered lazy sequences from the exclusive bound to the end of the endpoint. Since the physical execution layer was wired onto the routed engine (#119, 92c76be), eacl.engine.physical/classified-fetch-fn realized the entire remainder with vec on every command; only afterwards did stable-reducer/fetch-values keep its physical chunk of 64.

Consequences on the routed path (can?, lookups, counts, checkpoint replay):

  • walking an endpoint of degree n realized about n²/128 scan values instead of n (measured 504,000 for a synthetic 8,000-subject endpoint);
  • a first page of 20 over a wide endpoint realized the whole endpoint;
  • every command allocated a vector of the whole remainder.

None of the demo/deployment measurements recorded on 2026-08-14 include this: they predate the wiring merge.

Change

classified-fetch-fn now realizes exactly the descriptor's :limit (the reducer always sends its physical chunk size); descriptors without a positive integer limit keep realizing the complete scan (raw callers unchanged). No SPI change: adapters still receive the same scan options.

Correctness argument: the reducer already truncated every fetch to the same chunk (fetch-values (into [] (take physical-chunk-size) …)), so released values, more-physical?, :fetched-values/:max-values accounting, first-discovery order, pages, cursors and checkpoints are byte-identical by construction. Atomic partial-output discard and retry semantics are unchanged: the chunk is still realized inside the classification try.

Measurements (single runs, warm)

Local million-server Datomic demo database (recursive schema, second peer):

Operation realized before realized after ms before ms after
count-resources user-1 (24,024 results, 25,632 commands) 1,079,996 96,188 765 728
count-resources super-user :count-limit 30000 1,292,809 114,337 1,113 730
lookup-resources super-user first 20 568 128 15.7 8.6

The remaining count time here is per-command overhead (≈1 command per result on this schema), not realization; wide endpoints (10k–1M direct subjects/resources) are where the quadratic term dominates.

DataScript, one document with n direct viewers, count-subjects: realized 8,320 / 32,256 / 127,008 / 504,000 for n = 1k / 2k / 4k / 8k before; at most n + 64 after.

Verification

  • New routed-reads-realize-only-the-requested-chunk-test (per-command realization ≤ chunk, linear total, identical page/count, unlimited descriptors still realize fully) — fails 6 assertions under the previous behaviour.
  • CI-equivalent battery via nREPL: 659 tests / 26,510 assertions, 0 failures.
  • DataScript ClojureScript build: 203 tests / 7,428 assertions, 0 failures.
  • bin/formal source-closure passes; formal/verification/public-source-closure.json regenerated.

Follow-up (not in this PR): forward :limit as a scan option so adapters may bound their own work (e.g. Datahike's temporal-wrapper sort fallback).

Adapter scans are ordered lazy sequences from the exclusive bound to the
end of the endpoint. Since the physical layer was wired onto the routed
engine (92c76be), classified-fetch-fn realized the WHOLE remainder with
`vec` on every command, and only then did the reducer keep its physical
chunk of 64. Every scan command therefore re-read the endpoint tail:
walking an endpoint of degree n cost about n^2/128 realized values
instead of n, and a 20-result first page realized the entire endpoint.

The boundary now realizes exactly the descriptor's `:limit` (the reducer
always sends its physical chunk size); descriptors without a limit keep
realizing the complete scan. Released values, `more-physical?`, work
counters, order and pages are unchanged by construction: the reducer
already truncated to the same chunk. Atomic partial-output discard and
retry semantics are unchanged because the chunk is still realized inside
the try.

Measured on the local million-server Datomic demo (recursive schema,
warm second peer, single runs): count-resources for user-1 (24,024
results) realized 1,079,996 -> 96,188 scan values; a 30,000-limited
super-user count 1,292,809 -> 114,337 (1,113 -> 730 ms); super-user
first page of 20 realized 568 -> 128 values (15.7 -> 8.6 ms). On a
synthetic DataScript endpoint with 8,000 direct subjects an exhaustive
count realized 504,000 values before and at most 8,064 after.

Regression test: routed-reads-realize-only-the-requested-chunk-test
(fails 6 assertions under the previous behaviour). CI-equivalent
battery 659 tests / 26,510 assertions green; DataScript CLJS 203 / 7,428
green; public-source-closure ledger regenerated.
@theronic
theronic merged commit ac8d977 into main Aug 17, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant