DRAFT: [v8.0] Add cache, algorithmic improvements & all consistency semantics - #84
Merged
Conversation
Codex/eacl explorer pages deps
…cript # Conflicts: # README.md
…tomic, eacl-datascript)
Ports all applicable fixes from fix/audit-root-causes (merged herein for the
legacy top-level lineage) into the module code the DataScript branch and the
EACL Explorer actually run:
Shared module (modules/eacl):
- parser.cljc: parse failures throw :eacl.schema/parse-error instead of
coercing to an empty schema (write-schema! diffed that into retract-all);
// and /* */ comments supported via a comment-aware :auto-whitespace parser
(block-comment regex uses [\s\S]*? - JS RegExp has no inline (?s) flag);
duplicate definitions/relations and permission/relation name collisions
throw; parenthesized unions flatten (paren arrow bases rejected); arrow
target kinds resolve against ALL subject types (declaration-order
independent); ->eacl-schema returns :definitions for the empty-schema guard.
- schema/model.cljc: arrow validation checks every subject type of the
source relation (was last-declared-wins).
- cursor.cljc: invalid/expired tokens throw :eacl/invalid-cursor instead of
silently restarting pagination at page one; TTL is opt-in via
:cursor-ttl-seconds (default: no expiry).
- engine/indexed.cljc: cursors carry a two-part fingerprint
{:s cache-stamp :p paths-digest} (SHA-256 on JVM, goog.crypt.Sha256 in
CLJS); resuming after a schema change that altered this query's paths
throws :eacl/stale-cursor; unrelated changes resume normally.
DataScript adapter (modules/eacl-datascript):
- relation-datoms: nil-padded seek + prefix take-while replaces the
[:a]..[:z] keyword range that hid uppercase/z-prefixed/namespaced subject
types from permission evaluation. NOTE: DataScript sorts vectors
LENGTH-FIRST, so partial seek starts must be nil-padded to full tuple
arity (pinned by the bounded-scan parity tests).
- write endpoints resolve to EXISTING entities or throw :eacl/unknown-object;
find-one-relationship-id stays a nil-returning read.
- write-schema!: empty-schema guard + {:allow-empty-schema? true} opt-out;
count-relationships-using-relation now counts exactly (the old forward-attr
range spanned other relations of the same subject-type and overcounted).
- core: unknown subjects return empty pages (no AssertionError); typed
:eacl/unsupported-consistency and :eacl/not-implemented errors; make-client
validates option keys (:eacl/invalid-config) and accepts canonical
:entid->object-id; :cursor-ttl-seconds threaded through all token calls.
Datomic adapter (modules/eacl-datomic):
- relation-datoms prefix scan (as above, Datomic pads short seek starts).
- cache-stamp is now the schema-history digest scope (replaces
System/identityHashCode, which was a near-always-miss cache and could
collide across GC address reuse): derived from the composite tuple attr
histories of the queried db value, memoized per db value, positive
plain/as-of view classification, all failures degrade to cache misses.
- strict object-ID resolution on writes (existence via datom presence);
impl/tx-relationship requires {:allow-tempids? true} to mint tempids;
unknown-ID reads return empty pages; typed errors replace asserts;
make-client validates options; token TTL opt-in; write-schema! guard;
v6-era base/Relationship + rules/datalog namespaces removed.
Tests: module datomic tests updated from the fixed lineage (fixtures,
config/schema/parser/spice tests) plus schema-basis pinned-behavior tests,
seeded differential property tests, keyword-collation and strictness tests;
DataScript bounded-scan parity tests green. Root :test alias now scans the
module test dirs (the legacy top-level test/ tree is off-classpath).
Verified: cold clj -X:test = 56 tests / 879 assertions, 0 failures;
CLJS node runner (datascript + shared modules) = 0 failures, 0 errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Integrate audit root-cause fixes into the modular backends (shared core, DataScript, Datomic)
The schema-history digest (D8) recomputed per fresh db basis, i.e. after
every d/transact: unrelated relationship writes paid an O(schema-history)
scan for schema-change detection. That does not scale under write load.
Replace derivation with a signaled stamp: write-schema! asserts a fresh
:eacl/schema-version squuid on the schema singleton in the same transaction
as any definition change, and the path/plan caches key on
[(.id db) version resource-type permission-name]. Reading the stamp is one
AVET lookup — no history scans — and unrelated transactions leave every
cache key untouched.
Retained from the audit fixes:
- as-of correctness (§3): as-of views read their era's version datom, so
historic bases get their own cache slots — no cross-base slot sharing.
- Cross-peer invalidation: the stamp lives in the db, so a write-schema!
on any peer invalidates all peers (works without the local eviction,
which remains as hygiene + manual recovery hatch).
- Positive view classification: filter/since/history views bypass the
caches entirely; any stamp-read failure degrades to a miss, never a
stale hit.
- Counter-elision safety: a squuid (not a counter) cannot be re-asserted
to an unchanged value by a concurrent writer.
- Cursor fingerprints keep the {:s :p} shape (:s = version string) and
survive restarts since the stamp is stored.
Consciously accepted (by design, per #74): programmatic relation/permission
datom edits (raw d/transact, d/with, excision) no longer invalidate caches
and may serve stale paths until the next write-schema! or manual
evict-permission-paths-cache!. Users must not manage EACL schema outside
the API.
write-schema! installs the :eacl/schema-version attribute just-in-time on
databases created before it existed, and skips the version bump when the
definition delta is empty (no-op rewrites keep caches and cursors hot).
Tests: schema_basis_test.clj rewritten to pin the new contract (unrelated
transacts hit the cache; invalidation works with eviction disabled; as-of
resolves historical paths; raw-edit staleness documented as intended).
Cold clj -X:test: 56 tests, 854 assertions, 0 failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cl-datomic Port of the top-level revert (cherry-picked in the previous commit) to the modular Datomic backend: - modules/eacl-datomic indexed.clj: schema-cache-scope now returns [(.id db) schema-version] read via one AVET lookup — no history scans, no per-db-value memo. The scope feeds both the shared engine's path-cache keys and cursor fingerprints via :cache-stamp, so unrelated d/transact calls never invalidate either. Sentinel scopes for filter/since/history views now carry a string UUID (was a bare Object) so a cursor minted on an unclassifiable view still round-trips through token encoding. - modules/eacl-datomic schema.clj: v7-schema installs :eacl/schema-version; write-schema! installs it just-in-time on older databases and asserts a fresh squuid when the definition delta is non-empty. - schema_basis_test.clj rewritten to pin the new contract (same tests as the top-level lineage). The DataScript backend is untouched: its ds/listen! stamp already bumps only when a transaction touches schema attributes. The shared cljc engine is untouched (the spi/cache-stamp contract is unchanged), so no CLJS changes. Cold clj -X:test (all modules): 55 tests, 873 assertions, 0 failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix #74: only write-schema! busts the path cache (revert derived digest)
Merge the v7.3 mainline and carry its direction-scoped cursor frontiers, parser hardening, fail-closed query validation, and consistency checks into the shared CLJ/CLJS engine. Preserve the existing limit/cursor adapter contract so the explorer can upgrade by pinning this commit.
Keep the legacy DataScript pagination contract valid on an empty database instead of minting a boundary-less v7.3 cursor that the next request must reject.
Fixes the bugs found in the 2026-07-29 full-source hunt (docs/reports/2026-07-29-eacl-full-source-bug-hunt.md). Two of them produced silently wrong authorization answers: 1. A d/with speculative database inherits the database uuid, reads as :plain, and does not bump :eacl/schema-version, so it shared a permission-path cache slot with the committed database. Evaluating a permission against a speculative schema change published the speculative paths under the LIVE key and granted a permission the committed schema does not define, for the life of the process. 2. Databases that never called write-schema! (programmatic Relation / Permission writes, or migrate! without :schema) have a nil version stamp, so their cache scope never changed and no definition change was ever visible. Both are the same root cause: the scope was not a function of the schema it cached. It is now an exact fingerprint of the EACL definition datoms in the queried db value — count + entity hash + max tx over the two :db.unique/identity composite tuples, which Datomic re-asserts with a fresh tx whenever any component changes. The fingerprint is memoised on the db VALUE, so issue #74's requirement still holds and is now pinned by a test: an unrelated d/transact yields the same fingerprint and no path is recomputed. 3. :db.fn/retractEntity does not remove an entity's relationships. v7 relationships name their peer inside a tuple value, which retractEntity does not follow, so retracting a permissioned entity left the peer's half answering queries — a deleted resource still passed can?, a deleted subject still appeared in lookup-subjects — and the survivor was unreachable through write-relationships!. Adds eacl/delete-object! (protocol) / impl/tx-delete-object, orphan detection and repair helpers, a datom existence probe on the public can?, and README docs. Also fixed: recursive count-* replayed the traversal prefix per page (O(N^2), tripping :max-derived-grants before a large grant set could be counted) and now does one pass; recursive traversal limits are configurable via make-client; bare :last works for recursive permissions instead of throwing, matching the acyclic contract; {:subject/id nil} no longer defeats the read-relationships anchor guard; count-* no longer retain the whole result set; empty pages no longer advertise a next page they have no cursor for; a nil :after/:before throws instead of silently restarting at page 1; impl/can?'s map arity works; write errors are typed ex-info; relationship-exists? consults both indexes so a half-pair is repairable. Performance (fresh JVM per version, second run of the benchmark suite): pagination is flat within noise, with identical traversal-call counts by depth (282/184/79) confirming the v7.3 frontier algorithm is untouched. The public eacl/can? is 19-23% faster despite the added existence probes. The one cost is that a db value the scope memo has not seen pays a single definition-index scan: bounded by schema size, never by data size, and pinned by a new permission-check-benchmark plus a deterministic scope-computed-once-per-db-value test. Tests: 73 -> 91, assertions 1297 -> 1689, all green. New coverage for object deletion, API contracts, speculative/unstamped database cache scoping, and recursive REVERSE differential invariants (a path that previously had none). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adversarial review of the v7.4 cache candidate (report added under
docs/reports/). The existing suite was green throughout, because
differential-test evaluates raw impl against a bare db and never enters
the cache or consistency plumbing — every finding lived in that gap.
Critical: recursive-page-request-key omitted the pagination direction. Its
`direction` argument is the traversal axis (:forward/:reverse, constant per
API), not :asc/:desc, so a `:last/:before` page was stored under exactly the
key a later `:first/:after` request with the same cursor and size read back.
A randomized cursor fuzz returned the wrong page 89 times in 300, including
empty pages that stop a paginating caller early and under-report what a
subject may access. Reachable on the default configuration in an ordinary
Previous-then-Next flow.
Also fixed:
- A relationship write failing before it submits a transaction (:create
conflict, unknown object id, unsupported operation) bumped the coordinator's
:uncertain counter, flushing every live entry in the coherence scope. Writes
now signal cache/mutation-attempted! immediately before d/transact and only
a throw after that point fails closed.
- A client constructed before the database was schema-stamped kept a nil
generation for life, minted page-one tokens carrying :schema-version nil and
then rejected its own page two. It now adopts a stamp when one appears, and
selected-schema-version distinguishes "explicitly nil" from "absent".
- A lookup result naming an object with no external id raised a cache-flavoured
:eacl.consistency/snapshot-unavailable for the first offender. It is a data
integrity fault that also fires with {:cache false}: now
:eacl/unresolvable-object listing every eid, with cause and repair pointer.
- delete-object! held the coordinator write barrier across its whole batch loop.
20k relationships: concurrent lookups saw 277ms max latency and 13 completed
reads; per-batch it is 18ms and 2519 reads.
- fully-consistent reads never read the exact cache, so :exact-results? true
wrote an entry on every call that the default mode could not use. exact-key
pins basis-t, so reusing it at the same basis is sound.
- Page tokens were unbounded and EDN-parsed before authentication, and hostile
nesting threw StackOverflowError straight out of lookup-resources. Now length
bounded, Error-safe, and every cursor rejection carries
:eacl.pagination/invalid-cursor with a :reason.
- Cursor pages published live entries and latest-result pointers that nothing
could ever read, competing for the same admission budget.
- The barrier now covers only the coordinator-snapshot/DB pair and uses
StampedLock optimistic reads; store metrics moved out of the monitor's
critical section; the reader catch-up loop is bounded. can? with
:live-results? true at 8 threads went from ~2.3x slower than {:cache false}
to ~2.3x faster, and 1->8 thread scaling from negative to 4.9x.
- Dead :exact-only?/:required? branches removed; reverse :arrow-relation gained
the subject-type guard forward traversal already had; result shape no longer
depends on cache configuration.
Tests: 166/4516 -> 179/6335 assertions. cache-differential-test runs the
lookup/count invariants through make-client across five cache configurations
against a {:cache false} oracle, including random cursor jumps in both
directions from the same cursor. Every new test was verified to fail against
the pre-fix source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Profiling with the benchmark suite's own instrumentation shows the residual lookup overhead is page-token serialisation, not boundary EID coercion: encrypt-page-token is 88% of coerce-lookup-page and d/entity is 3%. The token cost is pr-str/edn-read, not AES-GCM (~1.2us of 41us). A positional encoding measures 0.49us encode / 0.59us decode. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Profiling a fully cached recursive page walk with the benchmark suite's own
instrumentation put ~50% of wall time in page-token serialisation against ~7%
in the authorization engine. The cryptography was ~3% of the token cost; the
rest was canonicalize + pr-str + edn/read-string, run three to four times per
token.
eacl.datomic.codec encodes exactly the value types a cursor payload contains —
nil, booleans, longs, strings, keywords, vectors, maps, sets — and rejects
anything else at encode time rather than silently round-tripping it as a
different value. The envelope is a fixed binary header whose bytes are used
verbatim as the AES-GCM AAD, so encrypt and decrypt cannot disagree about
canonical form; previously both sides re-derived it through canonicalize +
pr-str. The Cipher is reused per thread (getInstance was ~0.77us per call).
Measured, uninstrumented:
encrypt-page-token 41.1us -> 2.4us (17x)
decrypt-page-token 24.3us -> 2.6us (9.5x)
A page mints two cursors and reads one, so ~106us -> ~7us per paginated call:
lookup {:first 20}, 1 thread off 99-104 -> 47-58 live 89-99 -> 29-30
lookup {:first 20}, 8 threads off 19-24 -> 12-20 live 21-28 -> 6-8
Live results now win at 8 threads instead of losing. In the benchmark suite,
coerce-lookup-page drops 3.32ms -> 0.98ms across 40 pages, and token encode +
decode fall from 88% of it to 33%. can? is unaffected — it mints no cursors.
The prefix moves to eacl4_ and the payload version to 6. eacl3_ tokens are not
read: cursors are opaque, expire in 5 minutes by default, and every caller
already handles :eacl.pagination/invalid-cursor for expiry, so a rolling deploy
degrades to "restart your pagination", never to a wrong answer.
codec_test covers round-trip identity for every scalar and collection shape a
real cursor carries (including all four edge kinds and an explicit nil
schema-version), integer widening, rejection of unencodable values nested at
any depth, truncation at every byte offset, and unknown tags.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two algorithmic changes to can?, both in eacl.datomic.impl.indexed.
1. Arrow evaluation was O(fan-out). can* enumerated every intermediate of the
RESOURCE and point-checked each against the subject, never considering that
the subject side might be far smaller. A doc attached to N teams where the
user belongs to exactly one of them, min-of-N runs:
fan-out 1 2 5 10 100 1000 5000
before 5us 6us 10us 17us 133us 1337us 7164us
after 6us 7us 8us 9us 13us 13us 16us
10x 103x 448x
Both sides already come out of :eavt in ascending intermediate-eid order,
so this is a sorted-set intersection. can* now leapfrogs the two streams,
walking a short probe before paying for a seek so that densely interleaved
streams stay near a linear merge while a large gap costs one seek.
Soundness: the intersection only decides the question when every way to
satisfy the far side of the arrow IS a single relationship. When the target
is a permission with arrows or aliases of its own, an empty intersection
proves nothing and the check still recurses; a non-empty one is a sound
positive short-circuit. direct-grant-relations computes that distinction
from the schema and is memoised per generation. An arrow with a single
intermediate — by far the most common shape — keeps the old point probe,
since opening a second scan does not pay for itself there.
2. Path order was arbitrary. calc-permission-paths returned paths in
find-permission-defs order, which traces back to a clojure.set/difference
in write-schema! — so whether `owner + team->access` checked the relation or
the arrow first was hash order, unobservable to the schema author, and able
to change between deployments of the same schema. On a doc with 5000 teams
that was 6.8ms versus 3.4us. Paths are now sorted cheapest-first once per
schema generation (relation, self-permission, arrow-to-relation,
arrow-to-permission), so the union short-circuits on a direct grant.
This is a bet that a bounded number of point lookups beats an unbounded
arrow scan. It costs ~2us on the multipath benchmark, whose subject is a
platform super_admin so the direct shared_admin relation never matches
(can? warm 11.4us -> 13.5us, threshold 25us); it saves an order of
magnitude whenever a direct grant does match, and removes the cliff.
permission_check_test covers the soundness-critical cases: an arrow to an
all-relation permission at fan-out 1/2/50, an arrow to a permission with its
own arrow where the grant is reachable ONLY through the fallback, an arrow
whose target is a relation, and multi-subject-type relations. The nested case
deliberately gives the resource several intermediates — with one, can? takes
the fan-out-1 path and the test would pass even with the logic broken. Both
mutations (forcing exhaustive? true, disabling the intersection) fail 6
assertions. differential-test, which cross-checks can? against the lookup
engines over seeded random graphs, is unchanged and green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every page of an acyclic arrow lookup opened an index scan for each of the
subject's intermediates, just to learn where each stream starts. Only the
merge's winning elements were ever consumed, so a full walk re-derived nearly
all of that work on every page.
arrow-via-intermediates now takes an optional head-state: the heads a previous
page proved were beyond its boundary, and an atom collecting the heads this
page opens. A cached head is returned without touching the index, and the rest
of that stream is opened lazily only if the page consumes past it — so a page
re-opens one scan per intermediate it draws from, not one per intermediate that
exists. `surviving-heads` then keeps exactly the heads this page did not
consume, which is every intermediate except the handful that contributed.
Full walk, page size 50, min-of-3 (ms):
intermediates 250 500 1000 2000 4000
before 3 11 40 148 591
after 3 7 19 55 199
On the multipath benchmark, forward pagination goes 1.29-1.36 -> 0.80-0.87
ms/page and the deep-page median 0.92 -> 0.67ms. Reverse pagination is
unchanged: the continuation is forward-only, since a backward walk revisits
already-emitted ground and orders its heads the other way, and a second
rarely-exercised boundary rule is exactly where the direction-scoping bug in
this PR came from.
This is a large constant-factor win, NOT the complexity fix I predicted. Two
O(intermediates) terms per page remain: the intermediate stream is still
re-scanned, and the merge tournament is still built over every surviving
stream. Removing them needs the heads persisted in sorted order so a page can
take only the smallest page-size+1 of them and drive the merge from those —
a restructure of the acyclic engine around an explicit heap, which is a much
larger change than this one. Measured decay is ~N^1.9 before and ~N^1.5 after.
The continuation is stored under the same prefix as recursive continuations —
schema generation, query identity, relationship proof, cursor edge — so it is
subject to the same invalidation. A miss falls back to the existing per-path
frontier replay, so correctness never depends on the cache; cache-differential
already covers resumed pages against a {:cache false} oracle across five cache
configurations, and a new test asserts the resume path is actually reached so
that silently losing it cannot leave a green suite.
recursive-continuation-context is renamed continuation-context, and its
lookup-opts keys drop the "recursive-" prefix, since both engines now use it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ports the bench/v6 branch's version-comparison runner to the v7 :first/:after
API, keeping the same permission schema, graph, page size, page count and
warmup/sample methodology so the two are directly comparable. Adds the measured
results and a short report.
Same machine, one fresh JVM per version and per cache configuration, 300
accounts / 150,000 servers / 3,000 pages of 50, median of 3 walks. Every run
returned exactly 150,000 unique results.
v6 v7.4 off v7.4 default v7.4 live
full walk (median of 3) 44.39s 38.20s 15.61s 14.52s
throughput 67.6 p/s 78.5 p/s 192.2 p/s 206.6 p/s
vs v6 1.00x 1.16x 2.84x 3.06x
page 1 28.68ms 20.70ms 20.84ms 0.048ms
page 1,000 19.11ms 16.68ms 15.70ms 0.158ms
page 3,000 1.05ms 2.70ms 2.56ms 0.155ms
Three things worth stating plainly:
- Uncached, v7.4 is only modestly faster (1.16x on the walk, 1.39x on page 1).
The 2.84x for the default cache is the acyclic head continuation added
earlier on this branch, not result caching — the default cache retains no
results at all.
- Sampled page latencies do not improve under the default cache because a
sampled page is re-requested with the same :after cursor, and stream heads
are published under the produced page's cursor and read under the page's
bound. Only a sequential walk populates the key it later reads. :live-results
serves the repeat from the result cache instead, hence 0.15ms.
- v7.4 is ~2.5x SLOWER on the last page (2.70ms vs 1.05ms). Small in absolute
terms and the one place v6 wins, but recorded rather than omitted.
v6 has no recursive permission support, so nothing here exercises the traversal
engine, recursive counts, consistency modes or authenticated cursors — all of
which v7 pays for in every number above and v6 never did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The exact-result cache was keyed by Datomic basis-t, so any transaction
anywhere in the database minted a new key. Measured: 320 can? evaluations per
300 reads with one unrelated write between each — a 0% hit rate, and SLOWER
than running with no cache at all, because every read paid a publication it
could never read back.
An epoch is a t that changes only when EACL-relevant data changes, verified
against Datomic's transaction log.
one unrelated txn per read cache off basis-keyed log epoch
can? 13.9us 26.9us 12.1us
can? evaluations / 300 reads - 320 1
lookup-resources {:first 20} 86.9us 101.8us 49.2us
Loopholes found and closed while validating this, each with a test:
- Keying on the coordinator's observed-t instead is UNSOUND and was my first
proposal. A coordinator only sees writes made through its own process:
proven with two connections to one database, where the reader's observed-t
stayed at 0 while another connection's write flipped the answer. That is
privilege escalation, not staleness. The log sees every writer, including
raw d/transact of tx-relationship output, which no coordinator can.
- d/log is a SNAPSHOT. A Log obtained before a transaction never observes it
(verified: cached log saw 1 tx where a fresh one saw 2), so it is fetched
fresh on every check and must stay that way.
- d/tx-range's start is inclusive and t values are not guaranteed contiguous,
so the boundary transaction is filtered by :t. Without it the transaction
that established the boundary is re-read and a relationship write there
reports a change forever.
- Watched attribute eids were resolved once at construction. A client built
before the EACL schema is installed — which this repo's own benchmark seeding
does — held an empty watched set, detected nothing, and went stale
permanently. They are now re-resolved until complete, and an incomplete set
means "cannot verify", never "nothing changed".
- A single process-wide coordinator, my other proposal, breaks multi-database:
the reader catch-up floor is a raw t compared against a different database's
basis, so a young database waits for a t it will never reach.
Bounds: past :max-scanned-transactions (256) the window is abandoned and the
epoch advances — a miss, never a wrong answer. A connection with no usable log
disables exact retention rather than reverting to basis keying. Historical
reads (cursors, at-exact-snapshot) are their own epoch and deliberately not
made hot: EACL targets the current database and a cache per point in time is a
non-goal.
Mutation-tested: a constant epoch (the unsound process-local shortcut) fails 10
assertions, epoch = basis-t (the old behaviour) fails the precision test, and
pointing the watched set at a nonexistent attribute fails 4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hip-tuples' into codex/optimize-datascript-relationship-storage
…onship-storage' into agent/formally-verified-cache-v8
…' into codex/pr95-adversarial-fixes # Conflicts: # bin/formal # modules/eacl/src/eacl/relationships/relay.cljc # modules/eacl/test/eacl/relay_test.cljc
…to codex/cache-cursor-pages-v8 # Conflicts: # modules/eacl-datahike/src/eacl/datahike/core.clj # modules/eacl-datascript/src/eacl/datascript/core.cljc # modules/eacl/src/eacl/relay.cljc # modules/eacl/test/eacl/relay_test.cljc
…o codex/recoverable-compact-cursors-v8 # Conflicts: # docs/release-notes-v8.0.md # formal/smoke/clj/eacl/formal/production_kernel_test.clj # formal/smoke/cljs/eacl/formal/production_kernel_test.cljs # formal/tla/EaclTemporal.tla # formal/verification/temporal-model.md # modules/eacl/src/eacl/relationships/relay.cljc # modules/eacl/src/eacl/relay.cljc # modules/eacl/test/eacl/relay_test.cljc
…e-v8 DRAFT: Upgrade DataScript and Datahike to the shared v8 engine
…shness-cache Redesign cross-backend freshness and authorization caching
…hip-tuples Align Datahike relationship storage with Datomic
…onship-storage Optimize DataScript relationship storage with endpoint pairs
Verify and accelerate the EACL v8 authorization cache
Fix adversarial correctness and cache issues in #95
Eliminate Relay pagination cache overhead
…s-v8 Make cursors recoverable, query-scoped, and compact
Guard the integrated v8 cursor pipeline
Owner
Author
|
Integrated the reviewed v8 stack into
Final head: Final validation on the release head is green: umbrella tests, all four isolated modules, Dafny/generated boundaries, temporal models, parity/mutation checks, 265 Dafny obligations, local non-benchmark suite (384 tests / 13,163 assertions), DataScript CLJS (102 / 1,372), and heavy pagination/cache benchmark (9 / 3,403). The Explorer testing branch is pinned to this exact commit in theronic/eacl-explorer#2. |
Owner
Author
|
GitHub’s native stacked-PR async merge treated this PR as the root of the stack and marked it merged while #100 and #101 were being merged for GitHub does not permit reopening a PR once it has been marked merged, even after its merge commit is removed from the base branch. The continuing draft v8 → main review PR is #103. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EACL 8.0 offers several major performance improvements:
This is the culmination of over a year of bursty work and I expect it to satisfy the design goal of being fast enough for the design goal of 10M resource entities, but your benchmarks will depend on recursive vs non-recursive schema, cache tuning and the density of intermediate resources in your schema graph:
twith support for consistency semantics:full-consistency,minimize-latency,at-least-as-fresh&at-exact-snapshotjust like SpiceDB.O(N^2/page-size), which is inherently a difficult problem. We retain the optimized terminal resource optimizations for non-recursive schema because it's that fast and switching to recursive traversal strategy would only make it worse (tested). The cache is tunable and supports multiple backends.I have opted for a major version bump from v7.3 to v8.0 because it requires one new EACL Datomic attribute needed for caching:
:eacl/relation-version, which is an index overt. I tried to usetfor this directly, but retractions are invisible without scanningd/history(slow and could be large), so we need this for cache to work. Datomic sorts by EAVT/AEVT/AVET/VAET, but never bytexcept for the tx-log, which could retain many unrelated changes.This RC is likely to remain in production testing for 1-2 months before promoting to main.