Separate repos for Datomic / DataScript / Datahike? (declined) - #66
Closed
theronic wants to merge 21 commits into
Closed
Separate repos for Datomic / DataScript / Datahike? (declined)#66theronic wants to merge 21 commits into
theronic wants to merge 21 commits into
Conversation
theronic
force-pushed
the
eacl/datascript
branch
from
March 12, 2026 20:05
632d182 to
3c95095
Compare
Codex/eacl explorer pages deps
…cript # Conflicts: # README.md
…tomic, eacl-datascript)
Ports all applicable fixes from fix/audit-root-causes (merged herein for the
legacy top-level lineage) into the module code the DataScript branch and the
EACL Explorer actually run:
Shared module (modules/eacl):
- parser.cljc: parse failures throw :eacl.schema/parse-error instead of
coercing to an empty schema (write-schema! diffed that into retract-all);
// and /* */ comments supported via a comment-aware :auto-whitespace parser
(block-comment regex uses [\s\S]*? - JS RegExp has no inline (?s) flag);
duplicate definitions/relations and permission/relation name collisions
throw; parenthesized unions flatten (paren arrow bases rejected); arrow
target kinds resolve against ALL subject types (declaration-order
independent); ->eacl-schema returns :definitions for the empty-schema guard.
- schema/model.cljc: arrow validation checks every subject type of the
source relation (was last-declared-wins).
- cursor.cljc: invalid/expired tokens throw :eacl/invalid-cursor instead of
silently restarting pagination at page one; TTL is opt-in via
:cursor-ttl-seconds (default: no expiry).
- engine/indexed.cljc: cursors carry a two-part fingerprint
{:s cache-stamp :p paths-digest} (SHA-256 on JVM, goog.crypt.Sha256 in
CLJS); resuming after a schema change that altered this query's paths
throws :eacl/stale-cursor; unrelated changes resume normally.
DataScript adapter (modules/eacl-datascript):
- relation-datoms: nil-padded seek + prefix take-while replaces the
[:a]..[:z] keyword range that hid uppercase/z-prefixed/namespaced subject
types from permission evaluation. NOTE: DataScript sorts vectors
LENGTH-FIRST, so partial seek starts must be nil-padded to full tuple
arity (pinned by the bounded-scan parity tests).
- write endpoints resolve to EXISTING entities or throw :eacl/unknown-object;
find-one-relationship-id stays a nil-returning read.
- write-schema!: empty-schema guard + {:allow-empty-schema? true} opt-out;
count-relationships-using-relation now counts exactly (the old forward-attr
range spanned other relations of the same subject-type and overcounted).
- core: unknown subjects return empty pages (no AssertionError); typed
:eacl/unsupported-consistency and :eacl/not-implemented errors; make-client
validates option keys (:eacl/invalid-config) and accepts canonical
:entid->object-id; :cursor-ttl-seconds threaded through all token calls.
Datomic adapter (modules/eacl-datomic):
- relation-datoms prefix scan (as above, Datomic pads short seek starts).
- cache-stamp is now the schema-history digest scope (replaces
System/identityHashCode, which was a near-always-miss cache and could
collide across GC address reuse): derived from the composite tuple attr
histories of the queried db value, memoized per db value, positive
plain/as-of view classification, all failures degrade to cache misses.
- strict object-ID resolution on writes (existence via datom presence);
impl/tx-relationship requires {:allow-tempids? true} to mint tempids;
unknown-ID reads return empty pages; typed errors replace asserts;
make-client validates options; token TTL opt-in; write-schema! guard;
v6-era base/Relationship + rules/datalog namespaces removed.
Tests: module datomic tests updated from the fixed lineage (fixtures,
config/schema/parser/spice tests) plus schema-basis pinned-behavior tests,
seeded differential property tests, keyword-collation and strictness tests;
DataScript bounded-scan parity tests green. Root :test alias now scans the
module test dirs (the legacy top-level test/ tree is off-classpath).
Verified: cold clj -X:test = 56 tests / 879 assertions, 0 failures;
CLJS node runner (datascript + shared modules) = 0 failures, 0 errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Integrate audit root-cause fixes into the modular backends (shared core, DataScript, Datomic)
The schema-history digest (D8) recomputed per fresh db basis, i.e. after
every d/transact: unrelated relationship writes paid an O(schema-history)
scan for schema-change detection. That does not scale under write load.
Replace derivation with a signaled stamp: write-schema! asserts a fresh
:eacl/schema-version squuid on the schema singleton in the same transaction
as any definition change, and the path/plan caches key on
[(.id db) version resource-type permission-name]. Reading the stamp is one
AVET lookup — no history scans — and unrelated transactions leave every
cache key untouched.
Retained from the audit fixes:
- as-of correctness (§3): as-of views read their era's version datom, so
historic bases get their own cache slots — no cross-base slot sharing.
- Cross-peer invalidation: the stamp lives in the db, so a write-schema!
on any peer invalidates all peers (works without the local eviction,
which remains as hygiene + manual recovery hatch).
- Positive view classification: filter/since/history views bypass the
caches entirely; any stamp-read failure degrades to a miss, never a
stale hit.
- Counter-elision safety: a squuid (not a counter) cannot be re-asserted
to an unchanged value by a concurrent writer.
- Cursor fingerprints keep the {:s :p} shape (:s = version string) and
survive restarts since the stamp is stored.
Consciously accepted (by design, per #74): programmatic relation/permission
datom edits (raw d/transact, d/with, excision) no longer invalidate caches
and may serve stale paths until the next write-schema! or manual
evict-permission-paths-cache!. Users must not manage EACL schema outside
the API.
write-schema! installs the :eacl/schema-version attribute just-in-time on
databases created before it existed, and skips the version bump when the
definition delta is empty (no-op rewrites keep caches and cursors hot).
Tests: schema_basis_test.clj rewritten to pin the new contract (unrelated
transacts hit the cache; invalidation works with eviction disabled; as-of
resolves historical paths; raw-edit staleness documented as intended).
Cold clj -X:test: 56 tests, 854 assertions, 0 failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cl-datomic Port of the top-level revert (cherry-picked in the previous commit) to the modular Datomic backend: - modules/eacl-datomic indexed.clj: schema-cache-scope now returns [(.id db) schema-version] read via one AVET lookup — no history scans, no per-db-value memo. The scope feeds both the shared engine's path-cache keys and cursor fingerprints via :cache-stamp, so unrelated d/transact calls never invalidate either. Sentinel scopes for filter/since/history views now carry a string UUID (was a bare Object) so a cursor minted on an unclassifiable view still round-trips through token encoding. - modules/eacl-datomic schema.clj: v7-schema installs :eacl/schema-version; write-schema! installs it just-in-time on older databases and asserts a fresh squuid when the definition delta is non-empty. - schema_basis_test.clj rewritten to pin the new contract (same tests as the top-level lineage). The DataScript backend is untouched: its ds/listen! stamp already bumps only when a transaction touches schema attributes. The shared cljc engine is untouched (the spi/cache-stamp contract is unchanged), so no CLJS changes. Cold clj -X:test (all modules): 55 tests, 873 assertions, 0 failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix #74: only write-schema! busts the path cache (revert derived digest)
Merge the v7.3 mainline and carry its direction-scoped cursor frontiers, parser hardening, fail-closed query validation, and consistency checks into the shared CLJ/CLJS engine. Preserve the existing limit/cursor adapter contract so the explorer can upgrade by pinning this commit.
Keep the legacy DataScript pagination contract valid on an empty database instead of minting a boundary-less v7.3 cursor that the next request must reject.
Sets up `modules/eacl-datahike` against the modular SPI: module deps, root deps wiring (paths, dev/test/nrepl extra-paths, a :build-eacl-datahike alias), the shared-contract test, and PORTING.md. The three source files (core/impl/schema) are still to be ported from `modules/eacl-datascript`. DataScript is the template rather than Datomic, and that is the substantive finding here: both DataScript and Datahike reject a PARTIAL tuple as an `:avet` seek bound, where Datomic accepts one as a lower bound. Measured on datahike main in BOTH `:attribute-refs?` modes: seek [:room :owner] → [:kb :reader :party] bound ignored seek [:room :owner nil] → [:room :owner :party] correct seek [:room :owner :party] → [:room :owner :party] correct So the DataScript adapter's nil-padding is the correct approach on datahike too — and it is O(log n), not a workaround. An earlier attempt here treated the partial bound as a datahike compatibility bug and emulated Datomic by scanning the attribute segment; that was wrong on both counts. Requires replikativ/datahike#921 (composite `:db/tupleAttrs` were silently never derived, and tuple validation silently skipped, under `:attribute-refs?`). EACL's v7 schema is built on composite tuples, so before that fix every relation and permission tuple was absent and every permission denied. Merged to datahike main, not yet tagged, hence the :local/root — swap for an :mvn/version once released. The contract test is the definition of done: it runs the same `eacl.contract-support` suite as the DataScript backend, so the two are held to one definition rather than to tests that drifted apart.
Implements the six-fn backend SPI over datahike, ported from modules/eacl-datascript. DataScript rather than Datomic is the template because DataScript and datahike order tuple index bounds the same way. eacl.datahike.contract-test runs the shared eacl.contract-support suite — the same 23 assertions the DataScript backend answers — TWICE, once per attribute representation. Datahike reports a datom's :a as the attribute keyword by default and as a numeric ref under :attribute-refs? (Datomic's representation), and the second mode is where things break quietly, in three ways that all had to be handled: - composite tuples need datahike >= 0.8.1759 (replikativ/datahike#921). Before it they were silently never derived; since the tuples ARE the v7 engine, every permission check denied. - index-range's :attrid is the one accessor that rejects the attribute keyword under :attribute-refs?. It raises, so this one was loud. - comparing a datom's :a against a keyword set matches nothing, which for the cache-eviction listener fails OPEN: can? keeps answering from pre-change permission paths and grants what the schema just revoked. Adds eacl.datahike.db as a fourth namespace, so every divergence from the DataScript source (entid, attribute representation, the index accessors) sits in one file and the rest reads like its template. Adds eacl.datahike.backend-test for the paths the shared contract cannot reach. Worth stating plainly: the contract goes through make-client, which serves relation definitions from a prebuilt catalog, so it never exercises the tuple seek — deleting the nil padding leaves the contract suite green. Both mechanisms were verified by breaking them: no padding fails 12 assertions in both modes, a raw :a comparison fails exactly 1, only under :attribute-refs?. Module is JVM-only: datahike's CLJS API is async, the SPI is synchronous. Full suite: 61 tests, 951 assertions, 0 failures (all four modules).
The module deps.edn still carried :local/root "../../../datahike" and a comment saying the fix was unreleased, while the root deps.edn had already moved to 0.8.1759. A consumer building this module alone would have resolved a sibling checkout on whatever branch it happened to sit on — exactly the failure mode the comment warned about.
…-datahike-fixes # Conflicts: # deps.edn
A Datahike backend for the modular SPI
…sors Fix Datahike exhausted cursor resumption
Owner
Author
|
This has been solved in v8.0 as modules while retaining monorepo. Each module will be published separately to Clojars via #88 (in progress). |
theronic
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EACL has two database targets: Datomic Pro & DataScript, but will soon support Datahike too.
Ideally these should be separate repos, but I don't want to complicate it into multi-repo modules yet, so I'm keeping it as a branch for now.
DataScript is a lot slower than Datomic Pro because JS is single-threaded, so mainly for the in-browser EACL Explorer at this time, but Datahike will be a serious target. Datahike is compelling due to support for S3 as a backing store.