rc.12 stabilization integration — 5 findings (2 HIGH) from 4c+2e devnet sweep on integration/rc.12-stabilize-20260527-2147

[branarakic]

rc.12 Stabilization Integration Test — Report

• Started: 2026-05-27T19:47Z (worktree created)
• Ended: 2026-05-27T20:34Z
• Integration branch: integration/rc.12-stabilize-20260527-2147 @ fb86f6b9e
• Pushed to: ghorigin/integration/rc.12-stabilize-20260527-2147
• Diff vs release/rc.12: 67 files, +9,456 / −697

What's in the integration branch

Five clean (no-conflict) merges layered on top of release/rc.12 (8e19c4481):

order	commit	content
1	4bb474988	chore: merge main into release/rc.12 — pulls 6 main commits (#745 CG Overview S2, #754 SWM promote ownership, #756 SWM agent DID, #768/#769/#770 follow-ups)
2	a576d2897	PR #762 RFC-41 §6.2 edge npm update suite
3	171794c50	PR #765 node-ui query dedup
4	400b5a699	PR #766 Blazegraph + SPARQL-HTTP (RFC 120)
5	fb86f6b9e	PR #767 PR 716 review follow-ups

The 3 "Do Not Merge (pending: …)" PRs (#688, #700, #718) were skipped per directive.

Devnet topology

NUM_CORE_NODES=4 ./scripts/devnet.sh start 6 — 4 cores + 2 edges + 1 mid-run 7th core (phase 3 of stress) + Hardhat chain @ 1s blocks.

Headline numbers

• 223 confirmed VM publications across 4/4 cores (node distribution: 72/25/63/63), highest KC id observed 281. ≥200 bar met.
• All 11 RFC-38 LU scenarios PASS (lu5-pub, lu5-cur, lu7, lu8, lu9, lu10, e2e, xcg, mm, scale, lj) — covers the full curated × public CG matrix, multi-member, late-joiner, cross-CG isolation, scale.
• All 5 rc.12-specific probes PASS (hub-rotation, multi-rpc-failover, libp2p-tunables, cg-phonebook, ack-rejection-reasons) — every new rc.12 stabilization addition holds up.
• Staking, NFT transfer, claim/withdraw/restake all PASS (v10-stress phases 1, 5, 6).
• 5/5 cores submit RS proofs in Phase 4 (v10-stress) — better than the prior FINDINGS.md baseline of 2/5.

Test phase results

phase	result	notes
v10-stress phase 1 (20 stakers, 5 tiers)	✅ PASS	30,000 TRAC total, vault delta exact
v10-stress phase 2 (100 publishes mode-matrix)	❌ FAIL @ VM custodial 5/25	tx broadcast but no receipt within 180000ms — pre-existing FINDINGS Bug 2 nonce race, did not regress
v10-stress phase 3 (mid-run 7th core)	✅ PASS	node 7 staked + ask + RS proof in 11s
v10-stress phase 4 (RS reconciliation)	✅ PASS	5/5 cores submit proofs (was 2/5 previously)
v10-stress phase 5 (ERC-721 stake transfer + redelegate)	✅ PASS
v10-stress phase 6 (claim → withdraw → restake)	✅ PASS
200-vm-publishes driver — bulk lane	✅ 150/151	round-robin nodes 1–4, public/open CGs
200-vm-publishes driver — curated lanes	❌ 0/100	LU-5 source/target access-policy mismatch (driver bug + finding, see §Findings #1)
bulk-publish-topup (100 more)	✅ 100/100	brought confirmed publish count to 223
comprehensive: v10-rc-validation	❌ 32/35	3 sub-tests fail in §5 GOSSIP REPLICATION — see Finding #2
comprehensive: devnet-full-sweep	❌ 8/11	rfc38-unclean-restart (test calibration), swm-ownership-restart (Finding #3), invite-flow (Finding #4)
comprehensive: rc11-promote-crash	❌ FAIL	RFC §6.2 violation — recovery demotes running → queued (Finding #5)
comprehensive: rc11-shutdown-mid	✅ PASS
comprehensive: rfc38-all (11 scenarios)	✅ PASS	all curated/public CG combinations exercised end-to-end
comprehensive: probe-hub-rotation	✅ PASS
comprehensive: probe-multi-rpc-failover	✅ PASS
comprehensive: probe-libp2p-tunables	✅ PASS
comprehensive: probe-cg-phonebook	✅ PASS
comprehensive: probe-ack-rejection-reasons	✅ PASS

Findings

Finding #1 — context-graph register --access-policy doesn't update local CG registry; LU-5 guard refuses publish

Severity: low (workaround exists; UX/CLI bug, not data-loss)

Reproduction:

dkg context-graph create my-cg                 # creates local CG as public
dkg context-graph register my-cg --access-policy 1  # sets on-chain to curated; local still public
dkg publish my-cg --file x.nq                  # → LU-5 mismatch error, refuses

The pre-publish LU-5 guard correctly rejects: "source CG curated=false, target CG curated=true. Refusing to publish: encrypting against the wrong CG's policy would either leak plaintext or be rejected by cores."

Correct pattern (used by devnet-test-rfc38-lu5.sh): set accessPolicy at create time via the HTTP API (POST /api/context-graph/create {accessPolicy:1, register:true, allowedAgents:[...]}), not at register time via CLI. The single-call API works fine.

Fix direction: either (a) context-graph register --access-policy should also update the local CG registry's access policy, or (b) context-graph create CLI should grow --access-policy so policy is set at the right moment, or (c) document that the API path is the only supported path for curated CGs.

Finding #2 — Public-data gossip replication fails in v10-rc-validation §5

Severity: medium — needs reproduction outside this sweep

Reproduction: v10-rc-validation §5 writes Alice data on node 1 then immediately queries nodes 2/3/4. All three return EMPTY.

Caveats that should be checked before filing:

• Other tests that exercise gossip (RS proofs in v10-stress Phase 4, rfc38 lu7 catchup) all PASS, so gossip is not globally broken.
• May be timing-flaky — the validation script queries quickly after write; gossip may simply not have propagated yet at the moment of query.
• Worth re-running v10-rc-validation alone, with a longer settle window, to disambiguate genuine regression vs timing.

Finding #3 — SWM ownership owner-peer not replicated to node 2 after curator promote

Severity: medium — could cause durable owner ambiguity after restart

Reproduction: devnet-test-swm-ownership-restart.sh — owner promotes a root, query node 2's last owners for that root returns empty string (last owners=''). Expected node 2 to observe 12D3KooWQndKUTcWc2googkCt7Rg9Wo9dTYRkvHAETYEYepLjgHf (the promoter's peer id).

The script also has a bash bug (line 71: [: : integer expression expected) which fires when the API returns an empty number; the bug is cosmetic but obscures whether the underlying empty value is the same root cause as the missing owner.

Finding #4 — createContextGraph regression: DKG_CURATOR triple missing from _meta graph

Severity: high — every PROTOCOL_JOIN_REQUEST silently NACKs without this triple

Reproduction: devnet-test-invite-flow.sh step 1b: "DKG_CURATOR triple MISSING from invite-test-…'s _meta graph — createContextGraph regression. Without it, every PROTOCOL_JOIN_REQUEST for this CG silently NACKs."

The test message itself flags this as a regression — i.e. the test was passing before. Should bisect across the 6 main commits + 4 PRs to find when the curator triple stopped being written.

Note: devnet-test-cli-invite.sh (the CLI counterpart) PASSES, so the regression is specific to the API-driven POST /api/context-graph/create path (likely the change in packages/publisher/src/dkg-publisher.ts or metadata.ts from the main delta).

Finding #5 — RFC §6.2 violation: post-restart promote-async job demoted running → queued

Severity: high — explicitly forbidden by RFC; would cause duplicate execution

Reproduction: devnet-test-rc11-promote-crash-recovery.sh — enqueue promote-async job, catch state=running, SIGKILL supervisor+worker, restart node 5, query job status: state=queued, attempts=0. Test message: "RFC §6.2 explicitly forbids this; would cause duplicate execution."

The companion devnet-test-rc11-shutdown-mid-publish.sh PASSES, so graceful shutdown handling is fine. The break is in the unclean-crash recovery path specifically.

Finding #6 — pre-existing publisher nonce race (FINDINGS.md Bug 2)

Severity: known, pre-existing, in FINDINGS.md

v10-stress phase 2 failed at VM custodial 5/25 with tx broadcast but no receipt within 180000ms. Existing 2s back-off + retry-once mitigation in the test only handles the tentative kcId: 0 path, not the receipt-timeout path. The fix direction in FINDINGS (eth_getTransactionCount('pending')) still applies.

Finding #7 — rfc38-unclean-restart needs WRITES_COUNT/PAYLOAD_BYTES tuning

Severity: low — test calibration, not a product issue

The test must kill the core mid-catchup to exercise the lastHostCatchupSeqno resume path. With 200 writes × 16 KiB, M1 catches up in one shot (count=200 ≥ 200) and the kill window never opens. Bump the test's stress parameters.

Ship-readiness recommendation

rc.12 testnet release is plausible but two findings should land first:

• Finding the consensus is hardcoded #4 (DKG_CURATOR createContextGraph regression) — high severity, breaks API-driven invite flow. Bisect the integration branch to identify the offending commit; likely in the publisher / metadata changes that landed via the main delta. Either revert the culprit or land a forward fix.
• Finding Dockerize the node #5 (RFC §6.2 violation in promote-crash recovery) — high severity, RFC violation, can cause duplicate execution. Was rc11-promote-crash passing on release/rc.12 before the integration? If yes, this is a regression and must be fixed before tagging. If no, this is pre-existing and can ship with a known-issue note.

Lower-priority follow-ups (can ship and fix in rc.13):

• Finding Game coordinator gossip hardening (PR #29 follow-up) #1 (curated-CG CLI UX) — workaround exists
• Finding Sync drift between peers due to incomplete paranet scope and weak reconciliation #2 (gossip replication test failure) — needs reproduction outside the sweep
• Finding the context graph explodes even after 4-5 rounds in the game #3 (SWM ownership replication) — needs script bug fixed first
• Finding Refactor autoupdater #6 (publisher nonce race) — known, has documented mitigation
• Finding Swarm syncing not working #7 (test calibration) — test-only

Strong signals for ship-readiness (no regressions detected):

• All 11 RFC-38 LU scenarios PASS — the curated-CG access-policy matrix is solid.
• All 5 rc.12 stabilization probes PASS — hub rotation, multi-RPC failover, libp2p tunables, CG phonebook, ACK rejection reasons.
• All staking + RS-proof + NFT transfer + claim/restake lifecycle phases PASS.
• 5/5 cores submit RS proofs (up from 2/5 in prior FINDINGS).
• 223 confirmed VM publications distributed across 4/4 cores without throughput collapse.
• Inter-node messaging PASS (v10-rc-validation §13).
• 4/4 PR merges into integration applied cleanly with no conflicts.

Artifacts

• Comprehensive sweep results: .devnet/comprehensive-results/20260527T200927Z/
• Per-suite logs: .devnet/comprehensive-results/20260527T200927Z/*.log
• Driver logs: .integration-test-logs/{v10-stress,200-vm-publishes,bulk-topup,comprehensive}.log
• This report: .integration-test-logs/RC12_STABILIZATION_REPORT.md

[branarakic]

Investigated all 7 findings on integration/rc.12-stabilize-20260527-2147. Net result: 5 fixed (4 PRs open), 1 not reproducible, 1 pre-existing.

#	Sev	Reported as	Actual root cause	Status
4	HIGH	createContextGraph regression in publisher / metadata.ts — DKG_CURATOR missing on API path	The triple IS written correctly at create time (verified by direct store dump). Post-#749/#764 query scoping rules incorrectly reject the standard curator-lookup probe SELECT … WHERE { GRAPH <…/_meta> { … } } with contextGraphId scope. Cascading effect: every PROTOCOL_JOIN_REQUEST silently NACKs because the curator-derivation path also can't see the triple.	Fixed in #776 — add _meta to the EXPLICIT-IRI allow set for scoped queries (graph-variable expansion stays strict). Invite-flow test now PASSES end-to-end.
3	MED	SWM workspaceOwner triple not replicated to node 2	The triple IS replicated (verified via unscoped query against node 2 — workspaceOwner is present with the correct peer id). Same query-scoping root cause as F4, but for the /_shared_memory_meta graph.	Fixed in #776 (same PR — extended to also allow _shared_memory_meta on explicit-IRI). SWM-ownership-restart test now PASSES. Cosmetic bash bug fixed separately in #778.
1	LOW	dkg context-graph register --access-policy silently mismatches local CG	Same diagnosis as reported.	Fixed in #777 — fail fast at register time with clear remediation pointer instead of silently mismatching and tripping LU-5 30 minutes later on first publish.
2	MED	v10-rc-validation §5 gossip replication empty	Single sleep 5 + one-shot query was flaky on cold meshes (replication did happen, test polled too early).	Fixed in #777 — converted to per-node poll-until-found loop with 60s default budget; RC_VALIDATION_GOSSIP_BUDGET_S env knob added.
7	LOW	rfc38-unclean-restart calibration	Same diagnosis.	Fixed in #777 — bumped defaults from 200×16KiB to 1000×32KiB so the mid-catchup kill window stays open on faster hardware.
5	HIGH	RFC §6.2 violation in promote-crash recovery	Could not reproduce — 5/5 PASS on this machine against integration/rc.12-stabilize. Likely a worker-timing race where promoteStarted/swmInserted markers land between the test's last poll and the SIGKILL. The recovery code (reconcileExpiredRunning) correctly gates demotion on expiresAt > now and the 27 unit tests in async-promote-queue.test.ts cover all matrix cells. Needs a deterministic repro from the failing machine (state dump of the persisted job before/after restart, plus daemon log lines around the SIGKILL window) before a fix can be scoped.	Tracking — needs upstream repro
6	KNOWN	publisher nonce race	Pre-existing in FINDINGS.md with documented mitigation.	Out of scope for #774.

Bonus fix

#775 — strict-TS TS2352 in packages/cli/src/cli.ts blocked pnpm -r build on a clean clone of release/rc.12. (Already independently fixed on integration via #766 round-3 review.)

End-to-end validation (post-fix, on integration/rc.12-stabilize-20260527-2147 + the F4/F3 fix from #776 applied locally)

scripts/_devnet-full-sweep.sh: 9/11 PASS (up from 8/11 in the original report — swm-ownership-restart ✅ and invite-flow ✅ now pass).

Remaining 2:

• rfc38-unclean-restart — same calibration issue from F7; fix in fix(774): low/med follow-ups — F1 (CLI UX), F2 (gossip timing), F7 (test calibration) #777 (changes test-script defaults).
• sharing — doc-alpha (WM) not visible on Node 2 expected=0, got=8. Unscoped query, so unaffected by fix(query): allow same-CG _meta + _shared_memory_meta on explicit-IRI scoped queries (closes #774 F4 + F3 root cause) #776; the residual failure is either (a) a real WM-vs-SWM isolation issue, or (b) a separate timing flake. Not in rc.12 stabilization integration — 5 findings (2 HIGH) from 4c+2e devnet sweep on integration/rc.12-stabilize-20260527-2147 #774's scope but worth filing as a follow-up.

Ship recommendation

With #775 + #776 + #777 + #778 merged, the two HIGH findings (F4 + the F3 same-root-cause discovery) are resolved end-to-end, the LOW/MED follow-ups are in, and the only remaining running rfc.12 blocker is F5 — which I could not reproduce here. Recommend tagging rc.12 after merging these 4 PRs and getting a deterministic F5 repro from the original failing machine (or treating F5 as a known issue if it doesn't reproduce in a second run from the upstream environment either).