Skip to content

ref(conversations)!: Unify durable conversation history#916

Merged
dcramer merged 70 commits into
mainfrom
ref/conversation-event-pi-adapter
Jul 20, 2026
Merged

ref(conversations)!: Unify durable conversation history#916
dcramer merged 70 commits into
mainfrom
ref/conversation-event-pi-adapter

Conversation

@dcramer

@dcramer dcramer commented Jul 15, 2026

Copy link
Copy Markdown
Member

Junior now has one ordered SQL authority for durable conversation history: junior_conversation_events. Exact source and destination chat records use message; later delivery or hydration state uses message_updated; replayable Pi history uses agent_step. Turn lifecycle, authorization, tool activity, handoffs, compaction, and child-conversation lifecycle share the stream without conflating product messages with model input.

This removes junior_conversation_messages as a runtime authority and removes the search-projection write path. Transcript hydration reads message events directly, workspace search uses a partial GIN index over original message payloads, and source-thread summaries remain separate messages_summarized snapshots. Updated message text is projected for transcript hydration but is not folded into search in this cutover. Reporting exposes an allowlisted, privacy-aware event projection from SQL, while the dashboard reduces those reporting events into transcript views. Child agents keep their own event streams with immutable parent lineage.

Agent history is append-only unless an explicit compaction or handoff event supplies replacement history. The internal history_version column partitions those replacements; there is no synthetic initial-context event and no live automatic rollback. Legacy rollback remains read-compatible for retained histories. Volatile runtime bootstrap stays in resumable session state, is bound to the history version that produced it, and never enters SQL replay history. Final assistant history is committed only after destination delivery succeeds.

SQL is also the commit point across the Redis checkpoint boundary. If a worker dies after committing a compaction or handoff but before advancing its unfinished turn record, agent execution follows the newer SQL replacement and discards volatile context from the superseded history. Ordinary turn-record reads remain pinned to their checkpoint so stale callbacks cannot rewrite newer history.

Stored-event reads preserve unsupported event types or schema versions as opaque unknown facts so retained logs remain inspectable. Reporting and search ignore those facts, while active Pi replay rejects them rather than silently changing model context. Supported event types and live writes remain strict; upgrade scripts normalize obsolete formats when their semantics are known.

The event-schema cutover is intentionally not rolling-compatible. junior upgrade imports retained legacy session logs and copies physical message rows into the event stream while preserving compaction anchors and resumable cursors. The legacy message table is retained as a recovery source through the v0.107 deployment: rerunning the upgrade can recover tail rows written by old workers during cutover. The new runtime neither reads nor writes that table, and a v0.108.0 cleanup is scheduled to remove it after event-only writers have been deployed for one release.

@vercel

vercel Bot commented Jul 15, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
junior-docs Ready Ready Preview Jul 20, 2026 10:18pm

Request Review

Comment thread packages/junior-dashboard/src/mock-conversations.ts
Comment thread packages/junior-dashboard/src/mock-conversations.ts
Comment thread packages/junior-dashboard/src/mock-conversations.ts Outdated
Comment thread packages/junior-dashboard/e2e/dashboard.spec.ts
Comment thread packages/junior-dashboard/src/client/eventTranscript.ts Outdated
Comment thread packages/junior-dashboard/src/client/eventTranscript.ts
Comment thread packages/junior-dashboard/src/mock-reporting/fixtures.ts
dcramer added 5 commits July 19, 2026 09:40
Attempt the safe Slack failure reply even when failure-state persistence fails. Record the persistence terminal in conversation history and propagate the persistence error after delivery.
Keep legacy compaction anchors until visible-message events exist, then finalize the canonical suffix and reconcile truncated historical counts from retained thread state.
Scan retained Redis turn-session records before resequencing conversation events. Block unfinished cursors and invalidate terminal cursors even when a partial write omitted the summary indexes.
Only inspect and invalidate turn-session sequence cursors when visible events still require backfill. Repeated upgrades leave active and terminal session state untouched.
Derive per-model usage from message events in the existing privacy snapshot without exposing raw model payloads. Deduplicate context copies across epochs and keep aggregate dashboard tooltips for history without model metadata.
dcramer added 3 commits July 19, 2026 18:35
Store replacement model context on epoch markers, aggregate dashboard usage in SQL, and confine deployed history normalization to the upgrade path.

Keep dashboard mock reporting behind its own module boundary.
Fence upgrade resequencing against resumable turn cursors, close checkpoint deletion gaps, and keep visible compaction boundaries stable.

Use raw SQL existence checks for transitional imports and deduplicate idempotency keys within each append batch.
Validate legacy checkpoints and turn cursors before destructive resequencing. Keep idempotent appends dense and share the live turn-session key contract with the upgrade path.
Store chat messages and replayable agent steps in the canonical event log, search message payloads directly, and model compaction and handoff as explicit history replacements. Keep volatile runtime bootstrap in resumable session state so recovery no longer needs live rollback.
@dcramer dcramer changed the title ref(conversations)!: Make events the canonical transcript ref(conversations)!: Unify durable conversation history Jul 20, 2026
Preserve unknown event types and schema versions as opaque facts so retained logs remain readable. Keep canonical writes and known payloads strict, and reject opaque events only at active model-history replay.
Keep the legacy message table through the conversation-event cutover so a later upgrade can recover rows written by old workers during deployment. Schedule removal for v0.108.0.
Treat SQL compaction and handoff replacements as the commit point when an unfinished Redis turn record still references the prior history version. Keep ordinary record reads pinned so stale callbacks cannot rewrite newer history.
Keep the canonical session-record noun in the execution-specific getter and remove duplicate pinned-read coverage identified during final review.
@dcramer
dcramer merged commit 7d84746 into main Jul 20, 2026
33 of 35 checks passed
@dcramer
dcramer deleted the ref/conversation-event-pi-adapter branch July 20, 2026 23:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: high PR risk score: high

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants