Explore faster local JSONL scanning for cost history

[hhh2210]

Related to #1013 and #1014, but intentionally separate from that correctness fix.

While debugging the long turn_context attribution bug, one broader question came up: should CodexBar's local cost-history scanner eventually have a faster path for large JSONL histories?

The current bug is not primarily a raw throughput issue. In #1013, the failure was that the scanner dropped model state from a long Codex turn_context row and later fell back to gpt-5 for model-less token_count rows. #1014 keeps that fix small and in Swift: retain truncated prefixes, recover only the Codex turn-context model from that prefix, keep normal JSON parsing away from partial rows, and invalidate the Codex cost cache.

This issue is only for the follow-up performance/design question:

• A Rust/SIMD helper could plausibly make sense if local histories grow large enough. Libraries such as simdjson, Rust simd-json, or byte-search primitives like memchr/memmem are a good fit for NDJSON-style scanning and selective parsing.
• A pure Swift optimization may be enough first: mmap-style reads, fewer allocations, SIMD-friendly byte searches, and selective JSON parsing only for rows containing relevant markers.
• Metal/GPU is probably a much weaker fit for this specific workload unless profiling proves otherwise. JSONL scanning has irregular control flow, string escaping, row boundaries, and I/O/memory bandwidth constraints. GPU JSON parsers usually need a structural-index pipeline, which would add a lot of complexity to a menu-bar app.

Open questions for maintainers:

• Is scanner throughput currently a real user-facing issue, or should this remain speculative until there is profiling data?
• If it becomes a problem, would you prefer keeping the scanner pure Swift, or would an optional/native helper in Rust be acceptable for this project?
• Should we add a small benchmark fixture around Codex/Claude JSONL scanning before considering any non-Swift implementation?

My bias would be: land the correctness fix in #1014 first, keep this issue as a design/profiling placeholder, and only consider Rust/SIMD after a benchmark shows the Swift scanner is actually the bottleneck.

[hhh2210]

I think a benchmark would make this issue much easier to evaluate without turning it into a premature rewrite discussion.

One useful shape would be a two-layer benchmark:

1. Private-shape survey. Run against a real local ~/.codex/sessions tree, but print only aggregate statistics: file count, total bytes, line-length percentiles, count of rows over 32 KiB / 256 KiB, turn_context line-length distribution, model byte-offset distribution inside turn_context, and how often token_count rows omit an explicit model.
2. Synthetic public fixture. Generate JSONL with the same aggregate shape, but with all prompts/tool payloads replaced by filler strings. The fixture should include small common rows, large irrelevant rows, long turn_context rows with model near the front, and model-less token_count rows. Candidate scanners can then be compared on the synthetic fixture without publishing private logs.

I ran the first layer locally over the last 30 days of Codex session JSONL and got this anonymized shape:

files: 529
total bytes: ~1.20 GB
lines: 145,797
relevant Codex scanner lines: 57,063
line length p50/p90/p95/p99/max: 650 / 5,051 / 12,061 / 53,142 / 6,904,210 bytes
lines > 32 KiB: 2,584
lines > 256 KiB: 697
turn_context lines: 1,935
turn_context lines > 32 KiB: 207
turn_context lines > 256 KiB: 0 in this local 30-day sample
turn_context model offset p50/p95/max: 443 / 2,255 / 2,382 bytes
turn_context model offset < 32 KiB: 1,935 / 1,935
token_count rows missing an explicit model: 22,235 / 22,235

This supports keeping #1014 as a correctness/state-machine fix rather than a throughput rewrite: model-bearing turn_context rows can exceed 32 KiB, the model appears near the front, and subsequent token_count rows commonly depend on preserved model state.

For a future performance PR, I would suggest comparing at least:

• current Swift scanner
• a pure Swift optimized scanner using the same public fixture
• only if the Swift path is shown to bottleneck, an optional Rust/SIMD prototype

Metrics should include wall-clock rebuild time, MB/s, peak RSS if easy to collect, and a correctness check that the generated daily model breakdowns match exactly. I would not include Metal/GPU in the first benchmark unless profiling shows CPU byte scanning is the dominant bottleneck; the local shape above looks more like streaming byte search plus selective JSON parsing than a GPU-friendly workload.

[hhh2210]

Opened draft PR #1017 with the reproducible JSONL shape benchmark and local aggregate survey harness: #1017

[hhh2210]

Follow-up with measured data using the merged #1017 benchmark + survey on a real 30-day local Codex sessions tree (aggregate shape only, no private content).

Local shape (30 days, ~1.23 GB / 536 files / 158k lines)

relevant scanner lines: 61,703
line length p50/p90/p95/p99/max: 670 / 4,870 / 11,717 / 45,936 / 6,904,210 bytes
lines > 32 KiB:  2,719 / 158,044
lines > 256 KiB: 697   / 158,044
turn_context lines: 2,012  (264 over 32 KiB, 0 over 256 KiB)
turn_context model offset p50/p95/max: 429 / 2,255 / 2,382 bytes — all under 32 KiB
token_count rows missing an explicit model: 24,110 / 24,110

Synthetic benchmark (current Swift scanner on this machine, M-class)

divisor=20 bytes=19,621,792 lines=7,290 truncated=129
current=124.7 MB/s   baseline(front-buffer)=62.1 MB/s   speedup=2.0x

Extrapolated to a full 30-day local rebuild at 124.7 MB/s ≈ ~10 s wall-clock, run in the background and absorbed by the existing cache on subsequent reads. With the long turn_context correctness path landed in #1014, the scanner is already producing the right model breakdown on this workload.

A few observations that argue against a Rust/SIMD helper right now:

• Every turn_context row keeps model inside 32 KiB even when the row body itself exceeds 32 KiB (p95 = 2.2 KiB, max = 2.4 KiB on this sample). So the existing prefix-recovery path is the cheap correct fix; a SIMD scanner does not change the structural cost.
• The dominant cost is selective JSON parsing of the small relevant subset (~39% of lines), not raw byte throughput. Pure-Swift mmap + fewer allocations + memchr-style row split would likely capture most of the headroom without adding a Rust toolchain dependency to a menu-bar Swift app.
• gpt-5 token volumes / large monorepo Codex users may have a different shape; happy to extend the survey if anyone wants to share their own anonymized numbers.

Open question for direction before I sink more time into either path:

1. Treat as resolved / close — current ~125 MB/s + cache steady-state is enough; the benchmark stays in tree as future regression coverage.
2. Pure-Swift optimization round — I prototype mmap + reduced allocations + byte-search row split as a small draft PR, with before/after numbers on the same test: add Codex JSONL shape benchmark #1017 benchmark.
3. Optional Rust/SIMD helper — only if you actually want the dependency. Would be gated behind a build flag with the Swift scanner kept as the default.

My bias is (1) → leave open as placeholder, revisit only if a real user reports cost-history rebuild lag. But happy to do (2) if you'd rather have the headroom in place.

[hhh2210]

Follow-up PR opened: #1042.

This is not a JSONL throughput optimization round. It addresses the cache lifecycle gap exposed by the earlier parser fixes: a stale codex-v8.json can keep serving pre-fix model attribution until the artifact filename is bumped manually.

The final PR keeps the file path stable and stamps Codex cost caches with a conservative parser/source content hash, so old or mismatched Codex caches rebuild once while non-Codex caches keep their existing behavior. The PR body includes the local #1017 benchmark numbers and the cold-vs-cached real rebuild timing to make that one-time rebuild tradeoff explicit.

[hhh2210]

Closing as not-pursued for now, per the benchmark and shape data gathered above.

Recap of where this landed. On a real 30-day local Codex sessions tree (~1.24 GB / 545 files / 160k lines) the current Swift scanner sustains ~125–137 MB/s on the #1017 synthetic shape benchmark, and a full cold rebuild extrapolates to ~14s wall-clock with a cached read at ~1.15s. The 30-day shape survey also confirmed that every turn_context row keeps model within 32 KiB even when the row body itself exceeds 32 KiB (p95 = 2.2 KiB, max = 2.4 KiB), so the structural cost of the correctness path landed in #1014 is cheap — a SIMD/Rust scanner would not change that structural cost.

The dominant residual cost is selective JSON parsing on the relevant subset (~39% of lines), not raw byte throughput. If a future user reports cost-history rebuild lag, a pure-Swift round (mmap + reduced allocations + memchr-style row split) should be tried before introducing a Rust/SIMD helper.

The #1017 benchmark stays in tree as regression coverage for any future scanner change.

The cache-lifecycle work that came out of this thread has moved to #1020 and #1042 and is independent of throughput.

Reopen if profiling shows the Swift scanner is the actual bottleneck on a real workload.