[kfallah]

Evaluate a built project with uv run exp login, uv run exp build powerset, and uv run exp eval powerset.

Build owns trace import, scenario mining, provider selection, and world-model grounding. Eval requires a completed build, asks for worker models, repeats, and a judge, then reviews the evaluation before launch. It produces local HTML and JSON reports with a Pareto plot and side-by-side traces. Nothing is uploaded or activated for serving.

User flow

• Normal terminal scrollback, searchable multi-model pickers, and one animated progress line through preparation, rollouts, judging, and reporting. Interactive build always asks for providers and models, with saved choices preselected.
• Experiential Cloud model selection joins account-visible identities with published prices, capabilities, and deployment-specific reasoning choices. A changed judge retains the project rubric with provisional calibration.
• At least 20 distinct scenarios; defaults of 100 steps, 1,000,000 cumulative assistant output tokens, and eight parallel rollouts. Infrastructure retries remain separate from repeats. Invalid attempts do not contribute to quality or assistant cost.
• Review displays a trace-sized consumption estimate and an editable total spending limit. Cost details separate assistant, simulation, retrieval, and judging. Raising unused execution ceilings does not inflate the expected consumption estimate.

Durable execution and recovery

A shared SQLite request ledger reserves each provider request before dispatch and saves completed responses before releasing unused allowance. Pausing between an assistant response and a simulated observation reuses the completed paid request. Actual provider attempts reconcile unused retry reservations; unknown dispatch spend stays reserved. Insufficient allowance or cell contention pauses with a resume command. Request-ledger admission avoids serial scans of the rollout corpus under the shared cell lock.

Frozen execution identities and pricing remain separate from the approved spending allowance. Raising that allowance preserves the original simulation plan and saved requests. New preparation after a package upgrade receives a revision-bound pricing identity, preserving earlier immutable snapshots.

Judging preserves all visible task, tool, and final-output evidence and reserves the model's declared context minus its output allowance. Judge requests omit incompatible sampling controls. A transcript that cannot be admitted becomes an explicit per-cell exclusion; shared local parameter errors surface their actionable cause.

Resuming a run that stopped during judging reviews a new judging pass over the same saved rollouts. Original preparation and artifacts remain immutable, successful scores are reused, and fresh probe identities permit failed judgments to retry. Recovery honors the approved spending limit independently of the original quote. The report binds the new judging recipe and reconciles exact judge request coordinates across prior passes, including paid responses whose artifact writes failed, without counting worker requests or duplicating charges.

Grounding checks lineage eligibility before input admission or embeddings; a query with no eligible examples has zero retrieval cost and cannot fail an embedding input ceiling. Eligible queries retain their frozen input bound.

Empty Chat completions receive bounded retries. Completed empty responses and the gateway's explicit unavailable-replay response receive fresh client-owned idempotency keys, while caller-owned keys remain unchanged. Every attempted dispatch remains included in conservative retry accounting. An existing default hosted judge no longer prevents authoring a project rubric after build. Identical production trace projections retain their original immutable producer evidence across package upgrades, including older optional economics fields; changed projections still fail replay validation. Judge lineage recipes bind the frozen setup and evidence; identical persisted splits keep their original producer so a package upgrade reuses completed calibration reviews.

New evaluations freeze JSON-object output for their world model. The SDK forwards native JSON mode on Chat, Responses, and Gemini and adds explicit instructions on Anthropic, Bedrock, and Tinker. Tool-result IDs, schema fields, and terminal semantics remain strictly validated. Existing saved settings omit this opt-in and retain their original request fingerprints; no running recipe or paid response is rewritten. Admission includes any provider-added JSON instruction.

Validation

• Real acceptance run completed through 66 successful provider-backed judgments and HTML/JSON report generation. Exact CLI replay added zero provider requests. Browser verification covered the plot, twenty scenario selectors, model dropdowns, and developer-first side-by-side transcripts. The 34 earlier simulation exclusions remain explicit, leaving six shared valid scenarios; this validates execution and recovery, not a complete twenty-scenario benchmark.
• Six saved failing simulator requests completed with valid transitions and exact tool-call matching when replayed through Experiential with native JSON mode. These are formatting diagnostics, not quality judgments.
• 114 focused SDK, simulator, preparation, resume, and schema tests passed for JSON mode; the broader provider suite passed 1,119 tests before identifying an outdated generated-idempotency-key assertion, which was corrected and rechecked.
• 29 judge-upgrade tests passed, including a package upgrade after one completed review that dispatches only the two missing probes.
• 292 focused provider, retrieval, simulator, and evaluation tests passed; 51 provider retry and judge-authoring tests passed after the unavailable-replay fix. Full-repository Ruff, formatting, and ty passed.
• Two accounting regressions cover full-context recovery exceeding the original quote within the approved allowance, and a paid response followed by a failed artifact write.
• The actual CLI recovered a local 40-rollout fixture, retained one successful score, retried only 39 missing judgments, and made zero additional worker/world-model calls. Raising the allowance from $100 to $200 preserved rollout identity; exact replay made zero additional calls. All providers in this test were local fixtures.
• All 66 completed transcripts in the acceptance run passed offline full-context rendering and reasoning-compatible request conversion. Its actual resume command reaches review while preserving the original preparation and spending limit. No live provider call was made for this validation.
• Additional regressions cover parallel cell retries, partial-turn and pairwise-probe resume, served-model pins, source usage estimates, immutable pricing, report artifacts, and interruption cleanup. A real PTY exercise completed 40 fixture rollouts, judgments, and report generation without entering full screen.
• The existing real build completed 1,114 embedding chunks, prepared both RAG indexes, and replayed without new provider calls. The operator can choose and launch the paid eval in the TUI.

Prepared runs using the earlier cost contract require New evaluation; existing builds and immutable evidence remain available. No silent artifact migration is introduced.

Fixes #1105. This PR targets main; #1095 adds bounded embeddings and #1104 preserves distinct scenarios in the combined acceptance checkout.

[kfallah]

@greptileai review

[kfallah]

@greptileai review

[kfallah]

@greptileai review

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Adds evaluation command and refactors build provider setup.

The PR appears safe to merge based on this review; no new actionable issue or outstanding previous finding was identified.

Summary

The PR adds a resumable project-evaluation CLI, spending review, and offline reports. Since the previous review, it freezes JSON-object output for new evaluation world models and forwards that setting through provider adapters while preserving the default serialization of earlier requests.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Completed build] --> B[Prepare and review evaluation]
  B --> C[Saved simulation settings]
  C --> D[World-model requests with JSON mode]
  D --> E[Judging and resumable run]
  E --> F[Local JSON and HTML reports]

Loading

_{Reviews (35) · Last reviewed commit: "Freeze native JSON mode for evaluation w..."}

[kfallah]

@greptile-apps please review the latest head. All four findings are addressed, including the later OTLP observation-pairing finding. Expected costs now reuse the canonical transition extractor for arbitrary exporter span names and missing call IDs. Focused planning/runtime regressions and full lint, format and type checks pass.

[kfallah]

@greptile-apps please review the latest head. Resolved the new main-branch Capture command documentation conflicts, preserving both Capture and Eval. All four cost/resume findings remain fixed. Full Ruff, formatting and ty pass; 122 CLI, provider, evaluation, layout and release tests pass with one existing skip.

[kfallah]

@greptile-apps please review the latest head. The inherited outside-diff judge finding is fixed end to end: both the budget wrapper and ReservedJudgeClient accept the explicit served-model pin, enforce every remaining snapshot field, and bind successful artifacts to the finalized judge identity. Full worker/world/judge alias evaluation coverage and 22 focused regressions pass, along with full Ruff, formatting and ty.

[kfallah]

@greptileai review. The latest commit fixes pricing snapshot collisions after a package upgrade by including producer revision in snapshot identity. Both new upgrade regressions reproduced the failure before the fix; all 52 pricing, preparation, saved-run and CLI tests pass, along with full Ruff, formatting and ty checks. Prior snapshots and builds remain unchanged; no provider calls are made by preparation.

[kfallah]

@greptileai review. The latest commit makes progress visible immediately after Start, adds setup-stage updates before provider dispatch, and keeps one inline spinner and elapsed clock animated between rollout/judge completions. No execution limits or model behavior changed. All 33 focused progress/CLI/runtime tests and full Ruff, formatting and ty checks pass; a real terminal fixture completed 40 rollouts and judgments with local deterministic providers and normal scrollback.

[kfallah]

@greptile-apps please review current head 23e0d9e. The previous review check was cancelled. This adds immediate setup progress and an animated inline rollout/judge bar with elapsed time, preserving terminal scrollback. All 102 focused and shared-consumer tests pass; the combined checkout passes 34 progress, CLI and runtime tests. Full Ruff, formatting and ty checks and actual terminal execution with local test providers passed.

[kfallah]

@greptile-apps Please review current head 6671140. The latest change avoids full-corpus spend reconciliation during request-ledger cell admission, preserves all immutable validation and per-request spend enforcement, and makes genuine contention pause with a resume command. Two regression tests reproduced the old behavior; 79 focused tests and full Ruff/format/ty pass. The real CLI completed 40 fixture rollouts with eight workers and 32 injected first-pass failures, then replayed without additional calls. Please review the current commit and update the confidence verdict.

[kfallah]

@greptile-apps please review the latest head, including full-context judge requests and explicit judging recovery over unchanged rollouts.

[kfallah]

@greptile-apps Please re-review current head 8188341. Both accounting findings are fixed with regressions: full-context judging recovery uses the separately approved allowance without changing simulation identity, and exact request-ledger reconciliation includes paid responses whose artifact writes failed. The 185 focused tests, full Ruff/format/ty gates, and actual CLI recovery/replay fixture pass. No live customer provider calls were launched.

[kfallah]

@greptile Please review the latest head, including bounded empty-response and unavailable-replay recovery, excluded-corpus retrieval admission, and judge authoring after a default evaluation.

[kfallah]

@greptile Please review the latest head. The import-placement finding is fixed. Judge upgrades now also preserve older optional economics fields without rewriting saved evidence and use distinct lineage recipe identities.

[kfallah]

@greptileai review. Native JSON mode is frozen in new eval settings; prior saved request fingerprints are preserved. Calibration resume now retains unchanged split provenance across package upgrades. Focused regressions and live formatting probes are recorded in the description.