LAM-1909: Evals – stream datapoints in batches instead of retaining full dataset

[laminar-coding-agent]

Why

For huge datasets, running evals put memory pressure on the machine. Fetching was already batched (LaminarDataset pulls fetch_size pages lazily), but everything fetched — and every EvaluationResultDatapoint with its full data/target/output payloads — was retained in memory until the end of the run. The problem was retention, not eager fetching.

What

• LaminarDataset.__iter__ (new): streams the dataset — serves items already cached by __len__/__getitem__ first, then pulls the remaining pages of fetch_size without appending them to the in-memory cache. __getitem__/__len__ keep their accumulate-and-cache behavior for random access.
• EvaluationDataset ABC: gains a default __iter__ over __getitem__, so existing custom datasets keep working unchanged.
• _evaluate_in_batches: keeps the semaphore sliding-window concurrency (up to concurrency_limit datapoints in flight, no barrier at batch boundaries — one slow datapoint never stalls the window), but now:
  • aggregates score sums/counts incrementally instead of gathering all result datapoints and averaging at the end;
  • prunes finished eval tasks and background upload tasks as the run advances (re-raising any stored exception, preserving fail-fast behavior);
  • advances the dataset iterator via run_in_executor, since page pulls use the sync client and would otherwise block all in-flight datapoints on the event loop.

Peak memory is now bounded by concurrency_limit in-flight datapoints + one fetch_size page (+ the first page cached by the progress bar's len() call), independent of dataset size.

No breaking changes

• evaluate() return shape (EvaluationRunResult) is unchanged.
• get_average_scores remains public and unchanged (no longer used internally).
• Dataset __len__/__getitem__/slice semantics are unchanged.
• Concurrency behavior (semaphore window, default 5) is unchanged.

Test evidence

• New tests: streaming eval run over a paged LaminarDataset (asserts page count and that streamed items aren't retained), concurrency window assertion (max_in_flight == concurrency_limit), __iter__ streaming/cache-first behavior, ABC default __iter__.
• tests/test_evaluations.py: 13 passed.
• Full suite (excl. instrumentation cassettes): 336 passed, 1 failed — test_langchain.py::test_langchain_langgraph, which fails identically on a clean main checkout in this sandbox (pre-existing, unrelated).

LAM-1909

🤖 Generated with Claude Code

---

Note

Medium Risk
Core eval orchestration and dataset iteration changed; behavior is intended to be compatible but concurrency, error timing, and memory semantics deserve careful review on large runs.

Overview
Large eval runs no longer keep the full dataset and every EvaluationResultDatapoint in memory until the end.

LaminarDataset gains a streaming __iter__: it yields anything already cached by __len__/__getitem__, then pulls fetch_size pages from the API and yields them without appending to _fetched_items. Random access via __getitem__/__len__ is unchanged. EvaluationDataset gets a default __iter__ over __getitem__ so custom datasets still work with evaluate().

_evaluate_in_batches now walks iter(self.data) in a semaphore sliding window (same concurrency_limit behavior). It aggregates score sums/counts per completed datapoint and returns averages directly instead of gathering all result objects. Finished eval and upload tasks are pruned as the loop runs. Because dataset pulls are sync, the iterator advances via run_in_executor so page fetches don’t block other in-flight datapoints. A failing datapoint still schedules the rest; the first error is re-raised after everything finishes.

evaluate() return shape is unchanged; get_average_scores stays public but isn’t used internally. CLAUDE.md documents the eval memory model.

^{Reviewed by Cursor Bugbot for commit cae673a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 529797c. Configure here.}