Feature: Add on_group_complete callback for streaming eval results. #632
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
estsauver
pushed a commit
to estsauver/verifiers
that referenced
this pull request
Feb 5, 2026
…ai#755) Replace three separate callbacks (on_start, on_progress, on_log) with a single event-based system using TypedDict events with Literal discriminators. - Remove: StartCallback, ProgressCallback, LogCallback - Add 7 new event TypedDicts with type discriminator: - StartEvent: emitted at start with resolved counts - ProgressEvent: emitted after each rollout/group completes - GroupCompleteEvent: for PR PrimeIntellect-ai#632, includes State objects - LogEvent: for log messages with level/source/timestamp - LogStreamEvent: infrastructure for PrimeIntellect-ai#753 log streaming - SaveEvent: when results are saved (intermediate or final) - CompleteEvent: when generation finishes - Add EvalEvent union type and EventHandler callback type - LogStreamFileWriter: writes log_stream events to files for tailing - Update generate() and evaluate() signatures: single on_event parameter - Add _emit_event() helper using maybe_await() - Add _run_group_with_states() to preserve State objects - Emit events at all key points: - StartEvent with accurate num_examples calculation - ProgressEvent after each completion - GroupCompleteEvent for non-independent scoring (with States) - SaveEvent for intermediate and final saves - LogEvent for important messages - CompleteEvent at the end - Update run_evaluation() signature to use on_event - Rewrite run_evaluations_tui() with match/case event handler - Preserve all metric accumulation logic - Update comments to reflect new event system - Add TODO for full log_streams implementation - Event type structure tests - LogStreamFileWriter tests (creation, appending, custom paths) - All 35 eval-related tests pass Removed callback parameters from: - Environment.generate() - Environment.evaluate() - run_evaluation() Replaced with: on_event: EventHandler - Issue PrimeIntellect-ai#755: Consolidate callbacks for eval TUI - PR PrimeIntellect-ai#632: GroupCompleteEvent with State objects (infrastructure) - Issue PrimeIntellect-ai#753: Log streaming infrastructure (LogStreamEvent + writer)
14 tasks
estsauver
pushed a commit
to estsauver/verifiers
that referenced
this pull request
Feb 6, 2026
…ai#755) Replace three separate callbacks (on_start, on_progress, on_log) with a single event-based system using TypedDict events with Literal discriminators. - Remove: StartCallback, ProgressCallback, LogCallback - Add 7 new event TypedDicts with type discriminator: - StartEvent: emitted at start with resolved counts - ProgressEvent: emitted after each rollout/group completes - GroupCompleteEvent: for PR PrimeIntellect-ai#632, includes State objects - LogEvent: for log messages with level/source/timestamp - LogStreamEvent: infrastructure for PrimeIntellect-ai#753 log streaming - SaveEvent: when results are saved (intermediate or final) - CompleteEvent: when generation finishes - Add EvalEvent union type and EventHandler callback type - LogStreamFileWriter: writes log_stream events to files for tailing - Update generate() and evaluate() signatures: single on_event parameter - Add _emit_event() helper using maybe_await() - Add _run_group_with_states() to preserve State objects - Emit events at all key points: - StartEvent with accurate num_examples calculation - ProgressEvent after each completion - GroupCompleteEvent for non-independent scoring (with States) - SaveEvent for intermediate and final saves - LogEvent for important messages - CompleteEvent at the end - Update run_evaluation() signature to use on_event - Rewrite run_evaluations_tui() with match/case event handler - Preserve all metric accumulation logic - Update comments to reflect new event system - Add TODO for full log_streams implementation - Event type structure tests - LogStreamFileWriter tests (creation, appending, custom paths) - All 35 eval-related tests pass Removed callback parameters from: - Environment.generate() - Environment.evaluate() - run_evaluation() Replaced with: on_event: EventHandler - Issue PrimeIntellect-ai#755: Consolidate callbacks for eval TUI - PR PrimeIntellect-ai#632: GroupCompleteEvent with State objects (infrastructure) - Issue PrimeIntellect-ai#753: Log streaming infrastructure (LogStreamEvent + writer)
estsauver
pushed a commit
to estsauver/verifiers
that referenced
this pull request
Feb 6, 2026
…ai#755) Replace three separate callbacks (on_start, on_progress, on_log) with a single event-based system using TypedDict events with Literal discriminators. - Remove: StartCallback, ProgressCallback, LogCallback - Add 7 new event TypedDicts with type discriminator: - StartEvent: emitted at start with resolved counts - ProgressEvent: emitted after each rollout/group completes - GroupCompleteEvent: for PR PrimeIntellect-ai#632, includes State objects - LogEvent: for log messages with level/source/timestamp - LogStreamEvent: infrastructure for PrimeIntellect-ai#753 log streaming - SaveEvent: when results are saved (intermediate or final) - CompleteEvent: when generation finishes - Add EvalEvent union type and EventHandler callback type - LogStreamFileWriter: writes log_stream events to files for tailing - Update generate() and evaluate() signatures: single on_event parameter - Add _emit_event() helper using maybe_await() - Add _run_group_with_states() to preserve State objects - Emit events at all key points: - StartEvent with accurate num_examples calculation - ProgressEvent after each completion - GroupCompleteEvent for non-independent scoring (with States) - SaveEvent for intermediate and final saves - LogEvent for important messages - CompleteEvent at the end - Update run_evaluation() signature to use on_event - Rewrite run_evaluations_tui() with match/case event handler - Preserve all metric accumulation logic - Update comments to reflect new event system - Add TODO for full log_streams implementation - Event type structure tests - LogStreamFileWriter tests (creation, appending, custom paths) - All 35 eval-related tests pass Removed callback parameters from: - Environment.generate() - Environment.evaluate() - run_evaluation() Replaced with: on_event: EventHandler - Issue PrimeIntellect-ai#755: Consolidate callbacks for eval TUI - PR PrimeIntellect-ai#632: GroupCompleteEvent with State objects (infrastructure) - Issue PrimeIntellect-ai#753: Log streaming infrastructure (LogStreamEvent + writer)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add on_group_complete callback for streaming eval results.
Type of Change
Testing
uv run pytestlocally.Checklist