feat(experiment): add ExperimentTrendAnalyzer for cross-run regression detection (#186) - #389
Open
lifelmy wants to merge 1 commit into
Open
feat(experiment): add ExperimentTrendAnalyzer for cross-run regression detection (#186)#389lifelmy wants to merge 1 commit into
lifelmy wants to merge 1 commit into
Conversation
lifelmy
requested a deployment
to
manual-approval
September 1, 2026 02:26 — with
GitHub Actions
Waiting
lifelmy
requested a deployment
to
manual-approval
September 1, 2026 02:26 — with
GitHub Actions
Waiting
lifelmy
marked this pull request as ready for review
September 3, 2026 09:27
lifelmy
requested a deployment
to
manual-approval
September 3, 2026 09:30 — with
GitHub Actions
Waiting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
ExperimentTrendAnalyzer, a pure analysis layer that connectsoverall_scoreand pass rate across sequential experiment runs. Today eachrun_evaluations()produces a freshEvaluationReportwith no link to prior runs, so a trajectory like0.91 -> 0.85 -> 0.78 -> 0.71yields no signal. This analyzer fits an OLS slope (viastatistics.linear_regression) to each metric over a window of runs, labels the direction (improving/degrading/stablewith a small epsilon so run-to-run jitter is not a false trend), and flagsany_regressionwhen either metric is trending down.Key points:
src/strands_evals/trend.py—ExperimentTrendAnalyzer,ExperimentTrendReport,MetricTrend,RunPoint. No changes toEvaluationReport,Experiment, orEvaluationDataStore, so nothing existing changes behavior.EvaluationReports (run order, optionalrun_ids), orfrom_files(...)reading JSON written byEvaluationReport.to_file(with an optionalwindowto keep the most recent N runs).test_passes, falling back tooverall_scorewhen a report has none; a single run is reported asstablewith a zero slope.to_dict()/to_file()for downstream reporting.__init__.Related Issues
Closes #186
Type of Change
New feature
Testing
hatch test tests/strands_evals/test_trend.py-> 28 passed. Also ranhatch test tests/strands_evals/test_trend.py tests/strands_evals/types/-> 132 passed (no regressions).hatch fmt --formatter --check-> 362 files already formatted;hatch fmt --linter-> all checks passed;mypy -p srcvia hatch-static-analysis env -> Success, no issues in 236 source files.Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.