Skip to content

feat(experiment): add ExperimentTrendAnalyzer for cross-run regression detection (#186) - #389

Open
lifelmy wants to merge 1 commit into
strands-agents:mainfrom
lifelmy:agent-tasks/186
Open

feat(experiment): add ExperimentTrendAnalyzer for cross-run regression detection (#186)#389
lifelmy wants to merge 1 commit into
strands-agents:mainfrom
lifelmy:agent-tasks/186

Conversation

@lifelmy

@lifelmy lifelmy commented Sep 1, 2026

Copy link
Copy Markdown

Description

Adds ExperimentTrendAnalyzer, a pure analysis layer that connects overall_score and pass rate across sequential experiment runs. Today each run_evaluations() produces a fresh EvaluationReport with no link to prior runs, so a trajectory like 0.91 -> 0.85 -> 0.78 -> 0.71 yields no signal. This analyzer fits an OLS slope (via statistics.linear_regression) to each metric over a window of runs, labels the direction (improving / degrading / stable with a small epsilon so run-to-run jitter is not a false trend), and flags any_regression when either metric is trending down.

Key points:

  • New standalone module src/strands_evals/trend.pyExperimentTrendAnalyzer, ExperimentTrendReport, MetricTrend, RunPoint. No changes to EvaluationReport, Experiment, or EvaluationDataStore, so nothing existing changes behavior.
  • Two constructors: from in-memory EvaluationReports (run order, optional run_ids), or from_files(...) reading JSON written by EvaluationReport.to_file (with an optional window to keep the most recent N runs).
  • Pass rate is derived from test_passes, falling back to overall_score when a report has none; a single run is reported as stable with a zero slope.
  • to_dict() / to_file() for downstream reporting.
  • Exported from the package __init__.

Related Issues

Closes #186

Type of Change

New feature

Testing

hatch test tests/strands_evals/test_trend.py -> 28 passed. Also ran hatch test tests/strands_evals/test_trend.py tests/strands_evals/types/ -> 132 passed (no regressions). hatch fmt --formatter --check -> 362 files already formatted; hatch fmt --linter -> all checks passed; mypy -p src via hatch-static-analysis env -> Success, no issues in 236 source files.

  • I ran the linter, formatter, type checker, and the test suite for the affected package

Checklist

  • I have read the CONTRIBUTING document
  • I have reviewed and understand every line of code in this PR. I use a coding assistant to help implement, and I review and take responsibility for the final change.
  • My change is focused and reasonably small; I have split unrelated work into separate PRs
  • I have added any necessary tests that prove my feature works
  • I have updated the documentation accordingly (new public API; happy to add docs if maintainers point me at the docs source)
  • My changes generate no new warnings

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@github-actions github-actions Bot added enhancement New feature or request area-core Core eval framework: Case, Experiment, task handler, evaluation data stores labels Sep 1, 2026
@lifelmy
lifelmy marked this pull request as ready for review September 3, 2026 09:27
@lifelmy
lifelmy requested a review from a team as a code owner September 3, 2026 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-core Core eval framework: Case, Experiment, task handler, evaluation data stores enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(experiment): ExperimentTrendAnalyzer — cross-run overall_score regression detection

1 participant