Skip to content

feat(cli): webhook notifications for autoresearch events (closes #106) - #107

Open
Srinivasan8888 wants to merge 1 commit into
evo-hq:mainfrom
Srinivasan8888:feat/issue-106-notifications
Open

Srinivasan8888 wants to merge 1 commit into
evo-hq:mainfrom
Srinivasan8888:feat/issue-106-notifications

Conversation

@Srinivasan8888

Copy link
Copy Markdown

Closes #106.

Motivation

evo optimization loops run autonomously for a long time (SFT/RL cycles are 1–2h+ each, chained across many experiments), but there was no way to be notified when the loop found a new best score. Crawling the evo docs (JS-rendered) and grepping the codebase both confirm no notification/webhook/Slack integration exists anywhere — the public integrations are hosts + remote sandboxes + gates + dashboard. This fills that gap.

What's added

  • notify.py — send_notification(url, event, message, extra) POSTs a JSON payload and never raises (a down or misconfigured webhook must not break a run; it returns a bool). The payload sets both text (Slack incoming-webhook key) and content (Discord webhook key) to the message, plus source/event/data, so one URL works for Slack, Discord, or any generic receiver.
  • evo config set notify-webhook <url> — stores the webhook (http/https only; empty clears; shown in config show, readable via config get).
  • new_best event — at the end of evo run, when a commit beats the previous best (compare_scores(metric, score, best_before_score)), evo fires a best-effort notification:

    🎉 evo: new best score 0.8123 on exp_0007 (previous best 0.7913)

  • evo notify test — sends a test notification so users can verify setup without waiting for a run.

Design notes / scope

  • Best-effort by design. _maybe_notify and send_notification swallow all errors and only warn on stderr; a failed webhook never affects run outcome.
  • In: the notify module, the config field, the new_best event, and evo notify test.
  • Out (easy follow-ups): run-finished / budget-cap-hit / experiment-failed events, per-event routing, message templating, retries/backoff.

Tests

tests/unit/test_notify.py (TDD, written failing first) — 11 tests:

  • payload carries both text and content + structured data
  • send_notification returns True on 2xx, False on non-2xx, and False (no raise) on transport error
  • config set notify-webhook stores / rejects non-URLs / clears on empty
  • _maybe_notify is a no-op without a webhook and posts when one is set
  • evo notify test posts when configured and errors clearly when not

Also verified end-to-end (init → config set notify-webhook → config show → evo notify test fails gracefully against a bogus host, rc=1, no crash). Full tests/unit sweep green.

…rd/generic)

evo optimization loops run autonomously for hours, but there was no way to
be told when something happened. Add an opt-in webhook notifier.

- notify.py: send_notification(url, event, message, extra) POSTs a JSON
  payload and never raises (a down webhook must not break a run). Payload
  sets both `text` (Slack) and `content` (Discord) to the message plus
  structured fields, so one URL works for Slack, Discord, or any receiver.
- `evo config set notify-webhook <url>` (http/https; empty clears; shown in
  `config show`).
- `evo run` fires a best-effort `new_best` notification when a commit beats
  the previous best score.
- `evo notify test` sends a test notification to verify setup.

A grep of the code + docs confirms no notification/webhook integration
existed anywhere; this fills that gap.

Closes evo-hq#106.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: webhook notifications (Slack/Discord/generic) for autoresearch events

1 participant