Skip to content

feat(press-job): Alert Raven when a server lifecycle job fails - #7377

Open
balamurali27 wants to merge 4 commits into
developfrom
feat/alert-failed-server-jobs
Open

feat(press-job): Alert Raven when a server lifecycle job fails#7377
balamurali27 wants to merge 4 commits into
developfrom
feat/alert-failed-server-jobs

Conversation

@balamurali27

@balamurali27 balamurali27 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Problem

A failed Create Server, Archive Server or Resize Server leaves a server half provisioned, half archived, or on the wrong plan. Nobody watches the Press Job list, so these sit until a customer or a bill finds them.

What this does

PressJob.on_workflow_failure is the one place every job failure goes through, so the alert lives there. It fires only for the three job types that need a person:

ALERTED_JOB_TYPES = ("Create Server", "Archive Server", "Resize Server")

The message names the job type, the server, the step that failed, and links the Press Job:

**Resize Server failed** - f1.example.com

Step: Stop Virtual Machine
Job: https://frappecloud.com/app/press-job/331

The step comes from the Press Workflow Task that failed, so the reader knows whether the machine stopped, the resize call failed, or a later step gave up — without opening the job first.

Channel

Reused RAVEN_SERVER_ALERTS_CHANNEL (frappe-cloud-server-alerts), the channel the public server pool health alerts already post to. Adding a Press Settings field for one more channel did not seem worth it. Say so if these belong somewhere else — it is a one-line change.

Turning it off

Press Settings → Raven → Disable Press Job Failure Alerts. During an incident the same three job types fail again and again, and an alert nobody can silence is one nobody can live with. The check sits next to the Raven credentials, so whoever set the channel up finds the switch that stops it.

Choices

  • Explicit job types, not every failure. There are 22 press job types. Alerting on all of them would bury the three that need a person. Swap failures, disk warnings and prune jobs already have their own paths.
  • The alert runs before on_press_job_failure. The rollback handlers do real work and can throw. The alert goes out first so a broken rollback cannot swallow it.
  • A failed alert never fails the job. send_raven_message already logs an error and returns when the Raven credentials are missing or the post fails.

Tests

press.press.doctype.press_job.test_press_job — 5 tests:

  • a failed Resize Server alert names the server, the failed step and the channel
  • a failed Create Server is alerted, and shows Unknown when no task recorded a failure
  • a failed Archive Server is alerted
  • a failed job of another type is not alerted
  • nothing is sent when the setting disables the alerts
Ran 5 tests in 3.577s

OK

🤖 Generated with Claude Code

https://claude.ai/code/session_01XzhrGxqJ6nN5sVeK4swwt6

A failed Create Server, Archive Server or Resize Server leaves a server half
provisioned, half archived, or on the wrong plan. Nobody watches the Press Job
list, so these sat until a customer or a bill found them.

`on_workflow_failure` is the one place every job failure goes through, so the
alert lives there and names the failed step. Reused
`RAVEN_SERVER_ALERTS_CHANNEL`, the channel the public server pool health alerts
already use, instead of adding a Press Settings field for one more channel.

Kept the list of job types explicit. Alerting on every press job type would
bury the three that need a person.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XzhrGxqJ6nN5sVeK4swwt6
@balamurali27 balamurali27 added the backport-master For mergify backport to master label Sep 3, 2026
@greptile-apps

greptile-apps Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Confidence Score: 5/5

The PR appears safe to merge.

The previous comment-length finding is fixed, and no blocking failure remains.

Reviews (3): Last reviewed commit: "Merge branch 'develop' into feat/alert-f..." | Re-trigger Greptile

Comment thread press/press/doctype/press_job/press_job.py Outdated
@codecov-commenter

codecov-commenter commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.57143% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 61.14%. Comparing base (0b4ad21) to head (c10af6b).
⚠️ Report is 4 commits behind head on develop.

Files with missing lines Patch % Lines
press/press/doctype/press_job/press_job.py 91.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #7377      +/-   ##
===========================================
+ Coverage    61.05%   61.14%   +0.08%     
===========================================
  Files         1059     1059              
  Lines       100198   100303     +105     
  Branches      1649     1649              
===========================================
+ Hits         61175    61328     +153     
+ Misses       38983    38935      -48     
  Partials        40       40              
Flag Coverage Δ
dashboard 85.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mergify

mergify Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

balamurali27 and others added 3 commits September 4, 2026 12:17
An alert that cannot be turned off is one nobody can live with during an
incident, when the same three job types fail again and again and bury the
channel.

Put the check next to the other Raven fields, so the person who set the
channel up finds the switch that stops it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XzhrGxqJ6nN5sVeK4swwt6
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-master For mergify backport to master

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants