fix(server): Restore configs truncated by a full disk or a reboot

[balamurali27]

Problem

During a disk-full incident, or a sudden disk resize, config.json, common_site_config.json and site_config.json come back truncated. The site or the bench then fails to start, and we restore the file by hand.

The agent writes a config in two steps. It renames the live file to <name>.bak, then copies the new content into place (agent/base.py set_config, agent/bench.py set_bench_config). If a job is writing a config at the moment the disk fills up or the machine goes away, the live file is left empty, half-written, or missing. The .bak is then the last good copy, sitting right next to it.

What this does

A new playbook, restore_truncated_configs.yml. It finds the .bak files, checks the live file and the backup with python3 -m json.tool, and copies the backup over only the configs that do not parse. mode: preserve keeps the mode of the backup, so a config that the failed write removed does not come back world readable.

The play then verifies the configs it found unreadable and fails if one of them still does not parse. BaseServer.restore_truncated_configs() throws on a failed play, so a press job stops instead of restarting the benches on top of a broken config. The operator retries with the button.

Where it runs

Operation	Hook
Resize Server	@task at the end of the flow
Increase Disk Size	@task before the bench restart, so the benches come up with a good config
Stop and Start Server	@task at the end of the flow
Server.reboot()	enqueued after the reboot, with wait_for_reboot
Server.reboot_with_serial_console()	enqueued after the sysrq reset, with wait_for_reboot

The two reboot methods enqueue the play because neither goes through a press job. The hook stays on Server, where a reboot is a user-facing server action, and not on VirtualMachine.reboot, which is the low-level provider call.

reboot on AWS returns while the machine is still up, so wait_for_reboot makes the play wait for port 22 to close before it waits for the machine to come back. Without it the restore runs on the pre-reboot machine and the truncation survives. That wait is best effort: a busy long queue can start the play after the machine is already back, and a timeout there must not stop the restore.

A failed restore no longer reverts a resize that worked. ResizeServerJob.on_press_job_failure used to revert the Plan Change whenever any task failed, including the tasks that run after the machine is already on the new type. It now syncs the machine and keeps the plan when the resize itself succeeded.

The task sits in the press jobs, not in extend_ec2_volume, so the OCI disk path is covered by the same code.

Known limit

The .bak is the version before the last write, not the current config. A restore can therefore give back a config that is one write behind. That is still much better than an empty file, and Press pushes the current config again with Update Bench Configuration. Worth a follow-up if it turns out to matter.

Tested on a local Frappe Cloud

Broke three real configs on an app server, each in a different way:

• benches/<bench>/config.json — cut to the first 40 bytes, the way a disk-full write leaves it
• benches/<bench>/sites/common_site_config.json — truncated to zero bytes
• benches/<bench>/sites/<site>/site_config.json — renamed to .bak with no live file, the way the agent leaves it when it dies between the rename and the copy

Ran the real press job (Increase Disk Size on that server):

  Success      increase_disk_size
  Success      restore_truncated_configs  (5.0)
  Success      restart_active_benches     (13.0)

All three configs parsed as JSON afterwards, and the site config was byte-identical to the copy taken before the test.

Also checked:

• Idempotence — a second run changed no mtime, so a healthy config is left alone.
• Nothing to restore — with the live file and its backup both truncated, the play fails on Verify Configs and the method throws Failed to restore truncated configs on server: <name>. The benches are not restarted.
• A real reboot — broke the config, scheduled systemctl reboot, and started the play with wait_for_reboot. The play waited out the shutdown and restored afterwards (uptime -p reported up 0 minutes).
• A late start — ran with wait_for_reboot against a machine that was already up and would not reboot. The down-wait timed out after five minutes, and the play went on to restore the config.

Considered and left out

• Snapshot Disk and Upgrade MariaDB stop docker or mariadb, not the agent that writes the configs. The machine stays up.
• The auto-scale jobs insert an Auto Scale record for a secondary server on a shared volume. They do not stop or start the primary.
• Increase Swap and Reset Swap write a 4 GB swap file, which can push the disk to full. The risk is indirect and the job restarts nothing, so it is left out. The same one-line task covers it if truncation shows up there.
• Warn Disk only sends a warning. The repair belongs in the Increase Disk Size job that follows it.
• Create Server Snapshot stops and starts the machine, but is left out of this PR.

There is no scheduled sweep. The play runs only after the operations that cause the truncation.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XzhrGxqJ6nN5sVeK4swwt6

[greptile-apps]

Confidence Score: 4/5

The PR is not yet safe to merge because recovery can still execute on a pre-reboot host.

Ignoring the shutdown timeout allows the next connection check to succeed against a machine whose asynchronous reboot has not started, so restoration can finish before the reboot it is intended to follow.

Files Needing Attention: press/playbooks/roles/restore_truncated_configs/tasks/main.yml

Prompt To Fix All With AI

### Issue 1
press/playbooks/roles/restore_truncated_configs/tasks/main.yml:12
**Recovery precedes delayed reboot**

If an asynchronous provider reboot takes more than 300 seconds to close SSH, `failed_when: false` ignores the timeout and `wait_for_connection` accepts the still-running pre-reboot host, causing recovery to report success before the reboot and leaving subsequently truncated configuration unrepaired.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (7): Last reviewed commit: "fix(server): Do not fail the restore whe..." | Re-trigger Greptile}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.07407% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.04%. Comparing base (1801228) to head (8e902ee).
⚠️ Report is 78 commits behind head on develop.

Files with missing lines	Patch %	Lines
press/press/doctype/server/server.py	62.50%	3 Missing ⚠️
...press/doctype/press_job/jobs/increase_disk_size.py	50.00%	2 Missing ⚠️
...ss/doctype/press_job/jobs/stop_and_start_server.py	50.00%	2 Missing ⚠️

❌ Your patch status has failed because the patch coverage (74.07%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #7369      +/-   ##
===========================================
+ Coverage    60.85%   61.04%   +0.19%     
===========================================
  Files         1056     1059       +3     
  Lines        99682   100099     +417     
  Branches      1632     1643      +11     
===========================================
+ Hits         60658    61103     +445     
+ Misses       38986    38955      -31     
- Partials        38       41       +3     

Flag	Coverage Δ
dashboard	85.83% <ø> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[greptile-apps]

Recovery precedes delayed reboot

If an asynchronous provider reboot takes more than 300 seconds to close SSH, failed_when: false ignores the timeout and wait_for_connection accepts the still-running pre-reboot host, causing recovery to report success before the reboot and leaving subsequently truncated configuration unrepaired.

Prompt To Fix With AI

This is a comment left during a code review.
Path: press/playbooks/roles/restore_truncated_configs/tasks/main.yml
Line: 12

Comment:
**Recovery precedes delayed reboot**

If an asynchronous provider reboot takes more than 300 seconds to close SSH, `failed_when: false` ignores the timeout and `wait_for_connection` accepts the still-running pre-reboot host, causing recovery to report success before the reboot and leaving subsequently truncated configuration unrepaired.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.