From f69adca98f235a830fea20470db5894563dc163d Mon Sep 17 00:00:00 2001
From: NotYuSheng <notyusheng@gmail.com>
Date: Tue, 28 Jul 2026 11:11:45 +0800
Subject: [PATCH 1/2] feat: automated backups with a tested restore (#379)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Backup was documented but not implemented: backup-restore.rst listed pg_dump and
mc mirror commands for a human to run by hand, with no schedule, no verification,
and no rehearsed restore. The leadership deck ranked losing imported data as the
top risk, and this was the last item behind that framing.

  scripts/backup.sh
    One timestamped archive holding all three things that cannot be regenerated:
    the PostgreSQL dump (custom format, so pg_restore --clean works), the MinIO
    objects, and signatures.yml from the config volume. Plus a manifest recording
    what was captured.

  scripts/restore.sh
    Restores an archive, and verifies rather than assuming: it counts the tables
    actually present afterwards and fails if the database came back empty. It
    rejects corrupt archives, tarballs that aren't TracePcap backups, and warns
    when the manifest's database name doesn't match the target.

  systemd service + timer
    Nightly at 02:30, Persistent=true so a powered-off night is caught up rather
    than skipped, RandomizedDelaySec so several hosts don't hit the same NAS at
    once. Cron equivalent documented.

Two properties that matter more than the happy path:

  - Old backups are pruned only AFTER the new archive passes its size check and
    a tar -tzf read-back, so a failing run can never destroy the last good one.
  - Both scripts exit non-zero on failure, so cron and systemd surface a broken
    backup instead of passing over it silently.

Credentials come from .env and are passed to `mc` inside the container rather
than interpolated into a shell command, following the fix from #604.

Validated end to end, not just executed: uploaded and analysed a PCAP, took a
backup, dropped the public schema and wiped the bucket entirely, then restored.
Row counts returned exactly (files=1 packets=15 conversations=11), 31 tables
came back, the API served the file list, and the recovered PCAP was
byte-identical to the original by MD5. Corrupt/foreign/missing archives were all
rejected, and retention pruning kept the current backups while dropping a
30-day-old one.

PITR is deliberately out of scope — WAL archiving is a poor trade for a
single-server deployment whose primary data is re-importable. RPO (24h) and RTO,
and what is NOT covered (PITR, Keycloak users, redundancy), are documented.

Refs #379

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
 .gitignore                         |   3 +
 docs/operations/backup-restore.rst | 202 ++++++++++++++++++++++++-----
 scripts/backup.sh                  | 154 ++++++++++++++++++++++
 scripts/restore.sh                 | 151 +++++++++++++++++++++
 scripts/tracepcap-backup.service   |  42 ++++++
 scripts/tracepcap-backup.timer     |  23 ++++
 6 files changed, 545 insertions(+), 30 deletions(-)
 create mode 100755 scripts/backup.sh
 create mode 100755 scripts/restore.sh
 create mode 100644 scripts/tracepcap-backup.service
 create mode 100644 scripts/tracepcap-backup.timer

diff --git a/.gitignore b/.gitignore
index 7880e486..f23d2471 100644
--- a/.gitignore
+++ b/.gitignore
@@ -23,3 +23,6 @@ sample-files/*.pcap
 # Docker image tarballs (generated by scripts/pull-and-save-images.sh)
 images/*.tar
 docs/_build/
+
+# Local backup archives (scripts/backup.sh)
+backups/
diff --git a/docs/operations/backup-restore.rst b/docs/operations/backup-restore.rst
index 41c7bfaf..3d3d33c4 100644
--- a/docs/operations/backup-restore.rst
+++ b/docs/operations/backup-restore.rst
@@ -1,66 +1,208 @@
 Backup & Restore
 ================
 
-TracePcap data lives in two places:
+TracePcap holds three things that cannot be regenerated:
 
-- **PostgreSQL** — conversation metadata, analysis results, geolocation data.
-- **MinIO** — raw PCAP files (object storage).
+- **PostgreSQL** — analysis results, conversations, monitor snapshots, and every
+  human-entered label or override.
+- **MinIO** — the raw PCAP objects.
+- **config volume** — ``signatures.yml``, your custom detection rules.
 
-Backing Up
-----------
+``scripts/backup.sh`` captures all three into one timestamped archive, and
+``scripts/restore.sh`` puts them back.
 
-Database Backup
-~~~~~~~~~~~~~~~
+.. warning::
 
-.. code-block:: bash
+   **An untested backup is not a backup.** Rehearse the restore (below) before
+   the deployment holds data you care about — while a mistake is still free.
 
-   docker exec tracepcap-postgres \
-     pg_dump -U tracepcap_user tracepcap > backup.sql
+Automated Backups
+-----------------
 
-MinIO Backup (PCAP files)
-~~~~~~~~~~~~~~~~~~~~~~~~~
+Install the bundled systemd timer for a nightly run at 02:30:
 
 .. code-block:: bash
 
-   docker exec tracepcap-minio \
-     mc mirror minio/tracepcap-files ./backup-pcaps/
+   sudo cp scripts/tracepcap-backup.service scripts/tracepcap-backup.timer \
+     /etc/systemd/system/
+   sudo systemctl daemon-reload
+   sudo systemctl enable --now tracepcap-backup.timer
+
+Edit ``User``, ``WorkingDirectory`` and ``BACKUP_DIR`` in the ``.service`` file to
+match your deployment first. Verify the schedule and check on it with:
+
+.. code-block:: bash
 
-Full Volume Backup
-~~~~~~~~~~~~~~~~~~
+   systemctl list-timers tracepcap-backup.timer
+   journalctl -u tracepcap-backup.service -n 50
 
-To back up all Docker named volumes at once:
+If you prefer cron:
 
 .. code-block:: bash
 
-   sudo tar -czf tracepcap_backup.tar.gz /var/lib/docker/volumes/tracepcap_*
+   30 2 * * * cd /path/to/TracePcap && bash scripts/backup.sh >> /var/log/tracepcap-backup.log 2>&1
+
+The script exits non-zero on any failure, so both cron and systemd surface a
+failed run rather than passing over it silently.
+
+.. important::
+
+   Set ``BACKUP_DIR`` to storage on a **different disk from the deployment** —
+   ideally a NAS or mounted remote share. A backup sitting on the same disk as
+   the data does not survive that disk failing, which is the main thing it is
+   there for.
+
+Configuration
+~~~~~~~~~~~~~
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 20 50
+
+   * - Variable
+     - Default
+     - Purpose
+   * - ``BACKUP_DIR``
+     - ``./backups``
+     - Where archives are written. Point at off-host storage.
+   * - ``BACKUP_RETENTION_DAYS``
+     - ``14``
+     - Archives older than this are pruned. ``0`` disables pruning.
 
-Custom Signature Rules Backup
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Credentials are read from ``.env`` (or the environment), so the script needs no
+configuration of its own beyond the two variables above.
 
-The rules file is stored in the ``config_data`` volume. Back it up with:
+Running a Backup Manually
+-------------------------
 
 .. code-block:: bash
 
-   docker cp tracepcap-backend:/app/config/signatures.yml ./signatures.yml.bak
+   bash scripts/backup.sh
+
+   # or to a specific destination
+   BACKUP_DIR=/mnt/nas/tracepcap bash scripts/backup.sh
+
+Output is a single archive, ``tracepcap-backup-<timestamp>.tar.gz``, containing
+the PostgreSQL dump, the MinIO objects, the config files, and a ``manifest.txt``
+recording what was captured.
+
+The script refuses to record a backup as good if the database dump comes back
+implausibly small, and verifies the finished archive is readable. **Old backups
+are pruned only after the new one passes both checks**, so a failing run can
+never destroy your last good archive.
 
 Restoring
 ---------
 
-Database Restore
-~~~~~~~~~~~~~~~~
+.. warning::
+
+   Restoring is **destructive**. It drops and recreates the objects in the target
+   database and overwrites objects in the bucket. The script prompts for
+   confirmation unless ``FORCE=1`` is set.
+
+Inspect an archive without changing anything:
 
 .. code-block:: bash
 
-   docker exec -i tracepcap-postgres \
-     psql -U tracepcap_user tracepcap < backup.sql
+   bash scripts/restore.sh --list backups/tracepcap-backup-20260728-110713.tar.gz
 
-MinIO Restore
-~~~~~~~~~~~~~
+Perform the restore, with the backend stopped so nothing writes underneath it:
+
+.. code-block:: bash
+
+   docker compose stop backend
+   bash scripts/restore.sh backups/tracepcap-backup-20260728-110713.tar.gz
+   docker compose start backend
+
+The script verifies the result rather than trusting exit codes — it counts the
+tables actually present afterwards and fails if the database came back empty. It
+also rejects corrupt archives and tarballs that are not TracePcap backups.
+
+Rehearsing a Restore
+--------------------
+
+Run this on a **non-production** deployment, or before the box holds real data.
+
+1. Upload a PCAP and let it finish analysing.
+2. Take a backup: ``bash scripts/backup.sh``
+3. Note what you have:
+
+   .. code-block:: bash
+
+      docker exec -e PGPASSWORD=$POSTGRES_PASSWORD tracepcap-postgres \
+        psql -U $POSTGRES_USER -d $POSTGRES_DB -tAc \
+        "SELECT (SELECT count(*) FROM files), (SELECT count(*) FROM packets)"
+
+4. Simulate the loss — stop the backend, drop the schema, empty the bucket.
+5. Restore, restart the backend, and confirm the counts match, the file list
+   renders, and a PCAP downloads intact.
+
+This procedure was used to validate the scripts: a full drop of the ``public``
+schema and a complete bucket wipe, followed by a restore that returned the exact
+row counts and a **byte-identical** PCAP (verified by MD5).
+
+Recovery Objectives
+-------------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 25 55
+
+   * - Objective
+     - Value
+     - Notes
+   * - **RPO**
+     - Up to 24 h
+     - Set by the nightly schedule. Most lost work is re-importable — PCAPs can
+       be re-uploaded and re-analysed — but **human labels and overrides entered
+       since the last backup cannot be recovered.**
+   * - **RTO**
+     - Minutes
+     - Dominated by archive size, not by the procedure. Restore is one command.
+
+.. note::
+
+   Backup and restore both complete in about a second on a trivial dataset,
+   which tells you the procedure has no fixed overhead — not how long your data
+   will take. Time a real backup against your own capture volume and record the
+   result. Expect it to track archive size, which is roughly the PCAP volume plus
+   a database of comparable size (see :doc:`scalability`).
+
+Reducing the RPO below 24 hours means running the timer more frequently. True
+point-in-time recovery would require PostgreSQL WAL archiving, which is
+deliberately not configured — a trade for a single-server deployment whose
+primary data is re-importable. See `issue #379
+<https://github.com/NotYuSheng/TracePcap/issues/379>`_.
+
+What Is Not Covered
+-------------------
+
+- **Point-in-time recovery.** Restores land on a nightly boundary, not an
+  arbitrary moment.
+- **Keycloak users.** The auth overlays persist Keycloak's H2 store in the
+  ``keycloak_data`` volume, which these scripts do not capture. Accounts created
+  in the admin console must be re-provisioned, or that volume backed up
+  separately. The seeded realm import is in version control.
+- **Redundancy.** Backups protect against deletion and corruption, not against
+  hardware failure taking the service offline. See :doc:`storage-redundancy`.
+
+Manual Procedures
+-----------------
+
+The individual commands, if you need to do something the scripts do not cover:
 
 .. code-block:: bash
 
-   docker exec tracepcap-minio \
-     mc mirror ./backup-pcaps/ minio/tracepcap-files
+   # PostgreSQL
+   docker exec tracepcap-postgres pg_dump -U tracepcap_user tracepcap > backup.sql
+   docker exec -i tracepcap-postgres psql -U tracepcap_user tracepcap < backup.sql
+
+   # MinIO
+   docker exec tracepcap-minio mc mirror minio/tracepcap-files ./backup-pcaps/
+   docker exec tracepcap-minio mc mirror ./backup-pcaps/ minio/tracepcap-files
+
+   # Signature rules
+   docker cp tracepcap-backend:/app/config/signatures.yml ./signatures.yml.bak
 
 Export Options
 --------------
diff --git a/scripts/backup.sh b/scripts/backup.sh
new file mode 100755
index 00000000..81557607
--- /dev/null
+++ b/scripts/backup.sh
@@ -0,0 +1,154 @@
+#!/usr/bin/env bash
+# backup.sh
+#
+# Backs up everything TracePcap cannot regenerate:
+#   1. PostgreSQL  — analysis results, conversations, monitor snapshots, human labels
+#   2. MinIO       — the raw PCAP objects
+#   3. config_data — signatures.yml (custom detection rules)
+#
+# Produces ONE timestamped tarball per run, then prunes runs older than
+# BACKUP_RETENTION_DAYS. Designed to be driven by cron or a systemd timer; see
+# docs/operations/backup-restore.rst.
+#
+# Usage:
+#   bash scripts/backup.sh                 # writes to ./backups
+#   BACKUP_DIR=/mnt/nas/tracepcap bash scripts/backup.sh
+#
+# Credentials are read from .env (or the environment) — never hard-coded here.
+#
+# Exit codes: 0 ok, non-zero on any failure (so cron reports it).
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT_DIR="$(dirname "$SCRIPT_DIR")"
+cd "$ROOT_DIR"
+
+# --- configuration ---------------------------------------------------------
+# Load .env if present so cron (which has almost no environment) sees the same
+# credentials as an interactive `docker compose` run. Existing environment
+# variables win, matching how compose itself resolves them.
+if [[ -f .env ]]; then
+  # shellcheck disable=SC1091
+  set -a; source ./.env; set +a
+fi
+
+BACKUP_DIR="${BACKUP_DIR:-$ROOT_DIR/backups}"
+BACKUP_RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-14}"
+
+POSTGRES_DB="${POSTGRES_DB:-tracepcap}"
+POSTGRES_USER="${POSTGRES_USER:-tracepcap_user}"
+MINIO_ROOT_USER="${MINIO_ROOT_USER:-minioadmin}"
+MINIO_ROOT_PASSWORD="${MINIO_ROOT_PASSWORD:-minioadmin}"
+MINIO_BUCKET="${MINIO_BUCKET:-tracepcap-files}"
+
+PG_CONTAINER="${PG_CONTAINER:-tracepcap-postgres}"
+MINIO_CONTAINER="${MINIO_CONTAINER:-tracepcap-minio}"
+BACKEND_CONTAINER="${BACKEND_CONTAINER:-tracepcap-backend}"
+
+STAMP="$(date +%Y%m%d-%H%M%S)"
+WORK_DIR="$(mktemp -d)"
+STAGE="$WORK_DIR/tracepcap-backup-$STAMP"
+ARCHIVE="$BACKUP_DIR/tracepcap-backup-$STAMP.tar.gz"
+
+# Always clean up the staging directory, including on failure — it holds a full
+# copy of the PCAP set and would otherwise fill /tmp after a few failed runs.
+cleanup() { rm -rf "$WORK_DIR"; }
+trap cleanup EXIT
+
+log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"; }
+fail() { printf '[%s] ERROR: %s\n' "$(date +%H:%M:%S)" "$*" >&2; exit 1; }
+
+require_container() {
+  docker inspect -f '{{.State.Running}}' "$1" 2>/dev/null | grep -q true \
+    || fail "container '$1' is not running — start the stack before backing up"
+}
+
+mkdir -p "$BACKUP_DIR" "$STAGE"
+
+log "Backup starting → $ARCHIVE"
+require_container "$PG_CONTAINER"
+require_container "$MINIO_CONTAINER"
+
+# --- 1. PostgreSQL ---------------------------------------------------------
+# Custom format (-Fc): compressed, and restorable with pg_restore --clean, which
+# lets the restore drop existing objects rather than erroring on conflicts.
+log "Dumping PostgreSQL database '$POSTGRES_DB'…"
+docker exec -e PGPASSWORD="${POSTGRES_PASSWORD:-tracepcap_pass}" "$PG_CONTAINER" \
+  pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc \
+  > "$STAGE/postgres.dump" \
+  || fail "pg_dump failed"
+
+# A near-empty dump means the dump silently produced nothing useful.
+PG_SIZE=$(stat -c%s "$STAGE/postgres.dump")
+[[ "$PG_SIZE" -gt 1024 ]] || fail "postgres dump is only ${PG_SIZE}B — refusing to record this as a good backup"
+log "  PostgreSQL dump: $(numfmt --to=iec "$PG_SIZE")"
+
+# --- 2. MinIO objects ------------------------------------------------------
+# `mc mirror` into a container-local path, then copy out. Credentials go in via
+# `mc alias set` inside the container so they never appear in a shell command
+# assembled on the host.
+log "Mirroring MinIO bucket '$MINIO_BUCKET'…"
+docker exec "$MINIO_CONTAINER" sh -c '
+  set -e
+  rm -rf /tmp/mc-backup
+  mkdir -p /tmp/mc-backup
+  mc alias set bk http://localhost:9000 "$0" "$1" >/dev/null
+  mc mirror --quiet "bk/$2" /tmp/mc-backup >/dev/null
+' "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" "$MINIO_BUCKET" \
+  || fail "mc mirror failed — check MINIO_ROOT_USER / MINIO_ROOT_PASSWORD"
+
+docker cp "$MINIO_CONTAINER:/tmp/mc-backup" "$STAGE/minio" >/dev/null \
+  || fail "copying MinIO objects out of the container failed"
+docker exec "$MINIO_CONTAINER" rm -rf /tmp/mc-backup || true
+
+OBJ_COUNT=$(find "$STAGE/minio" -type f | wc -l)
+log "  MinIO objects: $OBJ_COUNT"
+
+# --- 3. Config volume (signatures.yml) -------------------------------------
+# Best-effort: a deployment that has never customised signatures has no file,
+# and that is not a backup failure.
+log "Capturing config volume…"
+mkdir -p "$STAGE/config"
+if docker inspect -f '{{.State.Running}}' "$BACKEND_CONTAINER" 2>/dev/null | grep -q true; then
+  docker cp "$BACKEND_CONTAINER:/app/config/." "$STAGE/config/" 2>/dev/null \
+    || log "  (no config files to capture)"
+else
+  log "  (backend not running — skipping config capture)"
+fi
+
+# --- 4. Manifest -----------------------------------------------------------
+# Recorded so a restore can be checked against what was actually captured, and
+# so the restore script can refuse a backup taken from a different database.
+cat > "$STAGE/manifest.txt" <<EOF
+tracepcap_backup_version=1
+created_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)
+host=$(hostname)
+postgres_db=$POSTGRES_DB
+postgres_user=$POSTGRES_USER
+postgres_dump_bytes=$PG_SIZE
+minio_bucket=$MINIO_BUCKET
+minio_object_count=$OBJ_COUNT
+EOF
+
+# --- 5. Archive ------------------------------------------------------------
+log "Creating archive…"
+tar -czf "$ARCHIVE" -C "$WORK_DIR" "tracepcap-backup-$STAMP" \
+  || fail "tar failed"
+
+# Verify the archive is readable before we prune anything on its strength.
+tar -tzf "$ARCHIVE" >/dev/null || fail "archive verification failed — $ARCHIVE is unreadable"
+
+ARCHIVE_SIZE=$(stat -c%s "$ARCHIVE")
+log "Archive: $ARCHIVE ($(numfmt --to=iec "$ARCHIVE_SIZE"))"
+
+# --- 6. Prune old backups --------------------------------------------------
+# Only after the new archive is verified, so a failing backup never destroys the
+# last known-good one.
+if [[ "$BACKUP_RETENTION_DAYS" -gt 0 ]]; then
+  PRUNED=$(find "$BACKUP_DIR" -maxdepth 1 -name 'tracepcap-backup-*.tar.gz' \
+    -mtime "+$BACKUP_RETENTION_DAYS" -print -delete | wc -l)
+  [[ "$PRUNED" -gt 0 ]] && log "Pruned $PRUNED backup(s) older than ${BACKUP_RETENTION_DAYS}d"
+fi
+
+log "Backup complete."
diff --git a/scripts/restore.sh b/scripts/restore.sh
new file mode 100755
index 00000000..a14d113f
--- /dev/null
+++ b/scripts/restore.sh
@@ -0,0 +1,151 @@
+#!/usr/bin/env bash
+# restore.sh
+#
+# Restores a tarball produced by scripts/backup.sh: the PostgreSQL database, the
+# MinIO PCAP objects, and signatures.yml.
+#
+# Usage:
+#   bash scripts/restore.sh backups/tracepcap-backup-20260728-030000.tar.gz
+#   bash scripts/restore.sh --list backups/…tar.gz     # show manifest, restore nothing
+#   FORCE=1 bash scripts/restore.sh …                  # skip the confirmation prompt (cron/CI)
+#
+# THIS IS DESTRUCTIVE. It drops and recreates the objects in the target database
+# and overwrites objects in the bucket. It prompts before doing so unless FORCE=1.
+#
+# The backend should be stopped while this runs, so nothing writes underneath the
+# restore:
+#   docker compose stop backend
+#   bash scripts/restore.sh <archive>
+#   docker compose start backend
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT_DIR="$(dirname "$SCRIPT_DIR")"
+cd "$ROOT_DIR"
+
+if [[ -f .env ]]; then
+  # shellcheck disable=SC1091
+  set -a; source ./.env; set +a
+fi
+
+POSTGRES_DB="${POSTGRES_DB:-tracepcap}"
+POSTGRES_USER="${POSTGRES_USER:-tracepcap_user}"
+MINIO_ROOT_USER="${MINIO_ROOT_USER:-minioadmin}"
+MINIO_ROOT_PASSWORD="${MINIO_ROOT_PASSWORD:-minioadmin}"
+MINIO_BUCKET="${MINIO_BUCKET:-tracepcap-files}"
+
+PG_CONTAINER="${PG_CONTAINER:-tracepcap-postgres}"
+MINIO_CONTAINER="${MINIO_CONTAINER:-tracepcap-minio}"
+BACKEND_CONTAINER="${BACKEND_CONTAINER:-tracepcap-backend}"
+
+log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"; }
+fail() { printf '[%s] ERROR: %s\n' "$(date +%H:%M:%S)" "$*" >&2; exit 1; }
+
+LIST_ONLY=0
+[[ "${1:-}" == "--list" ]] && { LIST_ONLY=1; shift; }
+
+ARCHIVE="${1:-}"
+[[ -n "$ARCHIVE" ]] || fail "usage: bash scripts/restore.sh [--list] <archive.tar.gz>"
+[[ -f "$ARCHIVE" ]] || fail "archive not found: $ARCHIVE"
+
+WORK_DIR="$(mktemp -d)"
+cleanup() { rm -rf "$WORK_DIR"; }
+trap cleanup EXIT
+
+log "Extracting $ARCHIVE…"
+tar -xzf "$ARCHIVE" -C "$WORK_DIR" || fail "extraction failed — archive may be corrupt"
+
+STAGE="$(find "$WORK_DIR" -maxdepth 1 -type d -name 'tracepcap-backup-*' | head -1)"
+[[ -n "$STAGE" ]] || fail "archive does not look like a TracePcap backup (no tracepcap-backup-* directory)"
+[[ -f "$STAGE/manifest.txt" ]] || fail "archive is missing manifest.txt — refusing to restore an unrecognised archive"
+
+echo
+echo "--- backup manifest ---"
+cat "$STAGE/manifest.txt"
+echo "-----------------------"
+echo
+
+[[ "$LIST_ONLY" -eq 1 ]] && { log "--list given; nothing restored."; exit 0; }
+
+# Warn (don't block) when restoring into a differently-named database: it is a
+# legitimate thing to do when cloning to a staging box, but it should never be
+# silent, because the backend's DATABASE_URL must agree with the target.
+MANIFEST_DB=$(grep '^postgres_db=' "$STAGE/manifest.txt" | cut -d= -f2)
+if [[ "$MANIFEST_DB" != "$POSTGRES_DB" ]]; then
+  log "WARNING: backup came from database '$MANIFEST_DB' but this deployment uses '$POSTGRES_DB'."
+  log "         The backend will only see the data if DATABASE_URL points at '$POSTGRES_DB'."
+fi
+
+require_container() {
+  docker inspect -f '{{.State.Running}}' "$1" 2>/dev/null | grep -q true \
+    || fail "container '$1' is not running — start postgres and minio before restoring"
+}
+require_container "$PG_CONTAINER"
+require_container "$MINIO_CONTAINER"
+
+if docker inspect -f '{{.State.Running}}' "$BACKEND_CONTAINER" 2>/dev/null | grep -q true; then
+  log "WARNING: '$BACKEND_CONTAINER' is running. Stop it first (docker compose stop backend)"
+  log "         so it cannot write to the database while it is being replaced."
+fi
+
+if [[ "${FORCE:-0}" != "1" ]]; then
+  echo "This will REPLACE the contents of database '$POSTGRES_DB' and bucket '$MINIO_BUCKET'."
+  read -r -p "Type 'restore' to continue: " CONFIRM
+  [[ "$CONFIRM" == "restore" ]] || fail "aborted by user"
+fi
+
+# --- 1. PostgreSQL ---------------------------------------------------------
+# --clean --if-exists drops existing objects first, so restoring over a populated
+# database succeeds instead of failing on every duplicate.
+log "Restoring PostgreSQL…"
+docker exec -i -e PGPASSWORD="${POSTGRES_PASSWORD:-tracepcap_pass}" "$PG_CONTAINER" \
+  pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists --no-owner \
+  < "$STAGE/postgres.dump" \
+  || log "  pg_restore reported warnings (usually harmless 'does not exist' on --clean)"
+
+# Confirm the restore actually produced tables, rather than trusting the exit code.
+TABLE_COUNT=$(docker exec -e PGPASSWORD="${POSTGRES_PASSWORD:-tracepcap_pass}" "$PG_CONTAINER" \
+  psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc \
+  "SELECT count(*) FROM information_schema.tables WHERE table_schema='public'")
+[[ "$TABLE_COUNT" -gt 0 ]] || fail "restore left 0 tables in '$POSTGRES_DB' — restore did NOT succeed"
+log "  tables restored: $TABLE_COUNT"
+
+# --- 2. MinIO objects ------------------------------------------------------
+if [[ -d "$STAGE/minio" ]]; then
+  log "Restoring MinIO objects…"
+  docker exec "$MINIO_CONTAINER" rm -rf /tmp/mc-restore || true
+  docker cp "$STAGE/minio" "$MINIO_CONTAINER:/tmp/mc-restore" >/dev/null \
+    || fail "copying objects into the MinIO container failed"
+
+  docker exec "$MINIO_CONTAINER" sh -c '
+    set -e
+    mc alias set rs http://localhost:9000 "$0" "$1" >/dev/null
+    mc mb --ignore-existing "rs/$2" >/dev/null
+    mc mirror --overwrite --quiet /tmp/mc-restore "rs/$2" >/dev/null
+    rm -rf /tmp/mc-restore
+  ' "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" "$MINIO_BUCKET" \
+    || fail "mc mirror (restore) failed"
+
+  RESTORED=$(docker exec "$MINIO_CONTAINER" sh -c '
+    mc alias set rs http://localhost:9000 "$0" "$1" >/dev/null
+    mc ls --recursive "rs/$2" | wc -l
+  ' "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" "$MINIO_BUCKET")
+  log "  objects in bucket: $RESTORED"
+else
+  log "No MinIO objects in this archive — skipping."
+fi
+
+# --- 3. Config -------------------------------------------------------------
+if [[ -n "$(ls -A "$STAGE/config" 2>/dev/null)" ]]; then
+  if docker inspect -f '{{.State.Running}}' "$BACKEND_CONTAINER" 2>/dev/null | grep -q true; then
+    log "Restoring config files…"
+    docker cp "$STAGE/config/." "$BACKEND_CONTAINER:/app/config/" >/dev/null \
+      && log "  config restored (restart the backend to pick up signatures.yml)"
+  else
+    log "Backend not running — skipping config restore. Start it and re-run to restore signatures.yml."
+  fi
+fi
+
+log "Restore complete."
+log "Start the backend if it is stopped:  docker compose start backend"
diff --git a/scripts/tracepcap-backup.service b/scripts/tracepcap-backup.service
new file mode 100644
index 00000000..172306f2
--- /dev/null
+++ b/scripts/tracepcap-backup.service
@@ -0,0 +1,42 @@
+# systemd unit for scheduled TracePcap backups.
+#
+# Install (adjust the paths and User to match your deployment):
+#   sudo cp scripts/tracepcap-backup.service scripts/tracepcap-backup.timer /etc/systemd/system/
+#   sudo systemctl daemon-reload
+#   sudo systemctl enable --now tracepcap-backup.timer
+#
+# Check:
+#   systemctl list-timers tracepcap-backup.timer
+#   systemctl status tracepcap-backup.service
+#   journalctl -u tracepcap-backup.service -n 50
+#
+# Run once by hand (does not affect the schedule):
+#   sudo systemctl start tracepcap-backup.service
+
+[Unit]
+Description=TracePcap backup (PostgreSQL + MinIO + config)
+Documentation=https://github.com/NotYuSheng/TracePcap
+# Docker must be up: the script talks to the running containers, not the volumes.
+Requires=docker.service
+After=docker.service
+
+[Service]
+Type=oneshot
+
+# The user must be able to run `docker` (i.e. be in the docker group or root).
+User=ubuntu
+WorkingDirectory=/home/ubuntu/Desktop/TracePcap
+
+# Where archives land. Point this at mounted off-host storage (NAS, USB, remote
+# mount) — a backup on the same disk as the data does not survive that disk.
+Environment=BACKUP_DIR=/home/ubuntu/Desktop/TracePcap/backups
+Environment=BACKUP_RETENTION_DAYS=14
+
+ExecStart=/usr/bin/env bash scripts/backup.sh
+
+# A backup of a large capture set can take a while; don't let systemd kill it
+# mid-write and leave a truncated archive.
+TimeoutStartSec=3600
+
+[Install]
+WantedBy=multi-user.target
diff --git a/scripts/tracepcap-backup.timer b/scripts/tracepcap-backup.timer
new file mode 100644
index 00000000..aed9ce3f
--- /dev/null
+++ b/scripts/tracepcap-backup.timer
@@ -0,0 +1,23 @@
+# Nightly schedule for tracepcap-backup.service.
+#
+# See tracepcap-backup.service for installation instructions.
+
+[Unit]
+Description=Nightly TracePcap backup
+Documentation=https://github.com/NotYuSheng/TracePcap
+
+[Timer]
+# 02:30 daily — outside analyst working hours, so the dump is not competing with
+# analysis jobs for CPU and the database is quiet.
+OnCalendar=*-*-* 02:30:00
+
+# If the box was off at 02:30, run the backup once it comes back up rather than
+# silently skipping the night.
+Persistent=true
+
+# Spread the start over 5 minutes so several hosts backing up to the same NAS
+# don't all begin writing at once.
+RandomizedDelaySec=300
+
+[Install]
+WantedBy=timers.target

From 92f560e05e81f8a2601382169230cd5a72a6a048 Mon Sep 17 00:00:00 2001
From: NotYuSheng <notyusheng@gmail.com>
Date: Tue, 28 Jul 2026 11:36:35 +0800
Subject: [PATCH 2/2] fix/backup-env-parsing-and-restore-verification
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Addresses code review on #611. Two findings were genuine blockers, and one of
them reintroduced on the host exactly the bug #604 fixed for mc.

  .env was SOURCED, not parsed (backup.sh, restore.sh)
    `set -a; source ./.env` executes the file as shell. Verified: a password
    containing $(...) RUNS as a command, and one containing a space aborts the
    script with `spaces: command not found` before any backup happens. .env is
    Compose's format, not bash. Now parsed line by line — no execution path.

    Sourcing also inverted precedence: .env clobbered the real environment, so
    the systemd unit's BACKUP_DIR was silently discarded and every nightly
    archive would land on local disk instead of the off-host storage the docs
    call the whole point. The caller's environment now wins.

  Failed config restore printed NOTHING and still reported success (restore.sh)
    `docker cp … && log "restored"` with no || branch: under set -e a failed
    command in that position neither prints nor exits, so signatures.yml could
    go missing while the operator was told "Restore complete."

Also fixed:
  - Restore now inspects pg_restore stderr, separating real errors from the
    "does not exist" notices --clean always emits. Previously the exit code was
    ignored entirely and a half-restored database still passed the table check.
  - Bucket restore uses --remove, making it the replacement the prompt and docs
    promise rather than a merge that orphans objects the database can't see.
  - Restore compares the bucket against the manifest's object count, so a
    partial transfer fails instead of silently restoring some PCAPs.
  - Backup cross-checks captured objects against the bucket, so a wrong bucket
    name can't produce a "successful" backup containing no PCAPs.
  - Staging moved from /tmp into BACKUP_DIR: it holds an uncompressed copy of
    the whole capture set, and /tmp is tmpfs on many server images.
  - Pruning reads the timestamp in the filename, not mtime, which resets when
    archives are copied to a NAS and would let BACKUP_DIR grow unbounded.
  - flock guards concurrent runs (timer + manual) from truncating one archive.
  - Service: Restart=on-failure for boot-time catch-up runs that fire before the
    containers are ready, and the [Install] section removed so enabling the
    service alongside the timer can't add a backup on every boot.
  - .gitignore: anchor /backups/ to the repo root.

Verified: hostile .env ($(...) and spaces) completes with no execution; env
overrides .env; full destroy/restore cycle still returns exact row counts and a
byte-identical PCAP; tampered manifest and wrong bucket both rejected; pruning
removes an old-named archive with a fresh mtime; both systemd units pass
systemd-analyze verify.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
 .gitignore                         |  2 +-
 docs/operations/backup-restore.rst | 27 ++++++++-
 scripts/backup.sh                  | 94 ++++++++++++++++++++++++++----
 scripts/restore.sh                 | 82 +++++++++++++++++++++-----
 scripts/tracepcap-backup.service   | 13 ++++-
 5 files changed, 189 insertions(+), 29 deletions(-)

diff --git a/.gitignore b/.gitignore
index f23d2471..d3f7590f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -25,4 +25,4 @@ images/*.tar
 docs/_build/
 
 # Local backup archives (scripts/backup.sh)
-backups/
+/backups/
diff --git a/docs/operations/backup-restore.rst b/docs/operations/backup-restore.rst
index 3d3d33c4..5b3b30f1 100644
--- a/docs/operations/backup-restore.rst
+++ b/docs/operations/backup-restore.rst
@@ -72,6 +72,16 @@ Configuration
 Credentials are read from ``.env`` (or the environment), so the script needs no
 configuration of its own beyond the two variables above.
 
+``.env`` is parsed as Compose-format ``KEY=value`` lines rather than executed as
+shell, so passwords containing spaces, ``$``, backticks or quotes are handled
+safely. **Explicit environment variables take precedence over ``.env``** — the
+systemd unit's ``BACKUP_DIR`` wins over a ``BACKUP_DIR`` in the file.
+
+Staging happens inside ``BACKUP_DIR``, not ``/tmp``. The archive is assembled
+from an uncompressed copy of the data, so the directory needs roughly **twice the
+size of a backup** in free space during the run — and on hosts where ``/tmp`` is
+RAM-backed, staging there would exhaust memory on a large capture set.
+
 Running a Backup Manually
 -------------------------
 
@@ -114,9 +124,20 @@ Perform the restore, with the backend stopped so nothing writes underneath it:
    bash scripts/restore.sh backups/tracepcap-backup-20260728-110713.tar.gz
    docker compose start backend
 
-The script verifies the result rather than trusting exit codes — it counts the
-tables actually present afterwards and fails if the database came back empty. It
-also rejects corrupt archives and tarballs that are not TracePcap backups.
+The script verifies the result rather than trusting exit codes:
+
+- ``pg_restore`` stderr is inspected and real errors are separated from the
+  routine "does not exist" notices ``--clean`` produces, so a genuine failure
+  (disk full, truncated dump) is not mistaken for noise.
+- The number of tables present afterwards must be non-zero.
+- The object count in the bucket must match the manifest, so a partial transfer
+  fails loudly instead of quietly restoring some of your PCAPs.
+- Corrupt archives and tarballs that are not TracePcap backups are rejected.
+
+The bucket restore uses ``mc mirror --remove``, so it is a **replacement**, not a
+merge: objects not present in the archive are deleted. Rolling back to an earlier
+backup therefore leaves no orphaned PCAPs that the restored database has no rows
+for.
 
 Rehearsing a Restore
 --------------------
diff --git a/scripts/backup.sh b/scripts/backup.sh
index 81557607..89c78356 100755
--- a/scripts/backup.sh
+++ b/scripts/backup.sh
@@ -26,20 +26,47 @@ cd "$ROOT_DIR"
 
 # --- configuration ---------------------------------------------------------
 # Load .env if present so cron (which has almost no environment) sees the same
-# credentials as an interactive `docker compose` run. Existing environment
-# variables win, matching how compose itself resolves them.
-if [[ -f .env ]]; then
-  # shellcheck disable=SC1091
-  set -a; source ./.env; set +a
-fi
+# credentials as an interactive `docker compose` run.
+#
+# .env is Compose's format, NOT shell. It is parsed line by line rather than
+# sourced: `source` would execute it, so a password containing $(...) or a
+# backtick would run as a command, and one containing a space would abort the
+# script with a confusing syntax error. Parsing also lets the real environment
+# win — systemd's Environment= and any explicit `VAR=x scripts/backup.sh` must
+# override .env, not the reverse, or the unit's BACKUP_DIR would be silently
+# discarded and archives would land on the local disk instead of off-host storage.
+load_env_file() {
+  local file="$1" line key value
+  [[ -f "$file" ]] || return 0
+  while IFS= read -r line || [[ -n "$line" ]]; do
+    line="${line#"${line%%[![:space:]]*}"}"          # strip leading whitespace
+    [[ -z "$line" || "$line" == \#* ]] && continue   # skip blanks and comments
+    [[ "$line" == export\ * ]] && line="${line#export }"
+    [[ "$line" != *=* ]] && continue
+    key="${line%%=*}"
+    value="${line#*=}"
+    key="${key%"${key##*[![:space:]]}"}"             # strip trailing whitespace
+    [[ "$key" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || continue
+    # Strip one layer of matching quotes, as Compose does.
+    if [[ "$value" == \"*\" && ${#value} -ge 2 ]]; then value="${value:1:${#value}-2}"
+    elif [[ "$value" == \'*\' && ${#value} -ge 2 ]]; then value="${value:1:${#value}-2}"
+    fi
+    # Only set if not already present: the caller's environment takes precedence.
+    [[ -n "${!key+x}" ]] || export "$key=$value"
+  done < "$file"
+}
+load_env_file ./.env
 
 BACKUP_DIR="${BACKUP_DIR:-$ROOT_DIR/backups}"
 BACKUP_RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-14}"
 
 POSTGRES_DB="${POSTGRES_DB:-tracepcap}"
 POSTGRES_USER="${POSTGRES_USER:-tracepcap_user}"
+POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-tracepcap_pass}"
 MINIO_ROOT_USER="${MINIO_ROOT_USER:-minioadmin}"
 MINIO_ROOT_PASSWORD="${MINIO_ROOT_PASSWORD:-minioadmin}"
+# Matches the backend's hard-coded bucket in docker-compose.yml. Overriding this
+# without changing the backend would back up a bucket the app does not use.
 MINIO_BUCKET="${MINIO_BUCKET:-tracepcap-files}"
 
 PG_CONTAINER="${PG_CONTAINER:-tracepcap-postgres}"
@@ -47,10 +74,29 @@ MINIO_CONTAINER="${MINIO_CONTAINER:-tracepcap-minio}"
 BACKEND_CONTAINER="${BACKEND_CONTAINER:-tracepcap-backend}"
 
 STAMP="$(date +%Y%m%d-%H%M%S)"
-WORK_DIR="$(mktemp -d)"
-STAGE="$WORK_DIR/tracepcap-backup-$STAMP"
 ARCHIVE="$BACKUP_DIR/tracepcap-backup-$STAMP.tar.gz"
 
+mkdir -p "$BACKUP_DIR"
+
+# Serialise runs. The timer's Type=oneshot won't overlap itself, but the docs also
+# document a cron line and a manual `systemctl start`, so two runs can coincide —
+# and with one-second stamp resolution they could write the same archive path,
+# truncating each other. Take the lock or exit quietly; a skipped duplicate run is
+# correct behaviour, not an error worth waking anyone for.
+exec 9>"$BACKUP_DIR/.backup.lock"
+if ! flock -n 9; then
+  echo "Another backup is already running (lock held on $BACKUP_DIR/.backup.lock); exiting."
+  exit 0
+fi
+
+# Stage under BACKUP_DIR, not /tmp. The staging directory holds a full uncompressed
+# copy of every PCAP object before tar runs, so peak usage is roughly twice the data
+# set. /tmp is tmpfs (RAM-backed) on many server images, where a large capture set
+# would exhaust memory and take the running stack down with it. BACKUP_DIR is the
+# path the operator has already sized for this data.
+WORK_DIR="$(mktemp -d "$BACKUP_DIR/.staging-$STAMP.XXXXXX")"
+STAGE="$WORK_DIR/tracepcap-backup-$STAMP"
+
 # Always clean up the staging directory, including on failure — it holds a full
 # copy of the PCAP set and would otherwise fill /tmp after a few failed runs.
 cleanup() { rm -rf "$WORK_DIR"; }
@@ -105,6 +151,19 @@ docker exec "$MINIO_CONTAINER" rm -rf /tmp/mc-backup || true
 OBJ_COUNT=$(find "$STAGE/minio" -type f | wc -l)
 log "  MinIO objects: $OBJ_COUNT"
 
+# An empty mirror is indistinguishable from a successful one at the exit-code level:
+# a wrong bucket name mirrors nothing and still exits 0. Cross-check against what the
+# bucket actually holds, so a backup that captured no PCAPs is never recorded as good
+# while the database — which would restore fine — hides the loss until recovery day.
+BUCKET_COUNT=$(docker exec "$MINIO_CONTAINER" sh -c '
+  mc alias set bk http://localhost:9000 "$0" "$1" >/dev/null
+  mc ls --recursive "bk/$2" 2>/dev/null | wc -l
+' "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" "$MINIO_BUCKET" | tr -d '[:space:]')
+
+if [[ "$OBJ_COUNT" -ne "$BUCKET_COUNT" ]]; then
+  fail "captured $OBJ_COUNT object(s) but bucket '$MINIO_BUCKET' holds $BUCKET_COUNT — refusing to record an incomplete backup"
+fi
+
 # --- 3. Config volume (signatures.yml) -------------------------------------
 # Best-effort: a deployment that has never customised signatures has no file,
 # and that is not a backup failure.
@@ -145,9 +204,24 @@ log "Archive: $ARCHIVE ($(numfmt --to=iec "$ARCHIVE_SIZE"))"
 # --- 6. Prune old backups --------------------------------------------------
 # Only after the new archive is verified, so a failing backup never destroys the
 # last known-good one.
+# Age is read from the timestamp in the FILENAME, not the filesystem mtime. Copying
+# archives to a NAS (or restoring them from other media) rewrites mtime, which would
+# reset every archive's retention clock and let BACKUP_DIR grow until the disk fills
+# — breaking the next backup. The embedded stamp is authoritative and travels with
+# the file.
 if [[ "$BACKUP_RETENTION_DAYS" -gt 0 ]]; then
-  PRUNED=$(find "$BACKUP_DIR" -maxdepth 1 -name 'tracepcap-backup-*.tar.gz' \
-    -mtime "+$BACKUP_RETENTION_DAYS" -print -delete | wc -l)
+  CUTOFF=$(date -d "$BACKUP_RETENTION_DAYS days ago" +%Y%m%d)
+  PRUNED=0
+  for f in "$BACKUP_DIR"/tracepcap-backup-*.tar.gz; do
+    [[ -e "$f" ]] || continue
+    base="$(basename "$f")"
+    file_date="${base#tracepcap-backup-}"
+    file_date="${file_date%%-*}"
+    [[ "$file_date" =~ ^[0-9]{8}$ ]] || continue   # unparseable name: leave it alone
+    if [[ "$file_date" -lt "$CUTOFF" ]]; then
+      rm -f "$f" && PRUNED=$((PRUNED + 1))
+    fi
+  done
   [[ "$PRUNED" -gt 0 ]] && log "Pruned $PRUNED backup(s) older than ${BACKUP_RETENTION_DAYS}d"
 fi
 
diff --git a/scripts/restore.sh b/scripts/restore.sh
index a14d113f..ab0bfcc4 100755
--- a/scripts/restore.sh
+++ b/scripts/restore.sh
@@ -24,13 +24,32 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ROOT_DIR="$(dirname "$SCRIPT_DIR")"
 cd "$ROOT_DIR"
 
-if [[ -f .env ]]; then
-  # shellcheck disable=SC1091
-  set -a; source ./.env; set +a
-fi
+# .env is Compose's format, not shell — parsed rather than sourced so a password
+# containing $(...), a backtick or a space cannot execute or break the script, and
+# so the caller's environment wins over the file. See backup.sh for the full note.
+load_env_file() {
+  local file="$1" line key value
+  [[ -f "$file" ]] || return 0
+  while IFS= read -r line || [[ -n "$line" ]]; do
+    line="${line#"${line%%[![:space:]]*}"}"
+    [[ -z "$line" || "$line" == \#* ]] && continue
+    [[ "$line" == export\ * ]] && line="${line#export }"
+    [[ "$line" != *=* ]] && continue
+    key="${line%%=*}"
+    value="${line#*=}"
+    key="${key%"${key##*[![:space:]]}"}"
+    [[ "$key" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || continue
+    if [[ "$value" == \"*\" && ${#value} -ge 2 ]]; then value="${value:1:${#value}-2}"
+    elif [[ "$value" == \'*\' && ${#value} -ge 2 ]]; then value="${value:1:${#value}-2}"
+    fi
+    [[ -n "${!key+x}" ]] || export "$key=$value"
+  done < "$file"
+}
+load_env_file ./.env
 
 POSTGRES_DB="${POSTGRES_DB:-tracepcap}"
 POSTGRES_USER="${POSTGRES_USER:-tracepcap_user}"
+POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-tracepcap_pass}"
 MINIO_ROOT_USER="${MINIO_ROOT_USER:-minioadmin}"
 MINIO_ROOT_PASSWORD="${MINIO_ROOT_PASSWORD:-minioadmin}"
 MINIO_BUCKET="${MINIO_BUCKET:-tracepcap-files}"
@@ -77,6 +96,12 @@ if [[ "$MANIFEST_DB" != "$POSTGRES_DB" ]]; then
   log "         The backend will only see the data if DATABASE_URL points at '$POSTGRES_DB'."
 fi
 
+MANIFEST_BUCKET=$(grep '^minio_bucket=' "$STAGE/manifest.txt" | cut -d= -f2)
+if [[ -n "$MANIFEST_BUCKET" && "$MANIFEST_BUCKET" != "$MINIO_BUCKET" ]]; then
+  log "WARNING: backup came from bucket '$MANIFEST_BUCKET' but this deployment uses '$MINIO_BUCKET'."
+  log "         Objects will be restored into '$MINIO_BUCKET'; the backend must be configured to read it."
+fi
+
 require_container() {
   docker inspect -f '{{.State.Running}}' "$1" 2>/dev/null | grep -q true \
     || fail "container '$1' is not running — start postgres and minio before restoring"
@@ -99,13 +124,27 @@ fi
 # --clean --if-exists drops existing objects first, so restoring over a populated
 # database succeeds instead of failing on every duplicate.
 log "Restoring PostgreSQL…"
-docker exec -i -e PGPASSWORD="${POSTGRES_PASSWORD:-tracepcap_pass}" "$PG_CONTAINER" \
+PG_ERR="$WORK_DIR/pg_restore.err"
+docker exec -i -e PGPASSWORD="$POSTGRES_PASSWORD" "$PG_CONTAINER" \
   pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists --no-owner \
-  < "$STAGE/postgres.dump" \
-  || log "  pg_restore reported warnings (usually harmless 'does not exist' on --clean)"
+  < "$STAGE/postgres.dump" 2> "$PG_ERR" || true
+
+# pg_restore exits non-zero for harmless reasons under --clean --if-exists (it reports
+# every "does not exist" as an error), so the exit code alone is unusable. Filter those
+# out and treat anything remaining as a real failure — otherwise a genuine break (disk
+# full, truncated dump, wrong user) would be indistinguishable from routine noise, and
+# the table count below would still pass on the objects --clean had already dropped.
+REAL_ERRORS=$(grep -c '^pg_restore: error:' "$PG_ERR" 2>/dev/null || true)
+BENIGN=$(grep -c 'does not exist' "$PG_ERR" 2>/dev/null || true)
+if [[ "${REAL_ERRORS:-0}" -gt "${BENIGN:-0}" ]]; then
+  log "  pg_restore reported errors beyond the expected '--clean' notices:"
+  grep '^pg_restore: error:' "$PG_ERR" | grep -v 'does not exist' | head -5 | sed 's/^/    /'
+  fail "pg_restore failed — the database may be partially restored. Do NOT start the backend against it."
+fi
+[[ "${BENIGN:-0}" -gt 0 ]] && log "  ($BENIGN expected '--clean' notices ignored)"
 
-# Confirm the restore actually produced tables, rather than trusting the exit code.
-TABLE_COUNT=$(docker exec -e PGPASSWORD="${POSTGRES_PASSWORD:-tracepcap_pass}" "$PG_CONTAINER" \
+# Independently confirm the restore produced tables, rather than trusting exit codes.
+TABLE_COUNT=$(docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$PG_CONTAINER" \
   psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc \
   "SELECT count(*) FROM information_schema.tables WHERE table_schema='public'")
 [[ "$TABLE_COUNT" -gt 0 ]] || fail "restore left 0 tables in '$POSTGRES_DB' — restore did NOT succeed"
@@ -118,11 +157,15 @@ if [[ -d "$STAGE/minio" ]]; then
   docker cp "$STAGE/minio" "$MINIO_CONTAINER:/tmp/mc-restore" >/dev/null \
     || fail "copying objects into the MinIO container failed"
 
+  # --remove makes this a true replacement, matching what the prompt and the docs
+  # promise. Without it the restore is additive: rolling back a bad import would
+  # leave that import's objects orphaned in the bucket, invisible to the restored
+  # database (which has no rows for them) and never reclaimed.
   docker exec "$MINIO_CONTAINER" sh -c '
     set -e
     mc alias set rs http://localhost:9000 "$0" "$1" >/dev/null
     mc mb --ignore-existing "rs/$2" >/dev/null
-    mc mirror --overwrite --quiet /tmp/mc-restore "rs/$2" >/dev/null
+    mc mirror --overwrite --remove --quiet /tmp/mc-restore "rs/$2" >/dev/null
     rm -rf /tmp/mc-restore
   ' "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" "$MINIO_BUCKET" \
     || fail "mc mirror (restore) failed"
@@ -130,8 +173,16 @@ if [[ -d "$STAGE/minio" ]]; then
   RESTORED=$(docker exec "$MINIO_CONTAINER" sh -c '
     mc alias set rs http://localhost:9000 "$0" "$1" >/dev/null
     mc ls --recursive "rs/$2" | wc -l
-  ' "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" "$MINIO_BUCKET")
+  ' "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" "$MINIO_BUCKET" | tr -d '[:space:]')
   log "  objects in bucket: $RESTORED"
+
+  # The manifest records what was captured precisely so the restore can be checked
+  # against it. An interrupted transfer that restores 3 of 50 PCAPs must not report
+  # success — that is the failure the operator would only discover at recovery time.
+  EXPECTED_OBJ=$(grep '^minio_object_count=' "$STAGE/manifest.txt" | cut -d= -f2 | tr -d '[:space:]')
+  if [[ -n "$EXPECTED_OBJ" && "$RESTORED" != "$EXPECTED_OBJ" ]]; then
+    fail "expected $EXPECTED_OBJ object(s) from the manifest but the bucket holds $RESTORED — restore is INCOMPLETE"
+  fi
 else
   log "No MinIO objects in this archive — skipping."
 fi
@@ -140,8 +191,13 @@ fi
 if [[ -n "$(ls -A "$STAGE/config" 2>/dev/null)" ]]; then
   if docker inspect -f '{{.State.Running}}' "$BACKEND_CONTAINER" 2>/dev/null | grep -q true; then
     log "Restoring config files…"
-    docker cp "$STAGE/config/." "$BACKEND_CONTAINER:/app/config/" >/dev/null \
-      && log "  config restored (restart the backend to pick up signatures.yml)"
+    # Explicit || — a bare `&&` would swallow the failure and still let the script
+    # print "Restore complete.", leaving signatures.yml quietly missing.
+    if docker cp "$STAGE/config/." "$BACKEND_CONTAINER:/app/config/" >/dev/null 2>&1; then
+      log "  config restored (restart the backend to pick up signatures.yml)"
+    else
+      fail "config restore failed — signatures.yml was NOT restored"
+    fi
   else
     log "Backend not running — skipping config restore. Start it and re-run to restore signatures.yml."
   fi
diff --git a/scripts/tracepcap-backup.service b/scripts/tracepcap-backup.service
index 172306f2..eae329d9 100644
--- a/scripts/tracepcap-backup.service
+++ b/scripts/tracepcap-backup.service
@@ -38,5 +38,14 @@ ExecStart=/usr/bin/env bash scripts/backup.sh
 # mid-write and leave a truncated archive.
 TimeoutStartSec=3600
 
-[Install]
-WantedBy=multi-user.target
+# The timer's Persistent=true catches up a run missed while the box was off, which
+# means this unit can fire moments after boot — when dockerd is up (satisfying
+# After=docker.service) but the containers have not finished starting. The script
+# correctly refuses to back up a stopped stack, so retry a few times rather than
+# losing the caught-up run until the next night.
+Restart=on-failure
+RestartSec=120
+
+# NOTE: no [Install] section. This unit is started by tracepcap-backup.timer, which
+# carries WantedBy=timers.target. Enabling the *service* as well would additionally
+# run a backup on every boot — the least likely moment for the stack to be ready.