diff --git a/.gitignore b/.gitignore index 7880e486..d3f7590f 100644 --- a/.gitignore +++ b/.gitignore @@ -23,3 +23,6 @@ sample-files/*.pcap # Docker image tarballs (generated by scripts/pull-and-save-images.sh) images/*.tar docs/_build/ + +# Local backup archives (scripts/backup.sh) +/backups/ diff --git a/docs/operations/backup-restore.rst b/docs/operations/backup-restore.rst index 41c7bfaf..5b3b30f1 100644 --- a/docs/operations/backup-restore.rst +++ b/docs/operations/backup-restore.rst @@ -1,66 +1,229 @@ Backup & Restore ================ -TracePcap data lives in two places: +TracePcap holds three things that cannot be regenerated: -- **PostgreSQL** — conversation metadata, analysis results, geolocation data. -- **MinIO** — raw PCAP files (object storage). +- **PostgreSQL** — analysis results, conversations, monitor snapshots, and every + human-entered label or override. +- **MinIO** — the raw PCAP objects. +- **config volume** — ``signatures.yml``, your custom detection rules. -Backing Up ----------- +``scripts/backup.sh`` captures all three into one timestamped archive, and +``scripts/restore.sh`` puts them back. -Database Backup -~~~~~~~~~~~~~~~ +.. warning:: -.. code-block:: bash + **An untested backup is not a backup.** Rehearse the restore (below) before + the deployment holds data you care about — while a mistake is still free. - docker exec tracepcap-postgres \ - pg_dump -U tracepcap_user tracepcap > backup.sql +Automated Backups +----------------- -MinIO Backup (PCAP files) -~~~~~~~~~~~~~~~~~~~~~~~~~ +Install the bundled systemd timer for a nightly run at 02:30: .. code-block:: bash - docker exec tracepcap-minio \ - mc mirror minio/tracepcap-files ./backup-pcaps/ + sudo cp scripts/tracepcap-backup.service scripts/tracepcap-backup.timer \ + /etc/systemd/system/ + sudo systemctl daemon-reload + sudo systemctl enable --now tracepcap-backup.timer + +Edit ``User``, ``WorkingDirectory`` and ``BACKUP_DIR`` in the ``.service`` file to +match your deployment first. Verify the schedule and check on it with: + +.. code-block:: bash -Full Volume Backup -~~~~~~~~~~~~~~~~~~ + systemctl list-timers tracepcap-backup.timer + journalctl -u tracepcap-backup.service -n 50 -To back up all Docker named volumes at once: +If you prefer cron: .. code-block:: bash - sudo tar -czf tracepcap_backup.tar.gz /var/lib/docker/volumes/tracepcap_* + 30 2 * * * cd /path/to/TracePcap && bash scripts/backup.sh >> /var/log/tracepcap-backup.log 2>&1 + +The script exits non-zero on any failure, so both cron and systemd surface a +failed run rather than passing over it silently. + +.. important:: + + Set ``BACKUP_DIR`` to storage on a **different disk from the deployment** — + ideally a NAS or mounted remote share. A backup sitting on the same disk as + the data does not survive that disk failing, which is the main thing it is + there for. + +Configuration +~~~~~~~~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 30 20 50 + + * - Variable + - Default + - Purpose + * - ``BACKUP_DIR`` + - ``./backups`` + - Where archives are written. Point at off-host storage. + * - ``BACKUP_RETENTION_DAYS`` + - ``14`` + - Archives older than this are pruned. ``0`` disables pruning. + +Credentials are read from ``.env`` (or the environment), so the script needs no +configuration of its own beyond the two variables above. + +``.env`` is parsed as Compose-format ``KEY=value`` lines rather than executed as +shell, so passwords containing spaces, ``$``, backticks or quotes are handled +safely. **Explicit environment variables take precedence over ``.env``** — the +systemd unit's ``BACKUP_DIR`` wins over a ``BACKUP_DIR`` in the file. -Custom Signature Rules Backup -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Staging happens inside ``BACKUP_DIR``, not ``/tmp``. The archive is assembled +from an uncompressed copy of the data, so the directory needs roughly **twice the +size of a backup** in free space during the run — and on hosts where ``/tmp`` is +RAM-backed, staging there would exhaust memory on a large capture set. -The rules file is stored in the ``config_data`` volume. Back it up with: +Running a Backup Manually +------------------------- .. code-block:: bash - docker cp tracepcap-backend:/app/config/signatures.yml ./signatures.yml.bak + bash scripts/backup.sh + + # or to a specific destination + BACKUP_DIR=/mnt/nas/tracepcap bash scripts/backup.sh + +Output is a single archive, ``tracepcap-backup-.tar.gz``, containing +the PostgreSQL dump, the MinIO objects, the config files, and a ``manifest.txt`` +recording what was captured. + +The script refuses to record a backup as good if the database dump comes back +implausibly small, and verifies the finished archive is readable. **Old backups +are pruned only after the new one passes both checks**, so a failing run can +never destroy your last good archive. Restoring --------- -Database Restore -~~~~~~~~~~~~~~~~ +.. warning:: + + Restoring is **destructive**. It drops and recreates the objects in the target + database and overwrites objects in the bucket. The script prompts for + confirmation unless ``FORCE=1`` is set. + +Inspect an archive without changing anything: .. code-block:: bash - docker exec -i tracepcap-postgres \ - psql -U tracepcap_user tracepcap < backup.sql + bash scripts/restore.sh --list backups/tracepcap-backup-20260728-110713.tar.gz -MinIO Restore -~~~~~~~~~~~~~ +Perform the restore, with the backend stopped so nothing writes underneath it: .. code-block:: bash - docker exec tracepcap-minio \ - mc mirror ./backup-pcaps/ minio/tracepcap-files + docker compose stop backend + bash scripts/restore.sh backups/tracepcap-backup-20260728-110713.tar.gz + docker compose start backend + +The script verifies the result rather than trusting exit codes: + +- ``pg_restore`` stderr is inspected and real errors are separated from the + routine "does not exist" notices ``--clean`` produces, so a genuine failure + (disk full, truncated dump) is not mistaken for noise. +- The number of tables present afterwards must be non-zero. +- The object count in the bucket must match the manifest, so a partial transfer + fails loudly instead of quietly restoring some of your PCAPs. +- Corrupt archives and tarballs that are not TracePcap backups are rejected. + +The bucket restore uses ``mc mirror --remove``, so it is a **replacement**, not a +merge: objects not present in the archive are deleted. Rolling back to an earlier +backup therefore leaves no orphaned PCAPs that the restored database has no rows +for. + +Rehearsing a Restore +-------------------- + +Run this on a **non-production** deployment, or before the box holds real data. + +1. Upload a PCAP and let it finish analysing. +2. Take a backup: ``bash scripts/backup.sh`` +3. Note what you have: + + .. code-block:: bash + + docker exec -e PGPASSWORD=$POSTGRES_PASSWORD tracepcap-postgres \ + psql -U $POSTGRES_USER -d $POSTGRES_DB -tAc \ + "SELECT (SELECT count(*) FROM files), (SELECT count(*) FROM packets)" + +4. Simulate the loss — stop the backend, drop the schema, empty the bucket. +5. Restore, restart the backend, and confirm the counts match, the file list + renders, and a PCAP downloads intact. + +This procedure was used to validate the scripts: a full drop of the ``public`` +schema and a complete bucket wipe, followed by a restore that returned the exact +row counts and a **byte-identical** PCAP (verified by MD5). + +Recovery Objectives +------------------- + +.. list-table:: + :header-rows: 1 + :widths: 20 25 55 + + * - Objective + - Value + - Notes + * - **RPO** + - Up to 24 h + - Set by the nightly schedule. Most lost work is re-importable — PCAPs can + be re-uploaded and re-analysed — but **human labels and overrides entered + since the last backup cannot be recovered.** + * - **RTO** + - Minutes + - Dominated by archive size, not by the procedure. Restore is one command. + +.. note:: + + Backup and restore both complete in about a second on a trivial dataset, + which tells you the procedure has no fixed overhead — not how long your data + will take. Time a real backup against your own capture volume and record the + result. Expect it to track archive size, which is roughly the PCAP volume plus + a database of comparable size (see :doc:`scalability`). + +Reducing the RPO below 24 hours means running the timer more frequently. True +point-in-time recovery would require PostgreSQL WAL archiving, which is +deliberately not configured — a trade for a single-server deployment whose +primary data is re-importable. See `issue #379 +`_. + +What Is Not Covered +------------------- + +- **Point-in-time recovery.** Restores land on a nightly boundary, not an + arbitrary moment. +- **Keycloak users.** The auth overlays persist Keycloak's H2 store in the + ``keycloak_data`` volume, which these scripts do not capture. Accounts created + in the admin console must be re-provisioned, or that volume backed up + separately. The seeded realm import is in version control. +- **Redundancy.** Backups protect against deletion and corruption, not against + hardware failure taking the service offline. See :doc:`storage-redundancy`. + +Manual Procedures +----------------- + +The individual commands, if you need to do something the scripts do not cover: + +.. code-block:: bash + + # PostgreSQL + docker exec tracepcap-postgres pg_dump -U tracepcap_user tracepcap > backup.sql + docker exec -i tracepcap-postgres psql -U tracepcap_user tracepcap < backup.sql + + # MinIO + docker exec tracepcap-minio mc mirror minio/tracepcap-files ./backup-pcaps/ + docker exec tracepcap-minio mc mirror ./backup-pcaps/ minio/tracepcap-files + + # Signature rules + docker cp tracepcap-backend:/app/config/signatures.yml ./signatures.yml.bak Export Options -------------- diff --git a/scripts/backup.sh b/scripts/backup.sh new file mode 100755 index 00000000..89c78356 --- /dev/null +++ b/scripts/backup.sh @@ -0,0 +1,228 @@ +#!/usr/bin/env bash +# backup.sh +# +# Backs up everything TracePcap cannot regenerate: +# 1. PostgreSQL — analysis results, conversations, monitor snapshots, human labels +# 2. MinIO — the raw PCAP objects +# 3. config_data — signatures.yml (custom detection rules) +# +# Produces ONE timestamped tarball per run, then prunes runs older than +# BACKUP_RETENTION_DAYS. Designed to be driven by cron or a systemd timer; see +# docs/operations/backup-restore.rst. +# +# Usage: +# bash scripts/backup.sh # writes to ./backups +# BACKUP_DIR=/mnt/nas/tracepcap bash scripts/backup.sh +# +# Credentials are read from .env (or the environment) — never hard-coded here. +# +# Exit codes: 0 ok, non-zero on any failure (so cron reports it). + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT_DIR="$(dirname "$SCRIPT_DIR")" +cd "$ROOT_DIR" + +# --- configuration --------------------------------------------------------- +# Load .env if present so cron (which has almost no environment) sees the same +# credentials as an interactive `docker compose` run. +# +# .env is Compose's format, NOT shell. It is parsed line by line rather than +# sourced: `source` would execute it, so a password containing $(...) or a +# backtick would run as a command, and one containing a space would abort the +# script with a confusing syntax error. Parsing also lets the real environment +# win — systemd's Environment= and any explicit `VAR=x scripts/backup.sh` must +# override .env, not the reverse, or the unit's BACKUP_DIR would be silently +# discarded and archives would land on the local disk instead of off-host storage. +load_env_file() { + local file="$1" line key value + [[ -f "$file" ]] || return 0 + while IFS= read -r line || [[ -n "$line" ]]; do + line="${line#"${line%%[![:space:]]*}"}" # strip leading whitespace + [[ -z "$line" || "$line" == \#* ]] && continue # skip blanks and comments + [[ "$line" == export\ * ]] && line="${line#export }" + [[ "$line" != *=* ]] && continue + key="${line%%=*}" + value="${line#*=}" + key="${key%"${key##*[![:space:]]}"}" # strip trailing whitespace + [[ "$key" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || continue + # Strip one layer of matching quotes, as Compose does. + if [[ "$value" == \"*\" && ${#value} -ge 2 ]]; then value="${value:1:${#value}-2}" + elif [[ "$value" == \'*\' && ${#value} -ge 2 ]]; then value="${value:1:${#value}-2}" + fi + # Only set if not already present: the caller's environment takes precedence. + [[ -n "${!key+x}" ]] || export "$key=$value" + done < "$file" +} +load_env_file ./.env + +BACKUP_DIR="${BACKUP_DIR:-$ROOT_DIR/backups}" +BACKUP_RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-14}" + +POSTGRES_DB="${POSTGRES_DB:-tracepcap}" +POSTGRES_USER="${POSTGRES_USER:-tracepcap_user}" +POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-tracepcap_pass}" +MINIO_ROOT_USER="${MINIO_ROOT_USER:-minioadmin}" +MINIO_ROOT_PASSWORD="${MINIO_ROOT_PASSWORD:-minioadmin}" +# Matches the backend's hard-coded bucket in docker-compose.yml. Overriding this +# without changing the backend would back up a bucket the app does not use. +MINIO_BUCKET="${MINIO_BUCKET:-tracepcap-files}" + +PG_CONTAINER="${PG_CONTAINER:-tracepcap-postgres}" +MINIO_CONTAINER="${MINIO_CONTAINER:-tracepcap-minio}" +BACKEND_CONTAINER="${BACKEND_CONTAINER:-tracepcap-backend}" + +STAMP="$(date +%Y%m%d-%H%M%S)" +ARCHIVE="$BACKUP_DIR/tracepcap-backup-$STAMP.tar.gz" + +mkdir -p "$BACKUP_DIR" + +# Serialise runs. The timer's Type=oneshot won't overlap itself, but the docs also +# document a cron line and a manual `systemctl start`, so two runs can coincide — +# and with one-second stamp resolution they could write the same archive path, +# truncating each other. Take the lock or exit quietly; a skipped duplicate run is +# correct behaviour, not an error worth waking anyone for. +exec 9>"$BACKUP_DIR/.backup.lock" +if ! flock -n 9; then + echo "Another backup is already running (lock held on $BACKUP_DIR/.backup.lock); exiting." + exit 0 +fi + +# Stage under BACKUP_DIR, not /tmp. The staging directory holds a full uncompressed +# copy of every PCAP object before tar runs, so peak usage is roughly twice the data +# set. /tmp is tmpfs (RAM-backed) on many server images, where a large capture set +# would exhaust memory and take the running stack down with it. BACKUP_DIR is the +# path the operator has already sized for this data. +WORK_DIR="$(mktemp -d "$BACKUP_DIR/.staging-$STAMP.XXXXXX")" +STAGE="$WORK_DIR/tracepcap-backup-$STAMP" + +# Always clean up the staging directory, including on failure — it holds a full +# copy of the PCAP set and would otherwise fill /tmp after a few failed runs. +cleanup() { rm -rf "$WORK_DIR"; } +trap cleanup EXIT + +log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"; } +fail() { printf '[%s] ERROR: %s\n' "$(date +%H:%M:%S)" "$*" >&2; exit 1; } + +require_container() { + docker inspect -f '{{.State.Running}}' "$1" 2>/dev/null | grep -q true \ + || fail "container '$1' is not running — start the stack before backing up" +} + +mkdir -p "$BACKUP_DIR" "$STAGE" + +log "Backup starting → $ARCHIVE" +require_container "$PG_CONTAINER" +require_container "$MINIO_CONTAINER" + +# --- 1. PostgreSQL --------------------------------------------------------- +# Custom format (-Fc): compressed, and restorable with pg_restore --clean, which +# lets the restore drop existing objects rather than erroring on conflicts. +log "Dumping PostgreSQL database '$POSTGRES_DB'…" +docker exec -e PGPASSWORD="${POSTGRES_PASSWORD:-tracepcap_pass}" "$PG_CONTAINER" \ + pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc \ + > "$STAGE/postgres.dump" \ + || fail "pg_dump failed" + +# A near-empty dump means the dump silently produced nothing useful. +PG_SIZE=$(stat -c%s "$STAGE/postgres.dump") +[[ "$PG_SIZE" -gt 1024 ]] || fail "postgres dump is only ${PG_SIZE}B — refusing to record this as a good backup" +log " PostgreSQL dump: $(numfmt --to=iec "$PG_SIZE")" + +# --- 2. MinIO objects ------------------------------------------------------ +# `mc mirror` into a container-local path, then copy out. Credentials go in via +# `mc alias set` inside the container so they never appear in a shell command +# assembled on the host. +log "Mirroring MinIO bucket '$MINIO_BUCKET'…" +docker exec "$MINIO_CONTAINER" sh -c ' + set -e + rm -rf /tmp/mc-backup + mkdir -p /tmp/mc-backup + mc alias set bk http://localhost:9000 "$0" "$1" >/dev/null + mc mirror --quiet "bk/$2" /tmp/mc-backup >/dev/null +' "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" "$MINIO_BUCKET" \ + || fail "mc mirror failed — check MINIO_ROOT_USER / MINIO_ROOT_PASSWORD" + +docker cp "$MINIO_CONTAINER:/tmp/mc-backup" "$STAGE/minio" >/dev/null \ + || fail "copying MinIO objects out of the container failed" +docker exec "$MINIO_CONTAINER" rm -rf /tmp/mc-backup || true + +OBJ_COUNT=$(find "$STAGE/minio" -type f | wc -l) +log " MinIO objects: $OBJ_COUNT" + +# An empty mirror is indistinguishable from a successful one at the exit-code level: +# a wrong bucket name mirrors nothing and still exits 0. Cross-check against what the +# bucket actually holds, so a backup that captured no PCAPs is never recorded as good +# while the database — which would restore fine — hides the loss until recovery day. +BUCKET_COUNT=$(docker exec "$MINIO_CONTAINER" sh -c ' + mc alias set bk http://localhost:9000 "$0" "$1" >/dev/null + mc ls --recursive "bk/$2" 2>/dev/null | wc -l +' "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" "$MINIO_BUCKET" | tr -d '[:space:]') + +if [[ "$OBJ_COUNT" -ne "$BUCKET_COUNT" ]]; then + fail "captured $OBJ_COUNT object(s) but bucket '$MINIO_BUCKET' holds $BUCKET_COUNT — refusing to record an incomplete backup" +fi + +# --- 3. Config volume (signatures.yml) ------------------------------------- +# Best-effort: a deployment that has never customised signatures has no file, +# and that is not a backup failure. +log "Capturing config volume…" +mkdir -p "$STAGE/config" +if docker inspect -f '{{.State.Running}}' "$BACKEND_CONTAINER" 2>/dev/null | grep -q true; then + docker cp "$BACKEND_CONTAINER:/app/config/." "$STAGE/config/" 2>/dev/null \ + || log " (no config files to capture)" +else + log " (backend not running — skipping config capture)" +fi + +# --- 4. Manifest ----------------------------------------------------------- +# Recorded so a restore can be checked against what was actually captured, and +# so the restore script can refuse a backup taken from a different database. +cat > "$STAGE/manifest.txt" </dev/null || fail "archive verification failed — $ARCHIVE is unreadable" + +ARCHIVE_SIZE=$(stat -c%s "$ARCHIVE") +log "Archive: $ARCHIVE ($(numfmt --to=iec "$ARCHIVE_SIZE"))" + +# --- 6. Prune old backups -------------------------------------------------- +# Only after the new archive is verified, so a failing backup never destroys the +# last known-good one. +# Age is read from the timestamp in the FILENAME, not the filesystem mtime. Copying +# archives to a NAS (or restoring them from other media) rewrites mtime, which would +# reset every archive's retention clock and let BACKUP_DIR grow until the disk fills +# — breaking the next backup. The embedded stamp is authoritative and travels with +# the file. +if [[ "$BACKUP_RETENTION_DAYS" -gt 0 ]]; then + CUTOFF=$(date -d "$BACKUP_RETENTION_DAYS days ago" +%Y%m%d) + PRUNED=0 + for f in "$BACKUP_DIR"/tracepcap-backup-*.tar.gz; do + [[ -e "$f" ]] || continue + base="$(basename "$f")" + file_date="${base#tracepcap-backup-}" + file_date="${file_date%%-*}" + [[ "$file_date" =~ ^[0-9]{8}$ ]] || continue # unparseable name: leave it alone + if [[ "$file_date" -lt "$CUTOFF" ]]; then + rm -f "$f" && PRUNED=$((PRUNED + 1)) + fi + done + [[ "$PRUNED" -gt 0 ]] && log "Pruned $PRUNED backup(s) older than ${BACKUP_RETENTION_DAYS}d" +fi + +log "Backup complete." diff --git a/scripts/restore.sh b/scripts/restore.sh new file mode 100755 index 00000000..ab0bfcc4 --- /dev/null +++ b/scripts/restore.sh @@ -0,0 +1,207 @@ +#!/usr/bin/env bash +# restore.sh +# +# Restores a tarball produced by scripts/backup.sh: the PostgreSQL database, the +# MinIO PCAP objects, and signatures.yml. +# +# Usage: +# bash scripts/restore.sh backups/tracepcap-backup-20260728-030000.tar.gz +# bash scripts/restore.sh --list backups/…tar.gz # show manifest, restore nothing +# FORCE=1 bash scripts/restore.sh … # skip the confirmation prompt (cron/CI) +# +# THIS IS DESTRUCTIVE. It drops and recreates the objects in the target database +# and overwrites objects in the bucket. It prompts before doing so unless FORCE=1. +# +# The backend should be stopped while this runs, so nothing writes underneath the +# restore: +# docker compose stop backend +# bash scripts/restore.sh +# docker compose start backend + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT_DIR="$(dirname "$SCRIPT_DIR")" +cd "$ROOT_DIR" + +# .env is Compose's format, not shell — parsed rather than sourced so a password +# containing $(...), a backtick or a space cannot execute or break the script, and +# so the caller's environment wins over the file. See backup.sh for the full note. +load_env_file() { + local file="$1" line key value + [[ -f "$file" ]] || return 0 + while IFS= read -r line || [[ -n "$line" ]]; do + line="${line#"${line%%[![:space:]]*}"}" + [[ -z "$line" || "$line" == \#* ]] && continue + [[ "$line" == export\ * ]] && line="${line#export }" + [[ "$line" != *=* ]] && continue + key="${line%%=*}" + value="${line#*=}" + key="${key%"${key##*[![:space:]]}"}" + [[ "$key" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || continue + if [[ "$value" == \"*\" && ${#value} -ge 2 ]]; then value="${value:1:${#value}-2}" + elif [[ "$value" == \'*\' && ${#value} -ge 2 ]]; then value="${value:1:${#value}-2}" + fi + [[ -n "${!key+x}" ]] || export "$key=$value" + done < "$file" +} +load_env_file ./.env + +POSTGRES_DB="${POSTGRES_DB:-tracepcap}" +POSTGRES_USER="${POSTGRES_USER:-tracepcap_user}" +POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-tracepcap_pass}" +MINIO_ROOT_USER="${MINIO_ROOT_USER:-minioadmin}" +MINIO_ROOT_PASSWORD="${MINIO_ROOT_PASSWORD:-minioadmin}" +MINIO_BUCKET="${MINIO_BUCKET:-tracepcap-files}" + +PG_CONTAINER="${PG_CONTAINER:-tracepcap-postgres}" +MINIO_CONTAINER="${MINIO_CONTAINER:-tracepcap-minio}" +BACKEND_CONTAINER="${BACKEND_CONTAINER:-tracepcap-backend}" + +log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"; } +fail() { printf '[%s] ERROR: %s\n' "$(date +%H:%M:%S)" "$*" >&2; exit 1; } + +LIST_ONLY=0 +[[ "${1:-}" == "--list" ]] && { LIST_ONLY=1; shift; } + +ARCHIVE="${1:-}" +[[ -n "$ARCHIVE" ]] || fail "usage: bash scripts/restore.sh [--list] " +[[ -f "$ARCHIVE" ]] || fail "archive not found: $ARCHIVE" + +WORK_DIR="$(mktemp -d)" +cleanup() { rm -rf "$WORK_DIR"; } +trap cleanup EXIT + +log "Extracting $ARCHIVE…" +tar -xzf "$ARCHIVE" -C "$WORK_DIR" || fail "extraction failed — archive may be corrupt" + +STAGE="$(find "$WORK_DIR" -maxdepth 1 -type d -name 'tracepcap-backup-*' | head -1)" +[[ -n "$STAGE" ]] || fail "archive does not look like a TracePcap backup (no tracepcap-backup-* directory)" +[[ -f "$STAGE/manifest.txt" ]] || fail "archive is missing manifest.txt — refusing to restore an unrecognised archive" + +echo +echo "--- backup manifest ---" +cat "$STAGE/manifest.txt" +echo "-----------------------" +echo + +[[ "$LIST_ONLY" -eq 1 ]] && { log "--list given; nothing restored."; exit 0; } + +# Warn (don't block) when restoring into a differently-named database: it is a +# legitimate thing to do when cloning to a staging box, but it should never be +# silent, because the backend's DATABASE_URL must agree with the target. +MANIFEST_DB=$(grep '^postgres_db=' "$STAGE/manifest.txt" | cut -d= -f2) +if [[ "$MANIFEST_DB" != "$POSTGRES_DB" ]]; then + log "WARNING: backup came from database '$MANIFEST_DB' but this deployment uses '$POSTGRES_DB'." + log " The backend will only see the data if DATABASE_URL points at '$POSTGRES_DB'." +fi + +MANIFEST_BUCKET=$(grep '^minio_bucket=' "$STAGE/manifest.txt" | cut -d= -f2) +if [[ -n "$MANIFEST_BUCKET" && "$MANIFEST_BUCKET" != "$MINIO_BUCKET" ]]; then + log "WARNING: backup came from bucket '$MANIFEST_BUCKET' but this deployment uses '$MINIO_BUCKET'." + log " Objects will be restored into '$MINIO_BUCKET'; the backend must be configured to read it." +fi + +require_container() { + docker inspect -f '{{.State.Running}}' "$1" 2>/dev/null | grep -q true \ + || fail "container '$1' is not running — start postgres and minio before restoring" +} +require_container "$PG_CONTAINER" +require_container "$MINIO_CONTAINER" + +if docker inspect -f '{{.State.Running}}' "$BACKEND_CONTAINER" 2>/dev/null | grep -q true; then + log "WARNING: '$BACKEND_CONTAINER' is running. Stop it first (docker compose stop backend)" + log " so it cannot write to the database while it is being replaced." +fi + +if [[ "${FORCE:-0}" != "1" ]]; then + echo "This will REPLACE the contents of database '$POSTGRES_DB' and bucket '$MINIO_BUCKET'." + read -r -p "Type 'restore' to continue: " CONFIRM + [[ "$CONFIRM" == "restore" ]] || fail "aborted by user" +fi + +# --- 1. PostgreSQL --------------------------------------------------------- +# --clean --if-exists drops existing objects first, so restoring over a populated +# database succeeds instead of failing on every duplicate. +log "Restoring PostgreSQL…" +PG_ERR="$WORK_DIR/pg_restore.err" +docker exec -i -e PGPASSWORD="$POSTGRES_PASSWORD" "$PG_CONTAINER" \ + pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists --no-owner \ + < "$STAGE/postgres.dump" 2> "$PG_ERR" || true + +# pg_restore exits non-zero for harmless reasons under --clean --if-exists (it reports +# every "does not exist" as an error), so the exit code alone is unusable. Filter those +# out and treat anything remaining as a real failure — otherwise a genuine break (disk +# full, truncated dump, wrong user) would be indistinguishable from routine noise, and +# the table count below would still pass on the objects --clean had already dropped. +REAL_ERRORS=$(grep -c '^pg_restore: error:' "$PG_ERR" 2>/dev/null || true) +BENIGN=$(grep -c 'does not exist' "$PG_ERR" 2>/dev/null || true) +if [[ "${REAL_ERRORS:-0}" -gt "${BENIGN:-0}" ]]; then + log " pg_restore reported errors beyond the expected '--clean' notices:" + grep '^pg_restore: error:' "$PG_ERR" | grep -v 'does not exist' | head -5 | sed 's/^/ /' + fail "pg_restore failed — the database may be partially restored. Do NOT start the backend against it." +fi +[[ "${BENIGN:-0}" -gt 0 ]] && log " ($BENIGN expected '--clean' notices ignored)" + +# Independently confirm the restore produced tables, rather than trusting exit codes. +TABLE_COUNT=$(docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$PG_CONTAINER" \ + psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc \ + "SELECT count(*) FROM information_schema.tables WHERE table_schema='public'") +[[ "$TABLE_COUNT" -gt 0 ]] || fail "restore left 0 tables in '$POSTGRES_DB' — restore did NOT succeed" +log " tables restored: $TABLE_COUNT" + +# --- 2. MinIO objects ------------------------------------------------------ +if [[ -d "$STAGE/minio" ]]; then + log "Restoring MinIO objects…" + docker exec "$MINIO_CONTAINER" rm -rf /tmp/mc-restore || true + docker cp "$STAGE/minio" "$MINIO_CONTAINER:/tmp/mc-restore" >/dev/null \ + || fail "copying objects into the MinIO container failed" + + # --remove makes this a true replacement, matching what the prompt and the docs + # promise. Without it the restore is additive: rolling back a bad import would + # leave that import's objects orphaned in the bucket, invisible to the restored + # database (which has no rows for them) and never reclaimed. + docker exec "$MINIO_CONTAINER" sh -c ' + set -e + mc alias set rs http://localhost:9000 "$0" "$1" >/dev/null + mc mb --ignore-existing "rs/$2" >/dev/null + mc mirror --overwrite --remove --quiet /tmp/mc-restore "rs/$2" >/dev/null + rm -rf /tmp/mc-restore + ' "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" "$MINIO_BUCKET" \ + || fail "mc mirror (restore) failed" + + RESTORED=$(docker exec "$MINIO_CONTAINER" sh -c ' + mc alias set rs http://localhost:9000 "$0" "$1" >/dev/null + mc ls --recursive "rs/$2" | wc -l + ' "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" "$MINIO_BUCKET" | tr -d '[:space:]') + log " objects in bucket: $RESTORED" + + # The manifest records what was captured precisely so the restore can be checked + # against it. An interrupted transfer that restores 3 of 50 PCAPs must not report + # success — that is the failure the operator would only discover at recovery time. + EXPECTED_OBJ=$(grep '^minio_object_count=' "$STAGE/manifest.txt" | cut -d= -f2 | tr -d '[:space:]') + if [[ -n "$EXPECTED_OBJ" && "$RESTORED" != "$EXPECTED_OBJ" ]]; then + fail "expected $EXPECTED_OBJ object(s) from the manifest but the bucket holds $RESTORED — restore is INCOMPLETE" + fi +else + log "No MinIO objects in this archive — skipping." +fi + +# --- 3. Config ------------------------------------------------------------- +if [[ -n "$(ls -A "$STAGE/config" 2>/dev/null)" ]]; then + if docker inspect -f '{{.State.Running}}' "$BACKEND_CONTAINER" 2>/dev/null | grep -q true; then + log "Restoring config files…" + # Explicit || — a bare `&&` would swallow the failure and still let the script + # print "Restore complete.", leaving signatures.yml quietly missing. + if docker cp "$STAGE/config/." "$BACKEND_CONTAINER:/app/config/" >/dev/null 2>&1; then + log " config restored (restart the backend to pick up signatures.yml)" + else + fail "config restore failed — signatures.yml was NOT restored" + fi + else + log "Backend not running — skipping config restore. Start it and re-run to restore signatures.yml." + fi +fi + +log "Restore complete." +log "Start the backend if it is stopped: docker compose start backend" diff --git a/scripts/tracepcap-backup.service b/scripts/tracepcap-backup.service new file mode 100644 index 00000000..eae329d9 --- /dev/null +++ b/scripts/tracepcap-backup.service @@ -0,0 +1,51 @@ +# systemd unit for scheduled TracePcap backups. +# +# Install (adjust the paths and User to match your deployment): +# sudo cp scripts/tracepcap-backup.service scripts/tracepcap-backup.timer /etc/systemd/system/ +# sudo systemctl daemon-reload +# sudo systemctl enable --now tracepcap-backup.timer +# +# Check: +# systemctl list-timers tracepcap-backup.timer +# systemctl status tracepcap-backup.service +# journalctl -u tracepcap-backup.service -n 50 +# +# Run once by hand (does not affect the schedule): +# sudo systemctl start tracepcap-backup.service + +[Unit] +Description=TracePcap backup (PostgreSQL + MinIO + config) +Documentation=https://github.com/NotYuSheng/TracePcap +# Docker must be up: the script talks to the running containers, not the volumes. +Requires=docker.service +After=docker.service + +[Service] +Type=oneshot + +# The user must be able to run `docker` (i.e. be in the docker group or root). +User=ubuntu +WorkingDirectory=/home/ubuntu/Desktop/TracePcap + +# Where archives land. Point this at mounted off-host storage (NAS, USB, remote +# mount) — a backup on the same disk as the data does not survive that disk. +Environment=BACKUP_DIR=/home/ubuntu/Desktop/TracePcap/backups +Environment=BACKUP_RETENTION_DAYS=14 + +ExecStart=/usr/bin/env bash scripts/backup.sh + +# A backup of a large capture set can take a while; don't let systemd kill it +# mid-write and leave a truncated archive. +TimeoutStartSec=3600 + +# The timer's Persistent=true catches up a run missed while the box was off, which +# means this unit can fire moments after boot — when dockerd is up (satisfying +# After=docker.service) but the containers have not finished starting. The script +# correctly refuses to back up a stopped stack, so retry a few times rather than +# losing the caught-up run until the next night. +Restart=on-failure +RestartSec=120 + +# NOTE: no [Install] section. This unit is started by tracepcap-backup.timer, which +# carries WantedBy=timers.target. Enabling the *service* as well would additionally +# run a backup on every boot — the least likely moment for the stack to be ready. diff --git a/scripts/tracepcap-backup.timer b/scripts/tracepcap-backup.timer new file mode 100644 index 00000000..aed9ce3f --- /dev/null +++ b/scripts/tracepcap-backup.timer @@ -0,0 +1,23 @@ +# Nightly schedule for tracepcap-backup.service. +# +# See tracepcap-backup.service for installation instructions. + +[Unit] +Description=Nightly TracePcap backup +Documentation=https://github.com/NotYuSheng/TracePcap + +[Timer] +# 02:30 daily — outside analyst working hours, so the dump is not competing with +# analysis jobs for CPU and the database is quiet. +OnCalendar=*-*-* 02:30:00 + +# If the box was off at 02:30, run the backup once it comes back up rather than +# silently skipping the night. +Persistent=true + +# Spread the start over 5 minutes so several hosts backing up to the same NAS +# don't all begin writing at once. +RandomizedDelaySec=300 + +[Install] +WantedBy=timers.target