CERT.PL Warning List Scraper

Automatically downloads and maintains an archive of CERT Polska phishing domain action logs from their public repository.

Features

• Downloads CERT.PL domain action logs for every year from 2020 to the current year
• Validates each file with SHA-256 checksums - skips writes when nothing has changed
• Maintains structured JSON metadata per file (size, checksum, download time, success history)
• Generates formatted summary reports
• Sets up cron jobs for fully automated operation

Requirements

• Bash
• curl
• jq
• shasum

Secrets and environment variables

This scraper requires no secrets and no environment variables. It runs on a local or self-hosted machine via cron, makes anonymous curl requests to the public hole.cert.pl endpoints, and writes only to the local filesystem.

• GitHub: none (the repo has no GitHub Actions)
• Cloudflare: none (not deployed to Cloudflare)

Setup

1. Make the scripts executable:

   chmod +x *.sh

2. Configure the cron schedule and prepare the directory structure:

   ./setup_cron.sh

3. Run an initial download:

   ./download_cert_files.sh

4. Generate a summary report:

   ./analyze_cert_data.sh

Directory structure

cert-warning-list-scraper/
├── data/                        # Downloaded .log files
├── reports/                     # Generated reports
├── download_cert_files.sh       # Downloads and validates files
├── analyze_cert_data.sh         # Generates summary report
├── setup_cron.sh                # Configures cron jobs
├── metadata.json                # Per-file download metadata
└── README.md

Metadata format

metadata.json stores one entry per downloaded file:

{
  "actions_YYYY.log": {
    "url": "https://hole.cert.pl/domains/v2/actions_YYYY.log",
    "size": 1234567,
    "checksum": "sha256...",
    "download_time_seconds": 12,
    "last_success": "YYYY-MM-DD HH:MM:SS",
    "last_check": "YYYY-MM-DD HH:MM:SS",
    "download_count": 4
  }
}

last_success is updated only when a file is actually saved (new or changed checksum). last_check is updated on every run. A divergence between the two means the most recent attempt failed.

Cron schedule

The setup script installs two jobs by default:

Time	Script
03:00	download_cert_files.sh
03:30	analyze_cert_data.sh

To adjust the schedule:

crontab -e

Logs are appended to cron_download.log in the project directory.

Sample output

Sample output files are available at: https://github.com/CyberKatalog/cert-warning-list-scraper/tree/sample_output

License

Released under the MIT License. See LICENSE for details.

Author

Created for Cyber Katalog by Silesian Solutions.