Recon Pipeline

A program-aware, highly concurrent reconnaissance pipeline for authorized security testing and bug bounty programs.

Overview

This project automates the reconnaissance lifecycle for multiple bug bounty programs, prioritizing performance and isolation. It uses an asynchronous, multi-threaded architecture with two primary pipelines: a Web Pipeline and an ASN Pipeline. All collected data is persisted into an isolated SQLite database, eliminating duplicate scans and enabling historical tracking.

The main scanner (recon.py) discovers assets, fingerprints them, port scans, runs vulnerability checks, and performs deep regex scanning for secrets. The companion exporter (export.py) generates rich, interactive HTML reports and JSON exports.

Features

• Multi-Pipeline Architecture: Independent threaded pipelines for Web and ASN tasks.
• Concurrent Execution: Parallel execution of Nuclei and Regex Scanning.
• Multi-Program Support: Maintains isolated state for each program.
• SQLite State Management: Non-blocking database writes via a dedicated DB thread.
• Web Recon: Subdomain discovery (Subfinder), active probing (HTTPX), and web crawling (Katana).
• ASN & Infrastructure: ASN enumeration (ASNMap), CIDR parsing (MapCIDR), and fast port scanning (Naabu).
• Vulnerability & Secret Scanning: Nuclei for templates, custom Regex Scanner for JS and HTML secret monitoring.
• Historical Tracking: Schema migrations and lifecycle tracking for discovered assets.
• Reporting: Advanced HTML dashboard and JSON export via export.py.

---

Architecture

The pipeline uses a threaded orchestrator that spawns separate workflows for Web and ASN tasks, ensuring they do not block each other. Database writes are handled by a thread-safe DbWriter.

Scan Workflow

flowchart TD
    A[Scope & Targets] --> O[Orchestrator]
    
    subgraph Web Pipeline
    O --> W1[Subfinder]
    W1 --> W2[New Subdomains]
    W2 --> W3[HTTPX]
    W3 --> W4[Katana Crawling]
    W4 --> W5[Parallel Execution]
    W5 --> W6[Nuclei]
    W5 --> W7[Regex Scanner]
    end
    
    subgraph ASN Pipeline
    O --> A1[ASNMap]
    A1 --> A2[MapCIDR]
    A2 --> A3[Naabu Port Scan]
    A3 --> A4[HTTPX on Alive Ports]
    end

    W6 --> DB[(SQLite DB)]
    W7 --> DB
    A4 --> DB

    DB --> EX[export.py]
    EX --> H[Interactive HTML]
    EX --> J[JSON Export]

Loading

Installation

Requirements

• Python 3.11+
• Linux environment (or WSL/Windows)
• Go (for ProjectDiscovery tools)
• SQLite3

Install Python Dependencies

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Install ProjectDiscovery Tools

go install github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
go install github.com/projectdiscovery/httpx/cmd/httpx@latest
go install github.com/projectdiscovery/naabu/v2/cmd/naabu@latest
go install github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
go install github.com/projectdiscovery/katana/cmd/katana@latest
go install github.com/projectdiscovery/asnmap/cmd/asnmap@latest
go install github.com/projectdiscovery/mapcidr/cmd/mapcidr@latest
go install github.com/projectdiscovery/notify/cmd/notify@latest

Project Structure

root/
├── recon.py                  # Main CLI entry point
├── export.py                 # HTML/JSON report generator
├── requirements.txt          # Python dependencies
├── .env                      # Configuration file
├── targets/                  # Folder for scope lists (*.txt)
├── logs/                     # Application logs
├── work/                     # Temporary working directories for tools
├── reports/                  # Generated HTML/JSON outputs
├── recon/                    # Core pipeline package
│   ├── config.py             # Env validation and config loading
│   ├── database.py           # SQLite schemas and queries
│   ├── pipeline/             # Pipeline orchestration (web, asn, orchestrator)
│   └── tools/                # Python wrappers for external CLI tools
└── recon.db                  # Central SQLite database

Configuration

Create a .env file beside recon.py based on .env.example. Ensure all required binaries are in your PATH or configured correctly.

Important variables include:

• ROOT_DIR: Base directory for logs, work, and databases.
• SUBFINDER_BIN, HTTPX_BIN, NAABU_BIN, NUCLEI_BIN, KATANA_BIN, ASNMAP_BIN, MAPCIDR_BIN, NOTIFY_BIN
• KATANA_DEPTH: Depth for crawling (default: 3)
• REGEX_SCANNER_THREADS: Thread pool size for fast async regex scanning
• Skip Flags: SUBFINDER_SKIP, HTTPX_SKIP, KATANA_SKIP, NUCLEI_SKIP, ASN_PIPELINE_SKIP, REGEX_SCANNER_JS_SKIP, REGEX_SCANNER_HTML_SKIP

Usage

Running Reconnaissance

To scan all targets found in the targets/ folder:

python3 recon.py

Automated Monitoring (Continuous Recon)

The core focus of this tool is continuous asset monitoring. It is designed to be run on a recurring schedule to catch new subdomains, ports, vulnerabilities, and leaked secrets.

1. Linux cron Job

To run the scanner automatically every day at midnight:

# Edit the crontab
crontab -e

# Add the following line (update paths as needed)
0 0 * * * cd /path/to/bugBounty-auto-recon && /path/to/bugBounty-auto-recon/.venv/bin/python recon.py >> logs/cron.log 2>&1

2. Linux systemd Timer

For better log management and process control, a systemd service/timer is highly recommended:

1. Create /etc/systemd/system/recon.service:

   [Unit]
   Description=Bug Bounty Auto Recon Scanner
   
   [Service]
   Type=oneshot
   WorkingDirectory=/path/to/bugBounty-auto-recon
   ExecStart=/path/to/bugBounty-auto-recon/.venv/bin/python recon.py
   User=your_user

2. Create /etc/systemd/system/recon.timer:

   [Unit]
   Description=Run Recon Scanner Daily
   
   [Timer]
   OnCalendar=*-*-* 00:00:00
   Persistent=true
   
   [Install]
   WantedBy=timers.target

3. Enable and start the timer:

   sudo systemctl enable --now recon.timer

3. Windows Task Scheduler

If running on Windows:

1. Open Task Scheduler and click Create Basic Task.
2. Name it "Bug Bounty Recon" and set the trigger to Daily.
3. Action: Start a program.
4. Program/script: C:\path\to\bugBounty-auto-recon\.venv\Scripts\python.exe
5. Add arguments: recon.py
6. Start in: C:\path\to\bugBounty-auto-recon

Generating Reports

Export an interactive HTML report for a specific program:

python3 export.py --db recon.db --program "company_name"

Export HTML and JSON for all tracked programs:

python3 export.py --db recon.db --all-programs

Database Schema

Primary tables managed by recon/database.py:

• Scope: programs, scope_domains
• Web: subdomains, httpx_results, katana_results
• Vulnerabilities & Secrets: nuclei_findings, javascript_monitor_files, regex_scanner_findings, regex_scanner_history
• Infrastructure: asn_ranges, asn_ips, ports, asn_httpx_results
• System: runs, schema_version

Best Practices

• Resource Limits: Adjust timeout and threading configurations in .env based on your VPS/machine limits.
• Authorization: Use only on authorized targets and bug bounty programs.
• Templates: Run nuclei -update-templates regularly to ensure coverage against the latest CVEs.
• Data Pruning: Use MAX_JOB_RETENTION_DAYS in .env to prevent the work/ directory from exhausting disk space.

License

Use only for authorized security assessments and bug bounty programs.