A Docker-based control panel for Scrapling — run scrape jobs locally or on a VPS with one toggle.
Built with FastAPI + React. No build step. No dependencies beyond Docker.
This dashboard wraps Scrapling's powerful scraping engine in a web UI you can run on your machine (or a VPS) with docker compose up. You get:
- Task cards with live progress — pages scraped, items found, elapsed time, and ETA countdown
- LOCAL ↔ VPS toggle — switch between your machine and a remote server in one click
- Inline results viewer — JSON, CSV, and table views expand below each task
- One-click CSV export — download results without leaving the Tasks tab
- Live log streaming — WebSocket-powered real-time output from Scrapling
- VPS diagnostics — API health, Docker status, memory, and active jobs at a glance
- All four Scrapling fetchers — StealthyFetcher (anti-bot), DynamicFetcher (JS), Fetcher (HTTP), AsyncFetcher (concurrent)
- Proxy rotation support — configure rotating residential proxies per-job
- Adaptive scraping — Scrapling's smart element tracking survives website redesigns
┌─────────────────────────────────────────────────────┐
│ Browser (localhost:3000) │
│ ┌───────────┐ ┌────────────────────────────────┐ │
│ │ Sidebar │ │ │ │
│ │ ─────────│ │ Task cards with progress bars │ │
│ │ [LOCAL|VPS]│ │ ▸ View → inline JSON/CSV/Table│ │
│ │ Default: │ │ ⬇ CSV one-click download │ │
│ │ VPS IP │ │ Live log streaming │ │
│ │ or Custom │ │ │ │
│ │ ─────────│ │ │ │
│ │ Job config │ │ │ │
│ │ [Start] │ │ │ │
│ └───────────┘ └────────────────────────────────┘ │
└──────────┬──────────────────────────────────────────┘
│ REST + WebSocket
▼
┌──────────────────────┐ ┌──────────────────────┐
│ LOCAL Docker │ OR │ VPS Docker │
│ (localhost:8000) │ │ (your-vps-ip:8000) │
│ FastAPI + Scrapling │ │ FastAPI + Scrapling │
│ StealthyFetcher │ │ StealthyFetcher │
│ ProxyRotator │ │ ProxyRotator │
└──────────────────────┘ └──────────────────────┘
git clone https://github.com/YOUR_USERNAME/scrapling-dashboard.git
cd scrapling-dashboard
cp .env.example .env
docker compose up --buildOpen http://localhost:3000 — that's it.
First build takes 5–10 minutes (pulls the Scrapling Docker image + installs Chromium). After that, starts in seconds.
For detailed step-by-step instructions including Docker installation, VPS setup, firewall config, and troubleshooting, see INSTALL.md.
The dashboard can send jobs to a remote server for long-running overnight crawls.
The API token is a password you create to protect your VPS endpoint. Pick any strong string:
# On your VPS, in /opt/scrapling-dashboard/.env
API_TOKEN=sk-scrapling-8f3a2b7c9d1e4f5aFor local-only use, no token is needed — auth is automatically skipped when the token is the default value.
chmod +x deploy-vps.sh
./deploy-vps.sh root@your-vps-ip- Toggle LOCAL → VPS in the header
- Your default VPS IP is pre-configured (or click Custom IP to enter another)
- Paste your token in the API bearer token field
- Submit jobs — they run on the VPS even if you close your laptop
| Method | Endpoint | Description |
|---|---|---|
GET |
/api/health |
Health check |
POST |
/api/jobs |
Submit a scrape job |
POST |
/api/spiders |
Submit a multi-page crawl |
GET |
/api/jobs |
List all jobs |
GET |
/api/jobs/{id} |
Job status + progress |
GET |
/api/jobs/{id}/results?format=json |
Results (json or csv) |
DELETE |
/api/jobs/{id} |
Delete a job |
WS |
/ws/jobs/{id}/logs |
Live log stream |
curl -X POST http://localhost:8000/api/jobs \
-H "Content-Type: application/json" \
-d '{
"url": "https://quotes.toscrape.com",
"selectors": ".quote",
"fetcher": "stealthy",
"headless": true
}'scrapling-dashboard/
├── docker-compose.yml # Orchestrates backend + frontend
├── .env.example # Environment template (copy to .env)
├── deploy-vps.sh # One-command VPS deploy script
├── INSTALL.md # Detailed installation guide
├── backend/
│ ├── Dockerfile # Based on pyd4vinci/scrapling
│ ├── app.py # FastAPI wrapping Scrapling
│ └── requirements.txt
├── frontend/
│ └── index.html # React SPA (no build step needed)
└── nginx/
└── default.conf # Reverse proxy + WebSocket support
| Fetcher | Use Case | Anti-bot | Speed |
|---|---|---|---|
| StealthyFetcher | Cloudflare, Turnstile, protected sites | ★★★★★ | ★★★ |
| DynamicFetcher | JS-heavy SPAs, client-rendered pages | ★★★ | ★★★ |
| Fetcher | Simple HTTP, static pages | ★★ | ★★★★★ |
| AsyncFetcher | High-volume concurrent scraping | ★★ | ★★★★★ |
| Scenario | Recommendation |
|---|---|
| Development / testing selectors | LOCAL |
| Quick scrapes (< 100 pages) | LOCAL |
| Sites with aggressive anti-bot | LOCAL + residential proxy |
| Overnight crawls (1000+ pages) | VPS |
| Scheduled recurring jobs | VPS |
| Scraping from a specific region | VPS in target region + proxy |
- Persistence — Replace the in-memory job store in
app.pywith SQLite or Redis - Scheduling — Add APScheduler or cron for recurring spiders
- Notifications — POST to Slack/Discord webhooks on job completion
- Auth UI — Add a login page if exposing the VPS publicly
- HTTPS — Use Caddy as a reverse proxy for automatic TLS (see INSTALL.md)
This project is a UI wrapper around Scrapling by Karim Shoair (D4Vinci). Scrapling is an adaptive web scraping framework that handles everything from single requests to full-scale crawls, with built-in anti-bot bypass, smart element tracking, and proxy rotation.
- Scrapling GitHub: github.com/D4Vinci/Scrapling
- Scrapling Docs: scrapling.readthedocs.io
- Scrapling License: BSD-3-Clause
- Scrapling Discord: discord.gg/EMgGbDceNQ
This dashboard was designed and built with the assistance of Claude by Anthropic, iteratively scaffolding the FastAPI backend, React frontend, Docker configuration, and VPS deployment tooling.
- FastAPI — REST + WebSocket API
- Scrapling — Web scraping engine
- Nginx — Reverse proxy
- Docker — Containerization
This dashboard is MIT licensed. Scrapling itself is BSD-3-Clause.
Contributions welcome! Please open an issue first to discuss what you'd like to change.
- Fork the repo
- Create a feature branch (
git checkout -b feature/my-feature) - Commit changes (
git commit -am 'Add my feature') - Push (
git push origin feature/my-feature) - Open a Pull Request