Aggregated search for car listings from autopapa.ge and myauto.ge - search by VIN, phone number, or free text.
Live: https://cars.demee-metreveli.workers.dev
Two of the biggest car listing sites in Georgia (autopapa.ge and myauto.ge) don't share data. If a car is up for sale on both, you can't easily search across them. If someone calls you offering a car, looking up its history means hunting through both sites manually.
This project scrapes both sites twice a day, normalizes everything into one Postgres table, and serves a single search endpoint that accepts:
- VIN - full 17 characters or a prefix
- Phone - any format (
+995555555555,995555555555,555555555) - Free text - fuzzy search across make, model, description and location
- Filters only - browse mode (e.g. all 2018-2022 cars under $20k)
The whole system is four moving parts: scrapers put listings into Postgres, a photo sync copies images to object storage, a FastAPI backend answers searches, and a static frontend talks to that backend. Scheduled jobs on GitHub Actions keep it all running.
autopapa.ge myauto.ge
│ │
(Playwright) (JSON API client)
│ │
└────────┬──────────┘
▼
src/parsers/*.py 1. scrape
│
▼
normalize → Car model 2. clean up
│
▼
Postgres `cars` table 3. upsert
│ │
│ └──────► sync_photos → Cloudflare R2 4. photos
▼
FastAPI /search 5. query
│
▼
web/index.html 6. render
Each source has its own parser in src/parsers/, because the two sites need
completely different approaches:
-
myauto.pytalks toapi2.myauto.ge/ka/productsdirectly and reads JSON. The public HTML site obfuscates prices and phone numbers with a custom font, so parsing the rendered page would give you garbage digits - the JSON API returns clean values. Product IDs are cached inexports/myauto-ids.jsonso repeat runs can skip work. -
autopapa.pyhas no usable API, so it drives a headless Chromium through Playwright.src/common/anti_detection.pysets up a lightly disguised browser context (realistic user-agent, no obvious automation flags). The scraper walks the listing pages, opens each car, and clicks the "show VIN" button, since the VIN isn't in the initial HTML.
Both parsers are async and scrape several pages concurrently
(CONCURRENT_PAGES). src/common/robots.py keeps the crawl polite.
Raw scraped values are messy, so everything is funnelled through one shape -
the Car model in src/common/models.py - before it goes near the database:
src/common/normalize.py- phone numbers into a canonical form, prices into an integer plus a currency, mileage into kilometres, and free-text fields into consistent casing.src/common/vin.py- pulls a VIN out of the description when the structured field is empty, and validates it (17 characters, noI/O/Q).
This is why searching works across sources: a phone number written five different ways on two sites ends up identical in the database.
src/common/db.py upserts into a single cars table. The key detail is the
unique constraint:
CONSTRAINT cars_source_id_unique UNIQUE (source, source_id)Re-scraping a listing updates it instead of creating a duplicate, and a
trigger bumps updated_at on every write. That timestamp is what the prune
job later uses to find stale listings.
The table also has a generated column that makes free-text search cheap:
search_blob TEXT GENERATED ALWAYS AS (
lower(manufacturer || ' ' || model || ' ' || description || ' ' || ...)
) STOREDPostgres maintains it automatically on every insert and update, and a GIN trigram index sits on top of it.
Scrapers only record the source CDN URLs in image_urls. Those URLs rot and
often block hotlinking, so src/scripts/sync_photos.py downloads each photo
and re-uploads it to Cloudflare R2 under a stable key:
{source}/{source_id}/{index}.jpg
The keys land in image_keys, and the API hands the frontend absolute URLs
built from R2_PUBLIC_URL. The job only picks rows that still have no
image_keys, so it is resumable - it can be stopped and restarted freely.
With --purge-local it streams straight to R2 without keeping a local copy,
which is how it runs in CI.
There is one main endpoint, POST /search, and it works out what you meant
rather than making you choose. _smart_route() in src/api/search.py checks
the query in this order:
- VIN - 17 valid VIN characters → exact match on the indexed
vincolumn. - Phone - mostly digits → matches on digits only:
regexp_replace(phone, '\D', '', 'g') LIKE '%suffix'. A leading wildcard normally forces a full table scan, so there is a trigram GIN index on exactly that expression to keep it fast. - Free text - every word must appear in
search_blob, then results are ordered by trigramsimilarity()against the make/model/year blob, so the closest titles come first. - Filters only - no text at all → browse mode, ordered by your chosen sort.
Filters (year, price, mileage, manufacturer, model, body, fuel, gearbox, drive, location, customs) apply to free-text and browse queries. They are deliberately ignored for VIN and phone lookups, which are exact by nature. Prices are converted to USD in SQL before any range comparison, so a mix of currencies still sorts correctly.
Supporting endpoints: /search/count (drives the live result counter on the
search button), plus /makes, /facets and /stats, which are cached
hourly because they change rarely and are expensive to compute.
The site is anonymous, so there is no account to limit. Limiting purely by IP
punishes shared connections - home Wi-Fi and mobile CGNAT put many people
behind one address. So src/api/rate_limit.py uses two identities:
- Primary: an anonymous token the browser generates and keeps in
localStorage, sent asX-Client-Id. Two people on the same network no longer eat each other's quota. - Backstop: the IP address, which exists only to stop someone rotating tokens to scrape the whole database.
There's a short cooldown between searches, an hourly per-token limit, and an
hourly per-IP ceiling. All timing is measured with the database's NOW(),
never the client's clock. The real client IP comes from CF-Connecting-IP,
which is trustworthy because the origin is only reachable through Cloudflare
and Cloudflare overwrites any forged value.
Plain files, no framework and no build step - the browser loads them as they are:
web/index.html- the markup.web/app.css- every style.web/app.js- all the behaviour: search, filters, compare, the detail page.web/i18n.js- all UI strings in Georgian, English, Russian and Kazakh.web/config.js- picks the API base URL (localhost in dev, Render in prod).
Every <select> is wrapped by a small custom dropdown component that renders
checkboxes, group headers and brand logos while keeping the native element as
the source of truth. All multi-selects start with everything checked,
which means "no filter" - you narrow by unchecking, or clear all and pick a
few. None-selected and all-selected both collapse to sending nothing.
Saved cars, saved searches, recent history and the comparison tray all live
in localStorage, so there are no accounts and nothing personal on the
server.
Everything runs on GitHub Actions (.github/workflows/):
| Workflow | Schedule | What it does |
|---|---|---|
parse.yml |
twice a day | Runs both parsers, then syncs photos. Waits a random delay first so requests don't arrive on a robotic schedule. |
prune.yml |
daily | Deletes listings that are genuinely gone. |
sync_backfill.yml |
every 6 hours | Works through any photo backlog. |
backfill_blitz.yml |
manual | Same as above but in 3 parallel shards, to clear a large backlog in one go. |
The prune job is deliberately careful: a listing is only a candidate when
nothing has touched it for --days, and every candidate is then re-checked
against the source. Only listings the source confirms are gone get deleted -
an old updated_at alone never deletes anything, because the scrapers skip
listings that already look unchanged.
If you want to understand this project properly, read the files in this order. Each step builds on the one before, and the whole path is about 3,000 lines.
In a hurry? Read just the three starred files below - models.py,
search.py and index.html - and you will understand roughly 80% of it.
| # | File | Why it comes here |
|---|---|---|
| 1 | (this README, "How it works") | The mental model before any code. |
| 2 | ⭐ src/common/models.py |
The Car model - one shape every part of the system agrees on. Read this and you know what a listing is. |
| 3 | db/schema.sql |
The same thing as tables: the unique constraint that makes re-scraping idempotent, the generated search_blob, and every index (each one exists for a specific query). |
| 4 | src/common/config.py |
Which environment variables exist and what the tunables are. Short. |
| 5 | src/parsers/myauto.py |
The simpler source: a JSON API client. Read scrape_one() and follow one listing from raw JSON into a Car. |
| 6 | src/parsers/autopapa.py |
The harder source: Playwright driving a real browser, clicking "show VIN". Same output shape, completely different technique. |
| 7 | src/common/normalize.py + vin.py |
Where messy input becomes comparable data. This is why search works across two sites that agree on nothing. |
| 8 | src/common/db.py |
Persistence, and the Windows event-loop workaround that forces sync psycopg onto worker threads. |
| 9 | src/common/storage.py + src/scripts/sync_photos.py |
The photo pipeline: atomic writes, R2 upload, and why a failed upload must not record a key. |
| 10 | src/api/schemas.py |
The API contract - request and response shapes, and the validation limits. |
| 11 | src/api/main.py |
App wiring: middleware, security headers, error handling. Small. |
| 12 | ⭐ src/api/search.py |
The heart of the backend. _smart_route() decides VIN vs phone vs text; _filter_clauses() builds the SQL; the CTE pagination and the LRU cache are both there for one reason - a slow free-tier disk. |
| 13 | src/api/rate_limit.py |
How an anonymous site throttles fairly: browser token first, IP only as a backstop. |
| 14 | src/api/makes.py + facets.py + stats.py |
The supporting endpoints and their hourly caches. makes.py has the fiddliest logic: collapsing trim noise into real model names. |
| 15 | ⭐ web/app.js |
All the frontend behaviour. Read enhanceSelect (the custom dropdown), buildSearchPayload (how filters become a request), and renderCompareModal. web/index.html is just the markup it drives. |
| 16 | web/i18n.js, web/config.js |
Translations for four languages, and the dev/prod API switch. |
| 17 | .github/workflows/ |
How it runs itself: parse.yml twice a day, prune.yml daily, sync_backfill.yml for the photo backlog. |
| 18 | Dockerfile, render.yaml, wrangler.toml, worker.js |
Deployment, plus the Worker cron that stops the free-tier backend sleeping. |
| 19 | tests/ |
247 tests. test_search.py and test_normalize.py double as executable documentation of the trickiest logic. |
A few decisions worth understanding, and where to find them:
- Why two totally different scrapers? myauto exposes clean JSON. autopapa
needs a real browser to reveal the VIN. Same
Carout of both - see step 5 vs 6 - Why is search one endpoint instead of three? So a user can paste anything
into one box.
_smart_route()insearch.py - Why a token instead of an IP for rate limiting? Shared home wifi and mobile
CGNAT put many people on one address.
rate_limit.py - Why a CTE for pagination?
descriptionand photo arrays live in TOAST storage. reading them for only the 25 rows on the page is dramatically faster_paginate()insearch.py - Why no framework on the frontend? Plain HTML, CSS and JS with no build step, so it loads fast and hosting costs nothing
src/
├── common/
│ ├── config.py Environment loading
│ ├── models.py Car model
│ ├── vin.py VIN extraction + validation
│ ├── normalize.py Phone, price, mileage cleanup
│ ├── anti_detection.py Playwright + light stealth
│ ├── robots.py Crawl politeness
│ ├── db.py Postgres helpers
│ ├── storage.py R2 + local photo storage
│ └── runtime.py Windows asyncio glue
├── parsers/
│ ├── autopapa.py Playwright scraper for autopapa.ge
│ └── myauto.py api2.myauto.ge JSON client
├── api/
│ ├── main.py FastAPI app
│ ├── schemas.py Request/response models
│ ├── search.py Search routing, filters, pagination
│ ├── makes.py Manufacturer → models
│ ├── facets.py Filter values
│ ├── stats.py Totals
│ ├── rate_limit.py Token + IP throttling
│ └── db_pool.py Connection pool
└── scripts/
├── init_db.py Apply schema.sql
├── migrate_csv.py Import legacy CSV dumps
└── sync_photos.py Download source photos → R2
db/schema.sql Tables, indexes, triggers
scripts/prune_dead.py Verify and remove dead listings
web/
├── index.html Markup
├── app.css Styles
├── app.js Search, filters, compare, detail page
├── config.js API base URL
└── i18n.js Translations (ka/en/ru/kk)
You need Python 3.12+, uv, and a running
Postgres instance (or use Docker: docker compose up -d postgres).
# Setup
uv sync
uv run playwright install chromium
# Config - copy and fill in real values
cp .env.example .env
# Initialize the database schema
uv run python -m src.scripts.init_db
# Scrape once (either or both)
uv run python -m src.parsers.autopapa
uv run python -m src.parsers.myauto
# Copy photos to R2 (needs the R2_* variables)
uv run python -m src.scripts.sync_photos
# Run the API
uv run uvicorn src.api.main:app --port 8765 --reload
# Serve the frontend
cd web && python -m http.server 5500
# → http://localhost:5500Run the tests with uv run pytest -q.
This repo runs on free tiers:
- Frontend: Cloudflare Workers static assets (
wrangler.toml) - Backend: Render web service (
render.yaml,Dockerfile) - Scheduled jobs: GitHub Actions (
.github/workflows/) - Database: Supabase Postgres
- Photo storage: Cloudflare R2
See DEPLOY.md for step-by-step instructions.
See SECURITY.md. In short: the search API is intentionally public and
anonymous, throttled per browser token with an IP ceiling behind it. The
backend connects with an owner role that bypasses RLS, and Supabase's
anonymous REST access is disabled.
Issues and PRs welcome.
Contact: @deme.brn
MIT.