Natural-language semantic image search — type "a dog running on grass" and instantly get the matching photos from a 3,000-image Flickr8k subset, ranked by how well they visually match your words.
No captions, no tags, no text metadata are used for retrieval — the system understands the images themselves. Images are embedded locally with the CLIP image encoder, the query with the CLIP text encoder, and retrieval is cosine similarity in their shared vector space: one numpy matmul over a cached index, so search is instant and works fully offline.
▶ Watch the full walkthrough on YouTube — semantic search, the 2D embedding Atlas, and the agent pipeline in action.
- 🔍 Semantic search (Поиск) — describe a scene in natural language, get ranked photos with a match %, metadata, and an explanation of why each image matched.
- 🗺️ Semantic Atlas (Атлас) — the entire 512-D CLIP index projected to 2D with PCA, clustered with KMeans, and each cluster auto-labeled with CLIP itself (zero-shot against candidate label prompts). Hover and click to explore how the model organizes the dataset.
- 🤖 Agent-style pipeline — the query is parsed into visual intent + metadata filters (location / date), filters are applied before ranking, results are explained, and a Markdown/JSON report is saved.
- ⚡ Offline-first — CLIP runs locally (GPU if available); the whole retrieval path makes zero API calls. An
OPENAI_API_KEYis optional and only adds generated captions/tags and richer explanations. - ✅ Honest retrieval — Flickr8k's
captions.txtis never used for indexing, retrieval, or ranking. The point of the project is that the system understands images directly.
offline (once) per query (instant)
┌────────────────┐ ┌──────────────────────┐ ┌──────────────────────────────────────┐
│ data/images/ │──►│ CLIP image encoder │ │ "people near water in Astana" │
│ 3,000 photos │ │ (ViT-B/32, batched, │ │ │ │
│ + metadata.csv │ │ L2-normalized) │ │ ▼ │
└────────────────┘ └──────────┬───────────┘ │ parse → visual query + filters │
▼ │ │ │
cache/image_embeddings.npy │ ▼ │
cache/embeddings_meta.json │ metadata filter (before ranking) │
│ │ │ │
▼ │ ▼ │
┌──────────────────────┐ │ CLIP text encoder → cosine top-k │
│ local vector index │◄──┤ │ │
└──────────────────────┘ │ ▼ │
│ explanations + Markdown/JSON report │
└──────────────────────────────────────┘
- Model:
open_clipViT-B-32 (laion2b_s34b_b79k) — embeds 3,000 images in ~1–2 minutes on a 4 GB GPU. - Index: pure CLIP image embeddings. Captions/metadata never enter the index vectors.
- Vision enrichment (optional): retrieved images can be captioned/tagged lazily by a vision model — used only for explanations and reports, never for retrieval.
git clone https://github.com/rassulz/image_extractor_agent.git
cd image_extractor_agent
conda activate practical-ai-engineering # or your own env: conda create -n image-extractor python=3.11
pip install -r requirements.txt # torch, open-clip, fastapi, …
# 1. Dataset — either unzip the bundled subset:
unzip images.zip -d data/ # -> data/images/ (3,000 images, ~65 MB)
# …or rebuild it deterministically from Kaggle:
# python scripts/prepare_flickr8k_subset.py --count 3000 --remove-raw
# 2. Frontend deps
cd frontend && npm install && cd ..
# 3. One command: build the CLIP index (once) + start backend + frontend
./run_demo.sh # CONDA_ENV=<name> ./run_demo.sh to override the env
# → open http://localhost:5173OPENAI_API_KEY in .env is optional (see .env.example): it only enables generated captions/tags and LLM explanations. The backend does pure-CLIP search by default, so a demo costs zero API tokens.
python scripts/build_index.py # build/refresh the CLIP index
uvicorn backend.main:app --port 8000 # backend → http://localhost:8000
cd frontend && npm run dev # frontend → http://localhost:5173The course-required notebook demo is image_extractor_agent_day5.ipynb — the same pipeline end-to-end (dataset → embeddings → search → filters → agent → report), regenerable with python scripts/make_notebook.py.
| Endpoint | Method | What it does |
|---|---|---|
/api/search |
POST | Natural-language query → parsed intent, filters, top-k ranked images with scores and explanations |
/api/atlas?k=6 |
GET | 2D PCA projection of the whole index + KMeans clusters auto-labeled with CLIP |
/api/samples?n=8 |
GET | Random sample images for the landing gallery |
/api/health |
GET | Index status, model name, image count |
/images/… |
GET | Static dataset images |
image_extractor_agent/
├── image_extractor_agent_day5.ipynb # notebook demo (required deliverable)
├── run_demo.sh # build index + start backend + frontend
├── requirements.txt / .env.example
├── scripts/ prepare_flickr8k_subset.py · build_index.py · make_notebook.py
├── src/ dataset_loader · embedding_index · search_tools · vision_extractor · report_writer · agent
├── backend/ main.py # FastAPI
├── frontend/ src/App.jsx · Atlas.jsx # React + Vite + Tailwind
├── report/ Image_Extractor_Agent_Report.pdf
├── data/ images/ · metadata.csv # local only (gitignored)
├── cache/ image_embeddings.npy · embeddings_meta.json · generated_image_descriptions.json
└── outputs/ search_results.json · search_report.md
- CLIP cosine scores are low in absolute terms (~0.2–0.32 for good cross-modal matches). They are meaningful for ranking, not thresholding — the UI maps them to a friendly match %.
- Synthetic metadata: Flickr8k has no real capture metadata, so
locationisunknown(none is invented) anddateis generated from a seeded RNG — purely to demonstrate filter-before-ranking metadata filtering. - Stale-cache protection: the index is validated against
metadata.csv(ids + order + model name) on load; a mismatched cache is rejected and rebuilt instead of silently returning wrong images. - Deterministic dataset: the 3,000-image subset is selected with sorted filenames + a fixed seed, so anyone can reproduce the exact same index.