Retrieval Uplift — a RAG benchmark

How much does a language model's answer quality improve when it can retrieve from a corpus, versus when it cannot? That "with-retrieval minus without-retrieval" accuracy gap is the uplift — the single number this project exists to produce.

The demo corpus is a set of plain-text Wikipedia articles about the Paris 2024 Summer Olympics. This pairing is deliberate (see Design decisions): gpt-4o-mini's knowledge ends before the Games happened, so it fails the questions from memory — and retrieval is what lets it succeed.

Result on this corpus

Condition	Accuracy
Closed-book (no retrieval)	0%
Open-book (RAG)	70%
Uplift	+70%

The model knows nothing about Paris 2024 results on its own; retrieval carries it from a total failure to 70% correct.

---

Quick start

Requires Docker and an OpenAI API key.

# 1. Configure your key
cp .env.example .env
# edit .env and set OPENAI_API_KEY

# 2. Fetch the corpus (~14 Wikipedia articles → ./data). Stdlib only, no deps.
python3 backend/scripts/fetch_corpus.py

# 3. Bring up the whole stack
docker compose up --build

Then open http://localhost:3000 and:

1. Click Ingest corpus — chunks and embeds the articles into pgvector.
2. Click Run evaluation — runs both conditions over the benchmark and polls until the job finishes.
3. Read the headline numbers and the per-question table.

The API is at http://localhost:8000 (/docs for the interactive schema).

---

How it works

                          ┌─────────────┐
   ./data/*.txt  ──chunk──►  embeddings  ──► Postgres + pgvector
   (Wikipedia)             │  (OpenAI)   │    chunks(source, content, embedding)
                          └─────────────┘            ▲  HNSW, cosine
                                                     │
   question ─────────────────────────────┬──────────┘
                                          │  embed query, top-k nearest
                    ┌─────────────────────┴───────────────────────┐
                    │                                               │
             closed-book                                     open-book (RAG)
        (no context, from memory)                  (top-k chunks in the prompt)
                    │                                               │
                    └──────────────►  LLM-as-judge grades both  ◄──┘
                                    CORRECT / INCORRECT vs reference
                                              │
                                     accuracies + uplift

Ingestion (once): each .txt is split into token-bounded chunks (~220 tokens, 40 overlap), embedded with text-embedding-3-small, and stored as (source, content, embedding) rows with an HNSW cosine index.

Evaluation (async RQ job): for every benchmark question,

• closed-book — ask the generation model with no context;
• open-book — embed the question, fetch the top-4 nearest chunks, and ask the model to answer only from them.

An LLM-as-judge grades each answer against a hand-written reference, and

uplift = open_book_accuracy − closed_book_accuracy

API

Method	Path	Purpose
GET	/health	liveness + configured generation model
POST	/ingest	chunk + embed + store the corpus → {files, chunks}
GET	/benchmark	the question set
POST	/eval/run	enqueue the eval job → {job_id, status}
GET	/eval/{job_id}	job status + result (accuracies, uplift, per-row detail)

---

Using your own corpus

The harness measures uplift on any topic — swap the data in three steps:

1. Provide the documents. Drop plain-text .txt files into ./data (one file per document; the filename becomes the top_source label). Either edit the ARTICLES list in backend/scripts/fetch_corpus.py to pull different Wikipedia articles, or just place your own .txt files there.

2. Write the questions. Replace BENCHMARK in backend/app/benchmark.py with your own question / reference pairs. Each answer should live in a single passage of your corpus (see design rule 4).

3. Re-run. POST /ingest then POST /eval/run from the dashboard.

For the uplift to be meaningful, keep the two invariants below intact: the corpus must contain facts the generation model doesn't already know (otherwise closed-book accuracy is high and the gap vanishes), and the embedding model must be the same at ingest and query time. If your documents are a different embedding dimension, update EMBEDDING_MODEL / embedding_dim in .env.example and config.py together.

---

Tech stack

• Backend: Python 3.11, FastAPI, Uvicorn
• Vector store: Postgres 16 + pgvector (HNSW index, cosine distance)
• Queue: Redis + RQ (one worker, automatic retry on the eval job)
• Models: OpenAI text-embedding-3-small (1536-dim) + gpt-4o-mini (generation & judge)
• Frontend: Vite + React + TypeScript (single-page dashboard)
• Orchestration: Docker Compose — postgres, redis, backend, worker, frontend

---

Design decisions

Four rules make the uplift real rather than an artifact. They are load-bearing — break one and the demo quietly stops meaning anything.

1. Prose, not tables. RAG works on narrative text where facts live inside sentences. We fetch the plain-text extract of each article (prop=extracts&explaintext=1) — not medal tables or bracket diagrams, which are structured data and the wrong tool for vector search.

2. The corpus is newer than the model's knowledge cutoff. gpt-4o-mini's knowledge ends ~October 2023; Paris 2024 happened after. The model therefore knows nothing about the results from memory, which is what makes the uplift large and clean. Do not "upgrade" the generation model to a newer one — it would already know 2024 and the uplift would collapse toward zero.

3. Same embedding model on both sides. The model that embeds documents at ingest must be the one that embeds the query at search time, or the vectors live in different spaces and retrieval is meaningless. (The generation model never sees a vector — only retrieved text.)

4. Single-passage questions. Each question's answer is findable in one retrievable chunk. Questions that require aggregating across many articles are a known weak spot of naive top-k retrieval (see below).

Sanity check

Run the closed-book condition first and confirm the model fails. That failing baseline (here, 0%) is the proof that the corpus is genuinely outside the model's knowledge — it is what makes the uplift real. If the model already answers correctly closed-book, the corpus isn't past its cutoff, or the model string is wrong.

---

Limitations & future work

• Naive top-k retrieval. A couple of benchmark answers exist in the corpus but rank outside the top-4 chunks, so open-book still says "I don't know." This is the honest behavior of single-vector top-k — a reranker or hybrid (keyword + vector) search would recover them. The dashboard's match score (cosine similarity of the top chunk) surfaces this: a miss often still retrieves the right source article at a respectable score, yet the specific fact lives in a different passage than the one top-k pulled — precisely the gap a reranker closes.
• Single-passage only. Questions needing aggregation/counting across articles are out of scope by design (rule 4).
• Hand-written benchmark. ~10 verified Q&A pairs to start; these could be auto-generated from the corpus and filtered to those with a supporting chunk.
• Not built (out of scope): auth, multi-tenancy, reranking/hybrid search, streaming, corpus upload, horizontal scaling.

---

Attribution

Corpus text is from English Wikipedia, licensed CC-BY-SA. It is fetched at build time via the Wikipedia Action API (no scraping) and is not committed to the repo.