Draft Detective

AI-powered assistant for academic peer review. Draft Detective runs a suite of targeted checks across language, citations, technical compliance, and substantive content — validating references against claims, flagging unsupported assertions, performing literature reviews, and suggesting relevant citations — helping reviewers and researchers assess rigor more efficiently.

Project funded by RAND: https://rand.org/

Goals

The main goal of Draft Detective is to assist and streamline the academic peer review process by reducing manual workload and improving the consistency, transparency, and rigor of evaluations.

Three ways to use Draft Detective

Draft Detective exposes the same review capabilities through three different surfaces. Pick whichever fits your workflow.

1. Claude Code plugin (skills)

Each review is packaged as a self-contained Claude Code skill under skills/, distributed as a plugin. The skills run the checks directly inside your Claude Code session — no running backend required — driving Claude through each review's procedure.

Install the plugin from the marketplace inside Claude Code:

/plugin marketplace add agencyenterprise/draft-detective
/plugin install draft-detective@draft-detective

Then invoke a check in plain language (e.g. "validate the references in this document with Draft Detective"), or ask "what can Draft Detective check?" to see the full menu.

2. MCP server

Connect Draft Detective to Claude, Codex, Opencode, or any MCP-compatible client and run reviews directly from your AI assistant. The backend mounts a Model Context Protocol server at /mcp with OAuth authentication.

Its tools let an agent list available analyses, create a project, upload documents, run a workflow, and export the results — the same pipeline the web app uses. Register it (pointing at your deployment's URL, e.g. http://localhost:8000/mcp/ in local dev):

# Claude Code
claude mcp add-json "draft-detective" '{"type":"http","url":"https://<your-deployment>/mcp/"}'

# Codex
codex mcp add draft-detective --url https://<your-deployment>/mcp/

# Opencode
opencode mcp add   # then follow the interactive prompts (Remote server, paste the URL)

The first time you use it, you'll be prompted to authenticate in the browser. The signed-in web app shows the exact command and URL for your deployment on its MCP Server page.

3. Web app

The full-featured way to use the tool. Upload a draft (.docx recommended, PDF also supported — you can upload just a section, such as the references), pick which analyses to run, and review the findings in-browser with the Document Explorer. From there you can:

• Export to Word as tracked comments.
• Share a project with colleagues via a read-only link.
• Fetch or upload the full text of references for the checks that need it (Claim Reference Validation).

Draft Detective also reaches into Microsoft 365 — an experimental Word add-in that surfaces the same reviews inside Word, and a Teams bot that answers questions about a document in chat. See microsoft/README.md.

To run the web app locally, see Development. The in-app About page documents every analysis type and data-handling detail.

What it checks

Analyses are grouped by category (shown here in app order). See the in-app About page (or ABOUT.md) for evaluation coverage and which checks require web search (#web_search) or full-text references (#full_text_refs).

Citation Check

• Reference Error Checker — uses web search to confirm each citation exists online and that its author, title, publisher, and year match public sources, catching typos and hallucinated references.

Substantive Review

• Claim Reference Validation — checks every citation against its referenced source (via RAG) and flags claims that are unsupported, only partially supported, or unverifiable.
• Internal Inference Validation — flags logical fallacies, unsupported conclusions, and arguments where the evidence doesn't back the claim.
• Methodological Alignment — characterizes standard methods in the field via web search, then compares your methodology and highlights gaps and risks.
• Reproducibility Check — extracts the main results and classifies each by how reproducibly it could be recreated from the document alone.
• Reviewer 2 — a simulated senior-reviewer peer review: strengths, weaknesses, next steps, and a devil's-advocate rebuttal.
• Recommendation Check — flags recommendations whose backing in the document's own findings is weak, indirect, missing, or contradictory.

Editorial & Style Review

• Abbreviation Scan — verifies each abbreviation is defined at first use and listed in an Abbreviations section.
• About This — checks the preface meets publication requirements (context, objectives, audience, funding, author bios).
• Document Contents — checks that required sections are present (About This, Acknowledgements, Methods, Results, Conclusion, References, Appendix).
• Figures & Tables Check — verifies every figure and table is titled, consistently numbered, cited in the body, and that all body-text references resolve.

Language

• Advocacy & Tone — flags trigger words, advocacy language, and subjective tone that departs from a neutral, objective voice.

Research & Writing Assistant

• Literature Review — searches the web for relevant academic sources you may have missed, both supporting and conflicting.
• Live Reports — searches for sources published after your document's date and produces an addendum of findings to update.

Some analyses are experimental and hidden by default in the web app (enable "Experimental Features" in your profile menu).

Architecture

Python backend (FastAPI + LangGraph agent workflows) with a Next.js frontend. Documents are ingested through a processing pipeline; each analysis is a LangGraph workflow that emits findings back into the document.

Development

For detailed setup instructions (backend, frontend, Docker, migrations, environment variables), see DEVELOPMENT.md.

# Backend (always use uv)
uv run dev.py                 # Start the full dev environment (API on :8000)

# Frontend (always use pnpm)
cd frontend && pnpm install
cd frontend && pnpm dev       # App on :3000

Deployment

• Railway: See docs/railway-deployment.md for production deployment on Railway.
• Kubernetes: See k8s/README.md for Kubernetes/OpenShift deployment.

Testing

Tests are organized by type:

• tests/unit/ — Fast, isolated unit tests
• tests/integration/ — Multi-component integration tests
• evals_inspectai/ — LLM-based evaluations using Inspect AI

# Run standard tests (default)
uv run pytest

# Run evaluations (see evals_inspectai/ for available eval suites)
uv run inspect eval evals_inspectai/e2e/reference_validation_v2/reference_validation_v2_e2e.py

License

See LICENSE file