Turn books and documents into an atomic, interlinked knowledge vault: one note per idea, densely cross-linked, every claim grounded in a verbatim citation, built to scale across hundreds of sources.
You buy a great technical book. You read it once. Three months later you cannot remember chapter 7 existed. Searching the PDF returns page numbers, not answers; asking an LLM about it risks hallucinated quotes and chapter locations; the notes you took sit unread.
Mycelia turns a document into a vault of small, single-idea notes (a mycelium: a network of atomic notes that reference each other and cite their source). Point it at a book and it emits an Open Knowledge Format v0.1 bundle: one note per concept, framework, principle, entity, method, or anti-pattern, densely interlinked, each grounded by a verbatim quote from the archived source text.
The payoff is convergence across books. When book A and book B both cover entropy, the vault holds one canonical note that accrues a citation from each source, not two duplicates. That check-before-create reconciliation is what turns a pile of notes into a second brain. The bundle is plain Markdown with standard links: drop it on any filesystem, and any OKF-aware agent can read it.
Mycelia was validated on a two-book scale test (Coding Theory, 355pp, and Computational Genomics with R, 463pp) ingested into one vault. Grounding was 100% (every cited quote verbatim-verifiable against the immutable
raw/), 0 dangling links, and cross-book reconciliation produced single canonical notes with citations from both sources. An orchestrated build grew the vault to 89 atomic notes, and a coverage pass raised concept recall from 67% to 92% on the tested region. Seedocs/mycelia-productization.md. The author's production vault currently holds 170+ books.
RAG works at query time: chunk, embed, return the vectors nearest your question. It is excellent for "find passages like this across 80 books."
Mycelia works at compile time: one deep pass names the author's actual concepts, captures their exact formulation, links them, and grounds each in a citation. The result is a stable, navigable graph you (or an agent) can reason over, not a bag of chunks retrieved fresh each time. For broad search across a large corpus, RAG wins; for a durable, interlinked, cited knowledge base that converges as you add books, Mycelia is the substrate. The two are complementary: Mycelia ships a local semantic index over raw text (mycelia index-raw / passage-search) so you can retrieve the source passage behind any citation, and the staging layer below is exactly that RAG half, at zero token cost.
Distilling a book with an LLM costs real money and takes hours. Extracting its text costs nothing. Mycelia splits those two halves, so you can have a book searchable today and decide later whether it deserves distillation:
ASSENT ──cold-extract.sh──▶ STAGED ──promote.sh──▶ IN_VAULT
(0 LLM tokens, (LLM build + dedupe,
minutes, local) $, hours)
└──────────── mycelize.sh (both in one job) ────────────┘
| Layer | Where | Contains | Cost | Query |
|---|---|---|---|---|
| Staging | ~/llm-wiki/data/staging-vault |
verbatim full_text.txt + local embedding index |
zero tokens | raw passages |
| Knowledge vault (KV) | ~/llm-wiki/data/knowledge-vault |
distilled atomic notes, cited, interlinked, plus the archived raw/ |
~$1 per chunk (Opus) | notes and raw passages |
scripts/kv-search.sh searches both at once and labels each hit [KV] or [STAGING], so from the querier's side there is a single source of truth. A book sitting in staging is already useful; promotion is an upgrade, not a prerequisite.
All three entrypoints are mechanical: slug, title, year, ISBN and authors are derived from the filename (Title (Year) [ISBN] — Authors.pdf), there are no interpretive prompts. They are idempotent and resumable, and every one of them takes --dry-run.
git clone https://github.com/strawberry-code/mycelia.git
cd mycelia
uv sync --all-extrasUse uv. --all-extras, not --extra all: the all extra covers the text extractors but deliberately omits embed (fastembed), without which index-raw / passage-search — and therefore the whole staging layer — do not work.
If you want a minimal install, pick the extras your formats need: pdf, epub, docx, html, rtf, mobi, docling (technical PDFs, large download), embed (local semantic index), rich (progress bars). Plain text, Markdown, reStructuredText and AsciiDoc need no extra.
Two things live outside Python:
claudeCLI onPATH. The LLM steps run headlessly (claude -p --model opus, driven bybookextract/runner.py); there is no interactive session to babysit and no slash command to install.- System extractors:
pdftotext(poppler) for text PDFs, Calibre'sebook-convertfor MOBI/AZW. See Requirements.
The console script is mycelia; book-extract remains as a backward-compat alias.
# 1. Zero-cost: make a book searchable now, decide about distillation later
scripts/cold-extract.sh "~/books/Coding Theory (2017) — Ling, Xing.pdf"
# 2. Query staging + KV as one source (local, zero tokens)
scripts/kv-search.sh "hamming distance bound" -k 8
# 3. Distill a staged book into the KV (this is the step that spends money)
scripts/promote.sh coding-theory
# …or do both in one job
scripts/mycelize.sh "~/books/Coding Theory (2017) — Ling, Xing.pdf"
# Bulk-stage a whole directory, skipping anything already staged or in the vault
scripts/cold-extract-batch.sh ~/books
# The deterministic gate that must pass before a vault is considered done
.venv/bin/mycelia lint ~/llm-wiki/data/knowledge-vaultpromote.sh (alias distill.sh) runs build-plan --append → build → dedupe --incremental --assemble → archives the original into raw/<slug>/ → scoped re-index → book ledger, then dismisses the book from staging. It is capped by INGEST_BUDGET (default $40) and it does not auto-commit the KV: commit it yourself after every promote.
The three wrappers accept pdf, epub and mobi — the formats books actually arrive in. The other supported formats (DOCX, HTML, Markdown, RTF, reStructuredText, AsciiDoc) go through the CLI primitives directly: mycelia <file> to extract, then build-plan --append → build --assemble → lint, which is exactly what the wrappers compose. See examples/README.md for a worked Markdown example.
The first ingest creates a fresh vault; every later one ingests into the existing vault, so a concept shared with an earlier book (say apoptosis) merges into the same canonical note carrying a citation from each source, rather than spawning a duplicate.
A complete, copyright-safe sample vault generated from a CC0 guide ships in examples/demo-output/mycelia-vault/: 13 atomic notes across all six note types, a per-book MOC, OKF index.md/log.md, grounded # Citations, zero dangling links.
| Variable | Default | Meaning |
|---|---|---|
KV_VAULT |
~/llm-wiki/data/knowledge-vault |
the distilled vault |
STAGING_VAULT |
~/llm-wiki/data/staging-vault |
the cold-extraction area |
INGEST_BUDGET |
40 |
dollar cap on a single build |
OCR_THRESHOLD |
0.1 |
mojibake % that triggers OCR in --ocr book/page |
SURYA_BIN |
~/.mycelia-surya-venv/bin/surya_ocr |
OCR binary |
The two vault defaults are the author's paths. Set KV_VAULT and STAGING_VAULT (export them, or prefix each command) if your layout differs.
pdftotext silently destroys formulas — glyphs come back as U+FFFD replacement characters — so a maths-heavy book extracts into confident nonsense. Surya 2 recovers them as LaTeX, locally, at zero token cost, but costs roughly 16.6s/page on an M-series Mac, so a large PDF means minutes to hours.
cold-extract.sh therefore takes --ocr:
| Mode | Behaviour |
|---|---|
full (default) |
OCR the whole book |
book |
OCR only if the book's average mojibake ratio ≥ OCR_THRESHOLD |
page |
OCR only the individual broken pages (selective, mixed provenance) |
none |
pdftotext only |
Caveat worth knowing: mojibake detection does not catch clean-font maths. A 2D formula flattened into garbage without a single U+FFFD still reads as "healthy" to the heuristic, so book/page will skip it. For maths, use full (the default) or force --redo on a book you already staged the cheap way.
--engine surya is the tested default and needs a one-time venv at ~/.mycelia-surya-venv (see scripts/surya_extract.py); --engine chandra is wired but not configured, and skips OCR with a warning if its venv is missing.
build-plan defaults to --append, which is safe. With --no-append (or an old enough version) it truncates sources.json and restarts chunk ids from 0; the next assemble then sees a single book and deletes every other book's .md files. This wiped 33,677 files on 2026-07-17.
The three entrypoints above always pass --append and do not expose --no-append. Use them; do not call build-plan or assemble by hand against a populated vault. Commit the KV after every promote — that incident was made worse by books that had never been committed.
A self-contained OKF v0.1 bundle, a directory of Markdown files you can copy anywhere:
cell-biology/
├── concepts/ # canonical single-idea notes (one file = one concept)
├── frameworks/ # named models with their application
├── principles/ # actionable rules
├── entities/ # people, organisms, organizations, named things
├── methods/ # procedures and techniques
├── anti-patterns/ # what to avoid and why
├── references/ # one note per ingested source book (type: Source)
├── moc/ # per-book maps of content (navigation hubs)
├── raw/<slug>/ # immutable archived extraction (full_text.txt + metadata.json)
├── index.md # root listing; the only file carrying okf_version
├── log.md # dated change history, newest-first (ISO-8601)
├── SCHEMA.md # editorial schema that keeps multi-book vaults from drifting
└── .mycelia.json # provenance manifest (generator version, sources, note counts)
Each note is one idea: 2 to 5 sentences in the practitioner voice, YAML frontmatter with type (required by OKF) plus recommended fields (title, description, tags, timestamp) and Mycelia governance extensions (aliases, confidence, status, contested) that a pure-OKF consumer ignores without breaking. Links are bundle-relative absolute (/concepts/apoptosis.md) so they survive moves, and every link has a reciprocal entry in the target note's ## Related. Sources live in a numbered # Citations body section, so a concept seen in multiple books accrues multiple citations on the same note.
Every atomic note carries a verifiable source: a chapter ref (always), a printed page folio when derivable (text-mode PDFs keep page breaks; Docling and joined formats give chapter-level refs), and a short verbatim quote:
[Ch 12, p.340] "apoptosis is a programmed sequence of events leading to cell death"
Every quote is grep-verified against the immutable raw/ extraction (matching folds ligatures and whitespace, so a quote can span line breaks). A quote that does not match is a fabrication and is dropped, never guessed. Page folios are computed from where each quote sits in raw/, not read by eye.
mycelia lint is the verifiable success criterion, a stdlib-only, deterministic check (no model, no trust required). It enforces:
- every non-reserved
.mdhas a non-emptytype; - every
](/…md)link resolves (zero dangling links); - citation coverage (every note carries at least a chapter ref);
- reciprocal backlinks are symmetric;
index.mdfiles carry no frontmatter (except the rootokf_version), andlog.mddates are ISO-8601, newest-first.
Fix every reported error before the vault is done. On the scale test the orchestrated headless build cost roughly $1 per chunk with Claude Opus; Haiku was ~6.7x cheaper but dropped ~40% of quotes on grounding, so Opus is the default for a citation-grounded vault.
The scripts are the entrypoints; these are the primitives they compose. MYCELIA.md is the recipe that defines the Note-JSON contract and the bundle spec the LLM steps follow.
| Subcommand | What it does |
|---|---|
(no subcommand) mycelia <file> |
extract text to a workdir (full_text.txt + metadata.json) |
build-plan |
chunk the source, archive raw/, register it in sources.json (always --append) |
build |
headless orchestration: claude -p per chunk, emits validated Note JSON |
assemble |
write the files, insert reciprocal links, ground citations, run lint |
lint |
the deterministic OKF gate (no model) |
dedupe |
conservative same-concept merger (--incremental for a scoped pass) |
cover |
loop-until-dry critic: re-read each chunk, add missed concepts |
verify |
LLM judge: is each note's body faithful to its cited quote? |
index-raw / passage-search / passage |
local semantic index over raw text, and passage retrieval |
retro-relations |
type the relations of an already-built vault, in resumable batches |
eval |
evaluation harness |
Vault maintenance lives in scripts/: book_ledger.py (per-book status table), organize-books.py (the raw/ archiving rite), reindex-semantic.sh (scoped, O(book) re-index — never run basic-memory reindex --embeddings by hand, it is O(vault)), zip-vault.sh / unzip-vault.sh.
scripts/ingest-book.sh is the older direct-to-KV ingest with no staging step. It is superseded by mycelize.sh and kept only for compatibility.
The extractor tries tools in order per format and uses the first available; if none is installed it prints the exact install command. Plain text, Markdown, reStructuredText and AsciiDoc need no extra dependency.
| Source | Preferred | Fallback | Notes |
|---|---|---|---|
| PDF, text-heavy | pdftotext (poppler) |
pypdf, pdfminer.six |
pdftotext/pdfminer keep page folios |
| PDF, technical | docling |
(none) | tables, code, figures; ~1.5s/page; chapter-level citations |
| PDF, maths | Surya 2 OCR | (none) | recovers formulas as LaTeX; ~16.6s/page; separate venv |
| EPUB | ebooklib + beautifulsoup4 |
stdlib zipfile |
|
| DOCX | python-docx |
stdlib ZIP/XML | |
| HTML | beautifulsoup4 |
stdlib | |
| RTF | striprtf |
regex | |
| MOBI/AZW/AZW3 | Calibre ebook-convert |
mobi package |
Python 3.10+. --extra rich adds a live progress bar and Docling spinner on a TTY; --extra embed (fastembed, included in --all-extras) builds the local semantic index. The embedding model is bge-small-en-v1.5, shared with basic-memory's cache; if HuggingFace is unreachable and the model is already cached, set HF_HUB_OFFLINE=1.
This repository is a fork of virgiliojr94/book-to-skill. It began as a tool that turned one book into a single Claude Code skill; it has since grown into Mycelia, a generator for atomic, interlinked OKF knowledge vaults that scale across a whole library. The Strawberry Code edition prioritizes reproducible local execution, a uv-first developer workflow, portable OKF-native output, and a deterministic lint gate as the safety net.
You are responsible for the material you process. Mycelia is a tool; it grants no rights to the books or documents you feed it.
- Only process material you own, have permission to use, or that is public domain / openly licensed.
- Generated vaults contain derived material (named concepts, definitions, and short verbatim quotes used as citations).
- Do not commit copyrighted source files, extracted text (
raw/), or vaults derived from protected books to public repositories unless you have the legal right to do so. The.gitignoreexcludes common book formats and extraction archives to help avoid accidental commits. - Using this tool does not remove your responsibility to comply with copyright and license terms. The only vault shipped in this repo is generated from a CC0 guide.
uv sync --all-extras --extra dev
uv run pytest -q # test suite; fixtures built in-process
uv run mycelia lint examples/demo-output/mycelia-vault # lint the shipped sample vault
uv run ruff check scripts/ tests/ # lint [required]
uv run mypy # --strict types [required]The bookextract package holds a strict semantic-LOC / typing / complexity standard; pytest, ruff, mypy, lizard and xenon all run as pre-commit hooks (.pre-commit-config.yaml). Thresholds live in pyproject.toml; CI runs the gate on Python 3.10 to 3.12 (.github/workflows/ci.yml). The staging layer's normative spec is docs/staging-layer-master-plan.md; the full recipe and bundle spec is MYCELIA.md.
MIT