doc-lok

Pre-prompt cache for link-bearing Markdown. Records SHA-256/ETag metadata for remote links so unchanged content can be skipped on repeat runs — saving network, latency, and a small number of LLM tokens per link.

What it does

doc-lok scans Markdown files for external http(s) links, validates them against the remote server, and records their content hash (SHA-256) and ETag in a local .doc-lok/lock.json lockfile. On subsequent runs, it uses those to detect change cheaply (HEAD only) and skip work it already did.

Modes

Mode	Flag	What it does
Condense	(default)	Replaces unchanged inline `[text](url)` links with a compact HTML comment marker (`<!-- doc-lok:cached#<sha> -->`). Saves ~5 tokens per cached link. Changed/new links are left intact.
Inline	`--inline`	Fetches linked page bodies, converts HTML→Markdown, caches results in `.doc-lok/cache/`. Injects content under each link. Repeat runs are HEAD-only — no body re-fetch. Default injects only a ~200-token table of contents; use `--section` to control what gets inlined.
Section	`--inline --section <name>`	Modifier on `--inline`: inject only specific sections instead of the full body. Saves LLM tokens vs. an agent fetching the full page — e.g. a 50K-token doc becomes a 2K-token section. `--section all` for full body (no token savings, but network/latency savings on repeat runs).
Check	`--check`	Validates all URLs and updates the lockfile, but does not modify the Markdown file. Non-destructive freshness probe.
Restore	`--restore`	Reverses condense/inline: replaces markers and inline blocks back with original `[text](url)` links, preserving anchor text.

What's NOT shipped yet

These are on the roadmap. The READMEs in some older releases may reference them as if they exist — they don't.

--summary — LLM-generated summaries of linked pages. This is where real LLM token savings would happen (a 50K-token doc becomes a ~300-token summary). Not implemented.
--diff — Line diffs between cached and fresh bodies. Not implemented.
--concurrency — Parallel URL validation. Currently sequential.
--ttl — Skip network entirely if recently checked. Not implemented.
--prune — Remove stale lockfile entries. Not implemented.

Install

npm install -g doc-lok   # CLI
npm install doc-lok      # library

From source:

git clone https://github.com/<your-org>/doc-lok.git
cd doc-lok
npm install
npm run build

Requires Node.js ≥ 18. Zero runtime dependencies — uses only Node.js built-in modules (node:crypto, node:https, node:fs, node:path). The optional turndown peer dependency is only needed for --converter turndown.

Usage

The basic workflow

doc-lok operates on a single Markdown file. You run it, it writes processed Markdown to stdout and diagnostics to stderr, and it creates/updates a .doc-lok/lock.json lockfile in the same directory.

Run 1 — cache warm-up. Every URL is fetched. No links are condensed yet — the lockfile is populated so the next run can detect unchanged content. No savings on this run.

$ doc-lok README.md --quiet > condensed.md
# (no savings — lockfile is being populated)

Run 2+ — savings kick in. URLs whose content hasn't changed are replaced with tiny markers. The prompt shrinks by ~5 tokens per unchanged link.

$ doc-lok README.md --quiet > condensed.md
# unchanged links → <!-- doc-lok:cached#<sha> --> markers

Without --quiet, you get a diagnostic report on stderr:

─ doc-lok ──────────────────────────────
  ✓ https://example.com/docs  [cached]  saved ~342 tok
  ↻ https://example.com/changelog  [updated]  saved ~0 tok
  ✗ https://example.com/broken  [error]  saved ~0 tok  (HTTP 503)
  Total est. tokens saved this run: 342
  Lockfile: /home/user/project/.doc-lok/lock.json
─────────────────────────────────────────

✓ cached — content unchanged, link replaced with marker
↻ updated — content changed (or first run), link left intact
✗ error — request failed (DNS, timeout, HTTP error), link left intact

Choosing a mode

Condense (default) — You have a Markdown file with links and want to shrink it before feeding it to an LLM. Unchanged links become markers; changed/new links stay visible. Best for repeat runs where most links haven't changed.

doc-lok README.md --quiet > condensed.md

Inline (--inline) — You want the LLM to see the content behind the links, not just the link text. doc-lok fetches each linked page, converts HTML to Markdown, and injects it under the link. By default only a table of contents is injected (~200 tokens). Use --section to control what gets inlined.

# TOC only (default) — tiny prompt, just section headings
doc-lok README.md --inline --allow-private

# Specific sections — only the parts you need
doc-lok README.md --inline --allow-private --section authentication

# Full body — everything
doc-lok README.md --inline --allow-private --section all

The default (TOC only) and --section <name> save LLM tokens compared to an agent fetching the full page — instead of reading a 50K-token doc, the agent gets a ~200-token TOC or a 2K-token section. --section all injects the full body, so no LLM token savings vs. fetching it yourself — but you still save network/latency on repeat runs (HEAD-only, no body re-fetch) and get byte-identical prompts that unlock provider-side prompt caching.

Check (--check) — You want to know if links are stale without modifying the file. Validates all URLs, updates the lockfile, but leaves the Markdown untouched. Useful before deciding whether to condense.

doc-lok README.md --check

Restore (--restore) — You have a condensed/inlined file and want the original links back. Reverses both cached markers and inline blocks, preserving the original anchor text. No network requests — uses the lockfile only.

doc-lok condensed.md --restore > original.md

Piping to other tools

stdout gets the processed Markdown; stderr gets diagnostics. This makes doc-lok safe to pipe:

# Python
doc-lok docs/spec.md --quiet | python3 prompt.py

# Go
doc-lok docs/spec.md --quiet | ./my-go-binary

# Capture diagnostics separately
doc-lok README.md > condensed.md 2> diagnostics.log

JSON output for automation

--json outputs a structured JSON object to stdout instead of raw Markdown. Works with all modes:

doc-lok README.md --json           # condense
doc-lok README.md --check --json   # check
doc-lok README.md --restore --json # restore
doc-lok README.md --inline --json  # inline

Schema (condense mode):

{
  "mode": "condense",
  "output": "... condensed markdown ...",
  "diagnostics": [
    { "url": "https://example.com/docs", "status": "cached", "tokensSaved": 342 },
    { "url": "https://example.com/broken", "status": "error", "tokensSaved": 0, "message": "HTTP 503" }
  ],
  "tokensSaved": 342,
  "lockfilePath": "/home/user/project/.doc-lok/lock.json",
  "lockfile": { "version": 3, "global_tokens_saved": 18432, "urls": { ... } }
}

On fatal errors with --json: { "error": "message" } with exit code 1. Per-URL errors are not fatal — they appear inside diagnostics with status: "error".

Using as a library

import {
  condenseMarkdown, inlineMarkdown, checkMarkdown, restoreMarkdown,
} from "doc-lok";

// Condense — replace unchanged links with markers
const result = await condenseMarkdown("./README.md");
console.log(result.output);         // condensed Markdown
console.log(result.tokensSaved);    // tokens saved this run
console.log(result.lockfilePath);   // .doc-lok/lock.json path
console.log(result.lockfile);       // full lockfile state

// Inline — fetch linked pages, inject content
const inline = await inlineMarkdown("./README.md", undefined, {
  allowPrivate: true,
  sections: ["authentication"],     // specific sections, or ["all"], or [] for TOC
});
console.log(inline.output);
console.log(inline.inlinedCount);

// Check freshness (no file modification)
const check = await checkMarkdown("./README.md");
for (const diag of check.diagnostics) {
  console.log(`${diag.url}: ${diag.status}`);
}

// Restore — reverse condense/inline
const restored = await restoreMarkdown("./condensed.md");
console.log(restored.restoredCount);

Additional exports: validateUrl, convertHtmlToMarkdown, matchSections, slugifyHeading, readLockfile, writeLockfile, resolveLockfilePath, hashUrl, estimateTokens, and cache functions (readBody, readMarkdown, readIndex, etc.). Full types included.

Agent integration

An agent workflow file (.windsurf/workflows/doc-lok.md in the repo) teaches agents how to detect and use doc-lok automatically — when to check, condense, inline, or restore, with 11 explicit rules and a decision tree. The --check --json and --json flags provide machine-readable output for automated workflows.

To use it with your IDE, copy the workflow file from the repo into your IDE's workflow/rules directory:

# Windsurf
curl -fsSL https://raw.githubusercontent.com/<your-org>/doc-lok/main/.windsurf/workflows/doc-lok.md \
  -o .windsurf/workflows/doc-lok.md

# Cursor — adapt the format for .cursor/rules/
# Claude Code — adapt for .claude/commands/

The workflow file is IDE-agnostic Markdown — the content works in any agent that reads workflow or rules files.

CLI reference

doc-lok <file.md> [mode] [options]

Modes:
  (default)         Condense — replace unchanged links with markers
  --inline          Inline — fetch and inject linked content
  --check           Check — validate URLs, don't modify the file
  --restore         Restore — reverse condense/inline

Options:
  --section <name>    Section(s) to inline (repeatable). Special: "all", "toc"
  --converter <mode>  HTML→Markdown: "minimal" (default) or "turndown"
  --max-bytes <n>     Max inline body size in bytes (default 1048576)
  --cache-dir <path>  Override .doc-lok/cache/ directory
  --lockfile <path>   Override lockfile path
  --allow-private     Allow private/loopback URLs (disables SSRF guard)
  --json              Structured JSON output to stdout
  -q, --quiet         Suppress stderr diagnostics
  -V, --version       Print version
  -h, --help          Show help

Exit codes: 0 (success), 1 (fatal error), 2 (arg error)

stdout receives the condensed/restored Markdown (or JSON with --json). stderr receives human-readable diagnostics. Safe to pipe.

Full CLI docs: docs/cli-reference.md Section guide: docs/sections.md

How it works

Lockfile

.doc-lok/lock.json stores per-URL metadata:

{
  "version": 3,
  "global_tokens_saved": 18432,
  "urls": {
    "https://example.com/docs": {
      "last_known_sha256": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
      "etag": "\"abc123\"",
      "token_cost_raw": 357,
      "token_cost_compressed": 18,
      "last_checked": "2026-07-08T11:30:00.000Z",
      "cached": true,
      "original_text": "documentation",
      "converted": true,
      "section_slugs": ["introduction", "authentication", "api-reference"]
    }
  }
}

Lockfile resolution order (first match wins):

--lockfile <path> argument
DOC_LOK_LOCKFILE environment variable
.doc-lok/lock.json in the Markdown file's directory
.doc-lok/lock.json in process.cwd()

The lockfile is written atomically (temp-file-then-rename) to prevent corruption from concurrent processes.

Network validation

Two-phase strategy to minimize network overhead:

HEAD request — compare server ETag against lockfile. Match = unchanged, no body transfer.
Streamed GET fallback — when ETag is missing or mismatched, stream the body through an incremental SHA-256 hasher (O(1) memory regardless of payload size). Compare hash to lockfile.

Redirects (301/302/307/308) are followed automatically (up to 5 hops), with SSRF re-check at each hop.

Inline cache

--inline stores fetched content in .doc-lok/cache/:

<sha>.raw — original response body
<sha>.md — converted Markdown
<sha>.index.json — section index

Repeat runs with unchanged URLs are HEAD-only — no body re-fetch, no re-conversion. The converted Markdown and section index are cached on disk so they're free on repeat runs too.

Code-block awareness

Links inside inline code, fenced code blocks, and indented code blocks are never touched — code examples stay readable and functional.

Error isolation

Each URL is validated independently. A 503, DNS failure, or timeout on one link produces an error diagnostic but never aborts the entire run.

Token savings — honestly

The baseline for --inline and --section is an agent fetching the full URL and reading the entire page — not the original Markdown file with just a link. Nobody uses --inline if they don't want the content in the prompt.

Mode	What's saved	How much
Condense	LLM tokens (link text → 18-token marker)	~5 tokens per unchanged link
Inline (TOC default)	LLM tokens (full page → ~200-token TOC)	~49,800 tokens per URL — agent sees the index, fetches only what it needs
Inline (`--section <name>`)	LLM tokens (full page → only the requested section)	Varies — a 50K-token doc with a 2K-token section saves ~48K tokens
Inline (`--section all`)	Network + latency only (HEAD-only repeat runs)	0 LLM tokens vs. fetching the page yourself — same content, just pre-cached
Summary (roadmap)	LLM tokens (full body → ~300-token summary)	~49,700 tokens per URL on repeat runs

All token counts are heuristic estimates (≈4 chars/token), not exact tokeniser measurements. Use them as a rough guide for comparing relative savings across runs, not as a billing figure.

A side benefit of --inline: byte-identical repeat prompts unlock provider-side prompt caching (Anthropic Claude, OpenAI both cache identical prompt prefixes and bill ~10x less for the cached segment).

Limitations

HTTP(S) only. Relative links, mailto:, and other schemes are ignored.
No authentication. doc-lok sends no Authorization headers. Private/authenticated URLs return 401/403 and are marked as errors. This is a security boundary, not a missing feature.
SSRF guard on by default. Blocks loopback, link-local, and private IP ranges. Use --allow-private to opt in for internal URLs.
Sequential validation. URLs are fetched one at a time. For documents with hundreds of links, this may be slow. Use the library API with validateUrl() directly if you need concurrent fetching.
One level deep. --inline fetches linked pages but does not recursively follow links within them.
--check still writes the lockfile. The Markdown file is untouched, but the lockfile is updated with current metadata.
Same URL, different anchor texts. The lockfile stores only the last-seen anchor text. All restored markers for that URL will use it.
Token estimates are approximate. The 4-chars/token heuristic is a rough guide, not a billing figure. CJK text is closer to 1 char/token; URLs are token-dense.

Project structure

doc-lok/
├── src/
│   ├── index.ts      # Public API re-exports
│   ├── cli.ts        # CLI entry point (argv parser → stdout/stderr)
│   ├── parser.ts     # Condense, inline, restore, check orchestration
│   ├── scanner.ts    # Code-block-aware Markdown link extraction
│   ├── network.ts    # HEAD-first → streamed SHA-256 GET
│   ├── state.ts      # Lockfile read/write, token estimation
│   ├── cache.ts      # Body cache (.raw, .md, .index.json) for --inline
│   ├── convert.ts    # HTML→Markdown converter + section detection
│   ├── sections.ts   # Section matching (3-tier: exact → case-insensitive → contains)
│   └── ssrf.ts       # SSRF guard (private/loopback/link-local blocking)
├── test/             # Test suite (vitest)
├── docs/
│   ├── cli-reference.md
│   └── sections.md
└── package.json

Development

npm install      # dev dependencies (typescript, @types/node, vitest)
npm run build    # compile src/ → dist/ (tsc)
npm test         # run test suite
npm run clean    # remove dist/

Contributing

Fork, create a feature branch.
Build and test: npm run build && npm test.
Keep zero runtime dependencies — Node.js built-ins only.
Maintain strict TypeScript (strict: true).
Update README when adding CLI flags or API changes.
Commit with conventional commits: feat:, fix:, docs:.

License

MIT

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
.windsurf/workflows		.windsurf/workflows
docs		docs
scripts		scripts
src		src
test		test
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json
vitest.config.ts		vitest.config.ts

Navigation Menu

Repository files navigation

doc-lok

What it does

Modes

What's NOT shipped yet

Install

Usage

The basic workflow

Choosing a mode

Piping to other tools

JSON output for automation

Using as a library

Agent integration

CLI reference

How it works

Lockfile

Network validation

Inline cache

Code-block awareness

Error isolation

Token savings — honestly

Limitations

Project structure

Development

Contributing

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

doc-lok

What it does

Modes

What's NOT shipped yet

Install

Usage

The basic workflow

Choosing a mode

Piping to other tools

JSON output for automation

Using as a library

Agent integration

CLI reference

How it works

Lockfile

Network validation

Inline cache

Code-block awareness

Error isolation

Token savings — honestly

Limitations

Project structure

Development

Contributing

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages