PAI v4.0.0 — Lean and Mean

[danielmiessler]

PAI v4.0.0 — Lean and Mean

63 skills, 180 workflows, 21 hooks — and 81% of the context window is free for your actual work.

---

Context Efficiency: The Whole Point of v4.0

Every AI agent system has the same fundamental constraint: context window size. Every token your system spends on scaffolding — skill descriptions, routing tables, hook definitions, identity configuration — is a token not available for actual work.

This matters more than people realize:

• Fewer tokens per turn = faster responses
• Lower base context = more turns before compaction kicks in
• More room for actual work = better quality output on complex tasks
• Less scaffolding noise = the model focuses on what you asked, not on parsing system overhead

v3.0 was powerful but heavy. 38 flat skill directories, each loading its own SKILL.md description into context every single turn. Dead systems taking up space. A static CLAUDE.md that had to be comprehensive because it couldn't adapt.

v4.0 re-architects PAI as a dynamic loading system. Super lean at startup, expanding only as needed.

The screenshot above is a live PAI v4.0 session at startup — 19% context usage with the full system loaded. That's down from ~38% in v3.0. Half the context footprint, same capabilities.

What This Means In Practice

• More turns before compaction — The biggest practical win. You can work longer in a single session before Claude Code compresses your conversation history. That means fewer context resets, better continuity, and less re-explaining.
• Faster responses — Less scaffolding for the model to process on every turn means faster time-to-first-token.
• Better output quality — When the model has more working memory available, complex tasks get better results. Planning, debugging, multi-file refactors — everything benefits.
• NATIVE mode intelligence — v4.0 is smarter about when to use the full Algorithm vs. lightweight NATIVE mode. Simple tasks don't need the full overhead, so they don't get it.

How We Got There

The context reduction came from three architectural changes:

1. Skill hierarchy — 38 flat directories compressed into 12 hierarchical categories. Each category loads ONE SKILL.md that routes to sub-skills. Instead of 38 competing descriptions in context, there are 12.

2. Dead system removal — Three entire systems (Components/, DocRebuild.hook.ts, RebuildSkill.ts) were silently failing for months and still consuming context. Removed entirely. Zero functionality lost.

3. Dynamic CLAUDE.md — Generated from a template at session start instead of being a static file. Only loads what's needed, with variables resolved automatically from settings.json.

---

Everything Else That Changed

Beyond context efficiency, v4.0 includes:

Algorithm v3.5.0

Jumped from v1.4.0 (shipped with v3.0) to v3.5.0. Now lives at PAI/Algorithm/v3.5.0.md with a LATEST file pointing to the current version.

CLAUDE.md Template System

CLAUDE.md is no longer a static file:

CLAUDE.md.template          ← Source of truth (with variables)
    ↓
PAI/Tools/BuildCLAUDE.ts   ← Resolves variables from settings.json + LATEST
    ↓
CLAUDE.md                   ← Generated output (loaded by Claude Code)

SessionStart hook runs BuildCLAUDE.ts automatically — keeps CLAUDE.md fresh without manual intervention.

Installer Updates

Every version reference across the entire installer was audited and updated. Electron fix: pre-built node_modules removed from release — Electron installs fresh on the target machine, avoiding the macOS quarantine crash that plagued v3.0.

Security & Sanitization

Comprehensive audit with multiple agent sweeps:

• Personal branding removed — references cleaned from 33+ files
• Identity templated — Pronunciation, timezone, voice clone ID, DA personality all use template variables
• Voice clone protected — Personal voice clone ID replaced with template variable
• Pronunciations cleaned — Removed DA-revealing phonetics from VoiceServer
• Pre-commit hook updated — New v4.0-aware bash hook replaces broken reference

---

By The Numbers

Metric	v3.0	v4.0	Change
Context at startup	~38%	~19%	-50% context footprint
Skill directories	38	13 categories	-66% top-level dirs
SKILL.md files	41	63	+22 (sub-skill routing)
Workflows	162	180	+18
Hooks	20	21	+1
Agent templates	13	14	+1
Dead systems	3	0	Cleaned
Algorithm version	v1.4.0	v3.5.0	+21 minor versions
Stale version refs	14+	0	All updated

---

Installation

# Clone the repo
git clone https://github.com/danielmiessler/Personal_AI_Infrastructure.git
cd Personal_AI_Infrastructure/Releases/v4.0

# Copy the release and run the installer
cp -r .claude ~/ && cd ~/.claude && bash install.sh

The installer handles prerequisites, configuration, and setup. Run source ~/.zshrc && pai after installation.

---

Upgrading from v3.0

If you're running v3.0:

1. Back up your ~/.claude/ directory
2. Copy v4.0 over your existing installation: cp -r .claude ~/
3. Run BuildCLAUDE to generate your CLAUDE.md: bun ~/.claude/PAI/Tools/BuildCLAUDE.ts
4. Verify your settings.json merged correctly — your identity, voice, and preferences should persist

---

Full release notes →

[icryptan]

You build amazing project, Daniel. Thank you for sharing

[jlacour-git]

Thank you very, very much, Daniel!

Can you tell us which of the issues are addressed with this update? I see some I would guess to be fixed, but I am not sure how your process works regarding what goes into a release and how. Some advice would be great!

Cheers,
Jochen

[jlacour-git]

Upgrade Experience: v3.0 → v4.0 (Customized Install)

TL;DR: Upgraded a heavily customized v3.0 (15 patches, custom hooks, Kokoro voice) to v4.0. Took ~4 hours for a careful, customized upgrade. The new architecture is a clear improvement — Algorithm v3.5.0, BuildCLAUDE.ts, and context routing are excellent. But cp -r is dangerous for customized installs: it silently overwrites settings.json and patched files. Found 10 gotchas including hook renames that silently drop functionality (#2), broken handler import paths (#7), and a statusline startup estimate that shows wrong context % (#10). 8 patches still needed, 3 custom hooks added. Details and fixes below.

---

Upgraded a heavily customized PAI v3.0 installation (15 active local patches, custom hooks, Kokoro voice server, custom skills) to v4.0. Here's what worked and what to watch out for.

What I did differently from the suggested cp -r

Daniel's suggested path (cp -r .claude ~/) works for fresh installs, but for customized setups it silently overwrites:

• settings.json — identity, hooks, permissions, all custom config gone
• Patched files — any local fixes get reverted without warning

Instead I used:

# 1. Backup
cp -a ~/.claude ~/.claude-backup-pre-v4-$(date +%Y%m%d)

# 2. rsync (preserves custom files, only adds/overwrites v4.0 files)
rsync -av --exclude='settings.json' --exclude='MEMORY/' \
  --exclude='CONCEPT.md' --exclude='LOCAL_PATCHES.md' \
  ~/PAI/Releases/v4.0/.claude/ ~/.claude/

# 3. Manual settings.json merge (v4.0 structure + my identity/hooks)

# 4. Copy USER files: skills/PAI/USER/ → PAI/USER/

# 5. Clean old paths: skills/PAI/Components/, context-tiers/, scripts/

# 6. Build: bun PAI/Tools/BuildCLAUDE.ts

Key gotchas discovered

1. USER files must be manually migrated — v4.0 moves USER from skills/PAI/USER/ to PAI/USER/. The cp -r copies the new scaffold but doesn't know about your populated files at the old path.

2. Hook renames are NOT 1:1 replacements — 32 → 22 hooks. Some renames dropped functionality silently. SessionSummary → SessionCleanup lost the session summary writer spawn. StopOrchestrator → 4 separate Stop hooks lost AlgorithmEnrichment (state enrichment from responses) and SessionSummaryDraft (rolling mid-session summaries). Don't assume the new hook does everything the old one did — diff them.

3. contextFiles should be empty — v4.0 loads core context via CLAUDE.md (native Claude Code behavior) and loadAtStartup in settings.json. If you still have v3.0-style contextFiles pointing to skills/PAI/SKILL.md, you'll load stale context.

4. Statusline bugs persist — Three bugs from v3.0 are still in v4.0's statusline: hardcoded America/Los_Angeles timezone (PRs fix(statusline): read timezone from settings.json instead of hardcoding #762), f-string syntax error (PR fix(statusline): fix f-string syntax error silently breaking reset time display #780), and broken context fallback after /clear (PR fix(statusline): remove broken context percentage fallback after /clear #806).

5. Old skills/PAI/ is now dead weight — After upgrade, skills/PAI/ still exists with 1300+ line SKILL.md and the Components/ build system. The new PAI lives at top-level PAI/. Remove skills/PAI/ entirely to avoid confusion.

6. PAI_DIR path expansion — verify your setup (potential, not verified) — v4.0 defaults to ${HOME}/.claude, which was the fix for the tilde expansion bug (Tilde (~) in PAI_DIR env variable not expanded, causing hook and statusline failures #478). This should work since ${HOME} expands in shell contexts. But if you previously set an absolute path or used ~/.claude somewhere, verify PAI_DIR resolves correctly in hook commands after upgrade. An absolute path (e.g., /Users/yourname/.claude) avoids any ambiguity.

7. Handler import paths break — Several handler files in hooks/handlers/ import from ../../skills/PAI/Tools/ (TranscriptParser, Inference). Since v4.0 moved PAI to top-level, these paths are stale. The handlers compile fine but crash at runtime. Fix: update imports to ../../PAI/Tools/.

8. Shell aliases break — If you had alias pai='bun ~/.claude/skills/PAI/Tools/pai.ts' in .zshrc (or similar), it now points to a deleted path. Update to PAI/Tools/pai.ts. Easy to miss because the alias error only shows when you open a new terminal.

9. Statusline display changes (personal preference) — v4.0's statusline simplified three display elements we had in v3.0: token count (48k/200k) on the CONTEXT line, full path (~/.claude) on the PWD line (now basename only), and model name on the USAGE line. These may be intentional design choices. We restored them locally because we prefer the additional detail, but this is preference, not a bug.

10. Statusline startup estimate shows wrong context % — v4.0 added a "self-calibrating startup estimate" that caches a base token count from the previous session and uses it to display context % before the first API call. Problem: if the previous session was deep in conversation (e.g., 66k cached base), fresh sessions show inflated values (41% instead of the real 20%). The startup context is deterministic — there's nothing to estimate. Fix: remove the estimation block (lines 105-154 in statusline-command.sh) and the cache file (MEMORY/STATE/context-base-tokens.txt). Context shows 0% for a few seconds until the first API response, which is honest. v3.0 did this and nobody complained.

What worked great

• BuildCLAUDE.ts is much cleaner than the old Components/ → SKILL.md pipeline
• Algorithm v3.5.0 at 337 lines (lazy-loaded) vs v1.8.0 at 1300+ lines (embedded) — major context savings
• CLAUDE.md as master config with NATIVE/ALGORITHM/MINIMAL modes is elegant
• Native context routing via CONTEXT_ROUTING.md (our ContextRouter concept from feat: Selective context loading via ContextRouter hook to reduce 83KB base context #690 realized!)

Post-upgrade state

• Smoke tests pass (voice, identity, memory, Algorithm, statusline, hooks)
• 10 local patches retired (superseded by v4.0 architecture changes)
• 8 patches still active (5 statusline, PostCompactRecovery, SystemFileGuard, handler import fixes)
• 3 custom hooks added to restore lost functionality (SessionSummarySpawn, SessionSummaryDraftStop, AlgorithmEnrichmentStop)
• Voice, identity, memory all intact
• Rollback: rm -rf ~/.claude && mv ~/.claude-backup-pre-v4-20260228 ~/.claude

Behavioral lesson for AI-assisted upgrades

If your DA helps with the upgrade, be aware that hook "renames" in changelogs are easily accepted at face value. My DA mapped SessionSummary → SessionCleanup and StopOrchestrator → 4 Stop hooks without diffing the files — missing that SessionCleanup dropped the summary writer and 2 of 6 StopOrchestrator handlers got no replacement. I added a steering rule that triggers on mapping verbs ("→", "replaces", "renamed to") — forcing a code-level diff before accepting any A → B equivalence claim during planning, not just implementation.

Recommendation for others with customized installs: Don't cp -r. Use rsync --exclude and merge settings.json manually. The rsync + settings merge takes ~15-20 minutes. Finding and fixing gotchas in a heavily customized install took us ~4 hours total.

[mikevalle12]

Jochen,

Thank you very much for this detailed post. I was able to create a Conductor plugin track to manage the upgrade and pointed it to this post and your comment. It created a plan for the upgrade and then addressed the issues you documented. My upgrade took less than 30 minutes and seems to be working great.

Your efforts are greatly appreciated.

Mike

[RubyWolff27]

Excellent upgrade, thank you for this Daniel.

[danielmiessler]

Closing this out as part of a cleanup of the discussion board. We're on the 7.x releases now and a lot has changed since this thread was active. Thanks to everyone who jumped in. If this is still relevant on 7.x, please open a fresh discussion or issue and we'll pick it up there.