ContextAudit.ts reports false confidence: placeholder blindness, cross-ref false positives, and a critical check blind to most sections

[tzioup]

Structured for machine parsing: claims are atomic, each carries the command that verifies it. Every command runs against a fresh clone of this repo — none depend on my machine, my install, or my configuration. Full reproduction at the bottom.

Summary

LIFEOS/TOOLS/ContextAudit.ts ships in the install payload and is the content-quality gate invoked at the end of the /interview context check-in (skills/Interview/Workflows/ContextCheckin.md Step 4). Its own header calls it a "read-only quality audit for constitutional context files."

Its checks are actually reference/placeholder hygiene, and several are unsound in ways that make its "clean" verdict untrustworthy — a validation tool that produces both false negatives and false positives doesn't just miss bugs, it launders a broken state as a passing one. Five independent issues, each verifiable from a fresh clone:

1. False negative — the placeholder check cannot see the markers the templates actually ship. scanPlaceholders looks for <your-name>, <TBD>, <...>, (seeded during interview); the shipped templates use (sample) and (interview). A 100%-unfilled template file scores zero placeholder findings.
2. False positive — the cross-reference check treats any backtick span as a literal path. A span like `LIFEOS/TOOLS/MemorySystem.ts add(item)` is existsSync-checked whole and warned as broken, though the file exists; symlink notation `A → B` is warned as a missing file.
3. The tool's own critical empty-section check is blind to most sections. scanPrincipalTelos fires only on three exact heading strings and uses exact-match heading resolution, so a one-character variant or any other heading (including Core Models, which is empty by construction — see Load Dynamic Requirements runs repeatedly? #4) is never flagged critical.
4. The generator emits empty sections unconditionally, and excludes two of them from its own fail-loud guard. GenerateTelosSummary.ts emits ## Core Models (and 6 other headings) with no if (array.length > 0) guard, while its per-section coreChecks safety net omits narratives and models.
5. The audit writes to a hardcoded, dated, personal-looking output path rather than a stable location.

The #1/#3/#4 chain is the sharp one: the generator reliably produces an empty ## Core Models, and the audit's dedicated critical check is structurally unable to catch it.

Claims and verification

git clone --depth 1 https://github.com/danielmiessler/LifeOS && cd LifeOS
# path-robust anchors (works regardless of the exact payload prefix):
CA=$(find . -path '*/LIFEOS/TOOLS/ContextAudit.ts' | head -1)
GEN=$(find . -path '*/LIFEOS/TOOLS/GenerateTelosSummary.ts' | head -1)

C1 — scanPlaceholders checks four patterns, none of which is (sample) or (interview); the shipped TELOS/identity templates use exactly those two.

grep -n -A6 'const patterns' "$CA" | grep -E 'label:|re:'
#   <your-name> | <TBD> | <...> | (seeded during interview)   -- the four checked

grep -rlE '\(sample\)|\(interview' install/USER | head
#   install/USER/TELOS/{MISSION,PROBLEMS,CHALLENGES,...}.md and the identity templates — all use (sample)/(interview)

Consequence: a fully-unfilled template file produces 0 placeholder findings. (Verified by unit-running the exported scanPlaceholders against a shipped (sample)-only template file: empty result.)

C2 — scanCrossRefs/normalizeReference take the entire backtick span as one filesystem path — no stop-at-whitespace, no code-span exclusion, no runtime-path allowlist.

sed -n '/function normalizeReference/,/^}/p' "$CA"
#   returns join(CLAUDE_DIR, value) for the whole span; only strips a trailing #anchor and rejects glob/URL

So a documentation span that embeds a real path plus trailing text — e.g. `LIFEOS/TOOLS/MemorySystem.ts add(item)` (a method reference) or `LIFEOS/USER → ~/.config/LIFEOS/USER` (symlink prose) — is existsSync-checked as the literal string and warned "broken," even though LIFEOS/TOOLS/MemorySystem.ts exists. This is the same class the project already documents for ReferenceCheck.ts: a raw missing-reference count that is majority false-positive without a runtime-path allowlist.

C3 — the critical empty-section check covers three literal headings via exact-match, so it is blind to near-miss spellings and to every other emitted section.

grep -n 'const headings' "$CA"
#   ["Missions", "Active Goals (2026)", "Problems Being Solved"]
sed -n '/function sectionBody/,/return out/p' "$CA" | grep -E 'headingRe|findIndex'
#   exact-match regex on the heading text — no case/fuzzy tolerance

## Core Models is not in the checked set and is never flagged critical by this dedicated check — even though it is empty by construction (C4). A heading spelled ## Mission (singular) is likewise invisible.

C4 — GenerateTelosSummary.ts emits ## Core Models unconditionally, and its coreChecks fail-loud guard omits narratives and models.

grep -n "Core Models" "$GEN"
#   pushed via lines.push('', '## Core Models', '', ...models, ...) with NO if-guard
grep -n -A8 'coreChecks' "$GEN" | grep -oE "'(mission|goals|problems|strategies|challenges|narratives|models)'"
#   mission, goals, problems, strategies, challenges   -- narratives and models absent

Guarded sections (traumas, wrong, narratives.secondary) use if (x.length > 0); ## Core Models and ## Active Narratives are both unconditionally emitted and outside the zero-items guard — so an empty parse renders a bare heading with no fail-loud signal. This is what produces the empty ## Core Models in every generated PRINCIPAL_TELOS.md that has no models data, which C3 then cannot catch.

C5 — the audit output path is a hardcoded dated slug under MEMORY/WORK/, created on any run.

grep -n "20260503-230000" "$CA"
#   AUDIT_PATH = .../MEMORY/WORK/20260503-230000_freshness-extends-to-constitutional-files/AUDIT.md

mkdirSync(..., {recursive:true}) materializes this dated directory on every run, regardless of the running user.

Scope of the claim

• C1–C5 are source-level and reproduce from a fresh clone with the commands above. C1 and the cross-ref behavior in C2 were additionally confirmed by unit-running the tool's own exported functions against crafted and shipped-template inputs on a byte-identical copy of the payload; I did not run the whole tool end-to-end against a fully-populated install in a clean-room.
• The cross-ref false-positive rate I observed (roughly 2 of 3 on crafted input, ≥3 of 7 on one real doc) is illustrative, not a population claim — it depends on how many doc references embed trailing text.
• One further check, scanSystemPromptModel, is technically correct but low-value: it fires whenever the system prompt lacks the current model-ID string (grep -c = 0 on the shipped LIFEOS_SYSTEM_PROMPT.md), which a behavior-spec file has no reason to contain — a standing info nag rather than a real signal. Minor. Separately, the empty-section threshold counts markdown/bold/ID markup toward its 20-char budget, so a bullet with a 2-char ID needs only ~10 real characters to pass as "substantive" — a weak threshold, not a hard bug.
• I make no claim about intent; these are observations about what the shipped code does.

Suggested fix

The five are connected (one tool's reliability) but separate by file, so this can land as one PR or split — steps 1–3 and 5 are in ContextAudit.ts, step 4 is in GenerateTelosSummary.ts:

1. C1 — add (sample) and (interview to scanPlaceholders patterns (or, better, source the marker set from the same place the templates define it).
2. C2 — in normalizeReference, take only the leading path token (stop at first whitespace) and/or apply the runtime-path allowlist this project already discusses for ReferenceCheck.ts, so code-embedded and prose paths stop counting as broken references.
3. C3 — drive the empty-section critical check from the generator's actual emitted-heading list (or make it case/whitespace-tolerant and cover all sections), so it can't silently skip the section that is empty by construction.
4. C4 — guard ## Core Models (and the other array-derived headings) with if (array.length > 0), and add narratives and models to coreChecks.
5. C5 — write the audit to a stable, non-dated, non-personal location.

I verified 1 and 4 locally and can open the PR if useful.

Reproduction

git clone --depth 1 https://github.com/danielmiessler/LifeOS && cd LifeOS
CA=$(find . -path '*/LIFEOS/TOOLS/ContextAudit.ts' | head -1)
GEN=$(find . -path '*/LIFEOS/TOOLS/GenerateTelosSummary.ts' | head -1)

# C1 (false negative): patterns vs the markers templates actually use
grep -n -A6 'const patterns' "$CA"
grep -rlE '\(sample\)|\(interview' install/USER

# C2 (false positive): whole-span-as-path
sed -n '/function normalizeReference/,/^}/p' "$CA"

# C3 (blind critical check): three exact headings, exact-match resolution
grep -n 'const headings' "$CA"

# C4 (generator empty-section): unconditional emit + coreChecks omission
grep -n "Core Models" "$GEN"; grep -n -A8 'coreChecks' "$GEN"

# C5 (hardcoded dated output dir)
grep -n "20260503-230000" "$CA"

[danielmiessler]

Fixed in source, all of it. This was the best-constructed report of the batch: every claim atomic, every one reproduced. C1: (sample) and (interview are now placeholder patterns. C2: normalizeReference keeps only the leading token of a backtick span. C3: the empty-section critical check covers all nine generator-emitted sections, prefix-matched so "Active Goals (2026)" matches. C4: the generator no longer emits an empty Core Models section at all. C5: AUDIT.md moved to a stable MEMORY/STATE/context-audit/ location. Verified live: the audit runs clean here and the one new warning it raised was a true positive. Ships with the next release.