/dataset-synthesize: mark as candidate / dogfood pending / silent-failure paths confirmed

[brandon-behring]

Tracks toolkit-internal acknowledgment that /dataset-synthesize shipped pre-burn-in
in PR #16 (closes #1) without entering the toolkit's normal dogfood loop. This issue
is a backlog marker; #21 already tracks the polish gaps.

Status (informational; no code change requested in this issue)

• Burn-in absent. BURN_IN_NOTES.md has no post-ship dogfood entry for this skill.
  Neither v2.3.0 nor v2.4.0 release summaries mention it; the skill is effectively
  orphaned from the project's burn-in process.
• Confirmed silent-failure path. scripts/dataset_synthesize.py:316-322 (_extract_text)
  filters response content blocks to type == "text" and silently returns "" on
  all-non-text responses with no warning. Tracked in /dataset-synthesize: post-merge polish (cost-table staleness, non-text block warning, mock fidelity, cost-invariant test, API-key check location) #21 item Cross-platform path portability: dossier cache_manifest.yml should never store absolute paths #2 but worth
  re-emphasizing — directly contradicts ~/.claude/rules/python-standards.md:
  "NEVER fail silently. Raise explicit ValueError/TypeError with context."
• Confirmed fail-late path. synthesize() (line 354) accepts client=None and
  lazy-instantiates anthropic.Anthropic() at line 417, but the ANTHROPIC_API_KEY
  presence check lives only in main() (line 593). Programmatic callers get a deep
  SDK error instead of the friendly env-var-missing message. Tracked in /dataset-synthesize: post-merge polish (cost-table staleness, non-text block warning, mock fidelity, cost-invariant test, API-key check location) #21 item research-kb-export produces orphan files: no consumer exists in research-kb #5.
• Test fidelity for cache_control placement is actually OK
  (tests/test_dataset_synthesize.py:194-232) — issue /dataset-synthesize: post-merge polish (cost-table staleness, non-text block warning, mock fidelity, cost-invariant test, API-key check location) #21 item agent-index description doesn't mention v2.2 atomic decomposition / Attribute-First / pre_selection_manifest #3 is narrower than
  its wording suggests. The actual gap is narrower: tests don't import anthropic.types
  to enforce the SDK's messages.create() signature. Low-severity.
• Distribution gap (per references/skill_audit_2026-05-26.md): the skill is
  absent from Makefile SKILLS, absent from the README quickstart symlink loop,
  and not present in the local ~/.claude/skills/ install. A skill nobody can
  invoke from a fresh session is not a first-class skill regardless of
  implementation quality. See companion issue on registry parity.

Recommended path to ready

1. Run /dataset-synthesize against one real recipe (any existing dogfood corpus)
   and log friction in BURN_IN_NOTES.md, mirroring the way /topic-discovery
   got commit 46d833e.
2. Fix /dataset-synthesize: post-merge polish (cost-table staleness, non-text block warning, mock fidelity, cost-invariant test, API-key check location) #21 items Cross-platform path portability: dossier cache_manifest.yml should never store absolute paths #2 and research-kb-export produces orphan files: no consumer exists in research-kb #5 (the two confirmed silent/late paths). /dataset-synthesize: post-merge polish (cost-table staleness, non-text block warning, mock fidelity, cost-invariant test, API-key check location) #21 items feat(skills): /dataset-synthesize — prompt-cached synthetic data generation #1,
   agent-index description doesn't mention v2.2 atomic decomposition / Attribute-First / pre_selection_manifest #3, dossier-build effectively skipped in v2.2 strict-live; skill body still describes it as agent-index's upstream #4 can stay deferred as labeled.
3. Add a ## Status section to .claude/skills/dataset-synthesize.md linking to
   this issue + /dataset-synthesize: post-merge polish (cost-table staleness, non-text block warning, mock fidelity, cost-invariant test, API-key check location) #21 until items Cross-platform path portability: dossier cache_manifest.yml should never store absolute paths #2 and research-kb-export produces orphan files: no consumer exists in research-kb #5 are closed.
4. Fix the registry/install drift (separate issue) so the skill is actually
   invokable from a fresh session.

Closes when items 1–3 above are done.

Context

Filed 2026-05-26 from two complementary audits — internal (behavioral, this body)
and external (references/skill_audit_2026-05-26.md, distribution layer). Plan
file at ~/.claude/plans/skill_audit_2026-05-26.md on the
Linux box has the full audit reasoning.

[brandon-behring]

Backlogged 2026-06-02 pending a worthwhileness review of the consumer repo (prompt-injection-portfolio) Lane 2 synthetic-data generation.

Scope clarification: this issue is two cheap reliability fixes — _extract_text should raise instead of returning "" on a no-text response, and the ANTHROPIC_API_KEY check should be hoisted into synthesize(). ~1h + tests, standalone value for any caller. The expensive part (generating + κ-auditing the ~20k Lane 2 corpus) is what's actually gated.

Cross-model refusal probe (2026-06-02): Claude (subagent) and llama3.1:8b refuse synthetic PI-attack generation even with defensive framing; qwen2.5:7b complies but low-diversity. Cost is not the blocker (qwen local $0; metered API ≈ $90–140 for 20k, hard-capped by --bail-at-cost) — refusal + diversity are. The API path therefore carries refusal risk, not just cost.

Gate: proceed only if the Lane 2 synthetic-carrier hypothesis is load-bearing for the portfolio thesis. If not → wontfix candidate alongside #21 / #23.