Follow-up hardening for /dataset-synthesize (shipped in PR #16,
closes #1). These are NICE-TO-HAVE items surfaced during the PR #16
code review but deferred to keep the initial PR focused. None are
blocking; all are low-risk polish.
1. PRICING_PER_MTOK staleness guard
scripts/dataset_synthesize.py hardcodes per-model pricing. If
Anthropic changes prices, --bail-at-cost math is silently wrong
(user thinks they capped at $80; actual differs).
Fix: add a comment with the "pricing verified as of YYYY-MM-DD"
date + the Anthropic pricing page URL next to the table. Optionally
a soft self-check / staleness note in --help.
2. _extract_text silently drops non-text content blocks
If the API returns tool_use / image / other non-text blocks,
_extract_text returns "" and the sample row gets content: ""
with no warning. Silent quality degradation.
Fix: emit a print("warning: ...", file=sys.stderr) when a
response yields zero text blocks, OR record a per-row flag in the
JSONL so downstream consumers can filter.
3. Mock fidelity — _FakeClient doesn't enforce real SDK shape
tests/test_dataset_synthesize.py uses hand-rolled _FakeClient /
_FakeMessage / _FakeUsage doubles (per docs/conventions/test-style.md,
correct pattern). But they don't assert the production call shape
(e.g., system must be a list of content blocks; cache_control
present). A future refactor breaking the prompt-caching contract
would pass tests but break real caching.
Fix: add one smoke test that imports anthropic.types (guarded
by the lazy-import availability flag) and asserts the kwargs
synthesize() passes match the SDK's expected messages.create
signature; OR assert _FakeMessages.create received a list-shaped
system with cache_control on the right blocks.
4. Cost-invariant test assertion
No test asserts sum(per_template[*].cost_usd) == total_cost_usd in
the manifest. A regression in cost aggregation (assign vs increment)
would produce a correct total but wrong per-template breakdown, or
vice versa, undetected.
Fix: add an assertion to test_synthesize_happy_path_3_samples
(or a dedicated test) verifying the per-template costs sum to the
manifest total.
5. API-key check location
The ANTHROPIC_API_KEY presence check lives in main() (CLI path
only). synthesize() is a public library function; calling it
directly with client=None + no env var raises a deep, non-obvious
SDK error instead of the friendly "ANTHROPIC_API_KEY env var not set"
message.
Fix: move (or duplicate) the env-var check into synthesize()
before the lazy anthropic.Anthropic() instantiation, OR document
the requirement clearly in the function docstring.
Context
Filed 2026-05-25 as deferred polish from PR #16 review. Consumer
(prompt-injection-portfolio Lane 2) does not need these before M3;
they harden the skill for broader reuse. Good first-issue candidates.
Follow-up hardening for
/dataset-synthesize(shipped in PR #16,closes #1). These are NICE-TO-HAVE items surfaced during the PR #16
code review but deferred to keep the initial PR focused. None are
blocking; all are low-risk polish.
1. PRICING_PER_MTOK staleness guard
scripts/dataset_synthesize.pyhardcodes per-model pricing. IfAnthropic changes prices,
--bail-at-costmath is silently wrong(user thinks they capped at $80; actual differs).
Fix: add a comment with the "pricing verified as of YYYY-MM-DD"
date + the Anthropic pricing page URL next to the table. Optionally
a soft self-check / staleness note in
--help.2.
_extract_textsilently drops non-text content blocksIf the API returns tool_use / image / other non-
textblocks,_extract_textreturns "" and the sample row getscontent: ""with no warning. Silent quality degradation.
Fix: emit a
print("warning: ...", file=sys.stderr)when aresponse yields zero text blocks, OR record a per-row flag in the
JSONL so downstream consumers can filter.
3. Mock fidelity —
_FakeClientdoesn't enforce real SDK shapetests/test_dataset_synthesize.pyuses hand-rolled_FakeClient/_FakeMessage/_FakeUsagedoubles (per docs/conventions/test-style.md,correct pattern). But they don't assert the production call shape
(e.g.,
systemmust be a list of content blocks;cache_controlpresent). A future refactor breaking the prompt-caching contract
would pass tests but break real caching.
Fix: add one smoke test that imports
anthropic.types(guardedby the lazy-import availability flag) and asserts the kwargs
synthesize()passes match the SDK's expectedmessages.createsignature; OR assert
_FakeMessages.createreceived a list-shapedsystemwithcache_controlon the right blocks.4. Cost-invariant test assertion
No test asserts
sum(per_template[*].cost_usd) == total_cost_usdinthe manifest. A regression in cost aggregation (assign vs increment)
would produce a correct total but wrong per-template breakdown, or
vice versa, undetected.
Fix: add an assertion to
test_synthesize_happy_path_3_samples(or a dedicated test) verifying the per-template costs sum to the
manifest total.
5. API-key check location
The
ANTHROPIC_API_KEYpresence check lives inmain()(CLI pathonly).
synthesize()is a public library function; calling itdirectly with
client=None+ no env var raises a deep, non-obviousSDK error instead of the friendly "ANTHROPIC_API_KEY env var not set"
message.
Fix: move (or duplicate) the env-var check into
synthesize()before the lazy
anthropic.Anthropic()instantiation, OR documentthe requirement clearly in the function docstring.
Context
Filed 2026-05-25 as deferred polish from PR #16 review. Consumer
(
prompt-injection-portfolioLane 2) does not need these before M3;they harden the skill for broader reuse. Good first-issue candidates.