Skip to content

/dataset-synthesize: post-merge polish (cost-table staleness, non-text block warning, mock fidelity, cost-invariant test, API-key check location) #21

Description

@brandon-behring

Follow-up hardening for /dataset-synthesize (shipped in PR #16,
closes #1). These are NICE-TO-HAVE items surfaced during the PR #16
code review but deferred to keep the initial PR focused. None are
blocking; all are low-risk polish.

1. PRICING_PER_MTOK staleness guard

scripts/dataset_synthesize.py hardcodes per-model pricing. If
Anthropic changes prices, --bail-at-cost math is silently wrong
(user thinks they capped at $80; actual differs).
Fix: add a comment with the "pricing verified as of YYYY-MM-DD"
date + the Anthropic pricing page URL next to the table. Optionally
a soft self-check / staleness note in --help.

2. _extract_text silently drops non-text content blocks

If the API returns tool_use / image / other non-text blocks,
_extract_text returns "" and the sample row gets content: ""
with no warning. Silent quality degradation.
Fix: emit a print("warning: ...", file=sys.stderr) when a
response yields zero text blocks, OR record a per-row flag in the
JSONL so downstream consumers can filter.

3. Mock fidelity — _FakeClient doesn't enforce real SDK shape

tests/test_dataset_synthesize.py uses hand-rolled _FakeClient /
_FakeMessage / _FakeUsage doubles (per docs/conventions/test-style.md,
correct pattern). But they don't assert the production call shape
(e.g., system must be a list of content blocks; cache_control
present). A future refactor breaking the prompt-caching contract
would pass tests but break real caching.
Fix: add one smoke test that imports anthropic.types (guarded
by the lazy-import availability flag) and asserts the kwargs
synthesize() passes match the SDK's expected messages.create
signature; OR assert _FakeMessages.create received a list-shaped
system with cache_control on the right blocks.

4. Cost-invariant test assertion

No test asserts sum(per_template[*].cost_usd) == total_cost_usd in
the manifest. A regression in cost aggregation (assign vs increment)
would produce a correct total but wrong per-template breakdown, or
vice versa, undetected.
Fix: add an assertion to test_synthesize_happy_path_3_samples
(or a dedicated test) verifying the per-template costs sum to the
manifest total.

5. API-key check location

The ANTHROPIC_API_KEY presence check lives in main() (CLI path
only). synthesize() is a public library function; calling it
directly with client=None + no env var raises a deep, non-obvious
SDK error instead of the friendly "ANTHROPIC_API_KEY env var not set"
message.
Fix: move (or duplicate) the env-var check into synthesize()
before the lazy anthropic.Anthropic() instantiation, OR document
the requirement clearly in the function docstring.

Context

Filed 2026-05-25 as deferred polish from PR #16 review. Consumer
(prompt-injection-portfolio Lane 2) does not need these before M3;
they harden the skill for broader reuse. Good first-issue candidates.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Nice-to-have / low priorityenhancementNew feature or requesttrackedCross-repo work-tracker aggregation label

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions