feat: add model-output corruption effects

[venkatkrish543re]

Description

Adds model-output corruption to the existing ChaosPlugin via a guarded MessageAddedEvent hook, alongside the tool-chaos hooks.

Effects: FormatCorruption (malformed JSON, EmptyResponse), Hallucination (confabulation), Refusal (full refusal), plus composable success-framing.

The MessageAddedEvent callback is guarded to corrupt only final assistant responses (role==assistant, no toolUse blocks present) so destructive effects can't delete toolUse blocks mid-turn and break the agent loop. Includes 12 tests covering each effect and all guard conditions.

Related Issues

• Builds on feat: add chaos testing module for fault injection #224 (tool-level chaos by @ybdarrenwang)

Documentation PR

PR 3827

Type of Change

New feature

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce new warnings.

• I ran hatch run prepare

Checklist

• I have read the CONTRIBUTING document
• I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
• My change is focused and reasonably small; I have split unrelated work into separate PRs
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[poshinchen]

@strandly-the-agent review the changes.

[strandly-the-agent]

TL;DR — no blockers; six 🟡 should-fixes worth a pass before this ships. Independent post-approval review (requested by poshinchen); jjbuck's approval stands and nothing here must gate the merge.

The six inline comments group into three themes:

1. Silent misfires — MalformedJson no-ops on prose-wrapped JSON while logging success; Confabulation reformats the answer it decorates.
2. Contract gaps — the module docstring's structured-output promise is false for 4 of the 5 effects; a pre+post combo validates then half-no-ops; and a reproduced marker race under parallel multi-agent subjects (its tier hangs on Q1 below).
3. A pre-release API window — three public-surface decisions (ChaosEffects/ChaosEffect near-collision, unexported ModelEffectUnion, unreachable apply() methods) that are free now and breaking changes after release.

Everything was verified at head cc98a3b: 147 chaos tests pass, ruff/mypy clean, and every 🟡 has a runnable repro. Full detail collapsed below.

✅ Evidence ledger

Check	Result
Branch/SHA	pr-284 @ cc98a3ba56a2af1bb1592b82a6743ca24a5acbb3 (base merge-base b99dc07d), clean tree
Tests	pytest tests/strands_evals/chaos -q → 147 passed
Lint / types	ruff check clean; mypy src/strands_evals/chaos clean
Marker race (comment 1)	✅ reproduced 5/5 runs — 3 vs 1 model calls across two parallel Graph nodes, direction flips per run; separate plugin instances don't help
Pre-effect + structured output (comment 2)	✅ FullRefusal + structured_output_model → StructuredOutputException, 0 model calls
Post effects on SO run (comment 2)	✅ Confabulation()+SuccessFraming() → output pristine, no log line
MalformedJson prose no-op (comment 4)	✅ prose-wrapped and fenced JSON byte-identical; _malform_text("{") == ""
Confabulation reformat (comment 6)	✅ abbreviation splice + newline collapse reproduced; suggested replacement re-tested against the same inputs
Pre+post combo (comment 3)	✅ constructs cleanly, post effects dropped at hook time
Mutation testing	15 mutations against the suite: 8 killed, 7 survived (appendix)
SDK contracts	verified against installed strands-agents 1.54.0; dep floor >=1.45.0 re-tested in a clean 1.45.0 venv (147 passed)

Artifacts (repros, logs, context pack) uploaded to the artifact store under strands-agents/evals/pr/284/.

❓ Questions

Decision-level (these decide two of the findings' tiers):

1. Is model chaos meant to support multi-agent subjects (Graph/Swarm)? If yes, the marker race (comment 1) is a blocker — silent double-chaos/zero-chaos across parallel nodes. If no, a “single-agent subjects only” docstring line makes it a documented limitation. No chaos+multi-agent example exists in this repo's docs today, which is why it's 🟡 not 🔴.
2. Should a pre-hook model effect cancel every model call in the invocation, or exactly one? Cancelling every call is what turns FullRefusal + structured output into an exception rather than a scoreable refusal. If “just one” is intended, the fix is in before_model_invocation, not the docstring.

Non-blocking:
3. Fixed catalogue or extensible effects? A ModelEffect subclass with a new effect_type is rejected by the closed ModelEffectUnion (union_tag_invalid) — already true of ToolEffect, so pre-existing in kind. If “fixed catalogue, contribute upstream” is the stance, one docstring line on ChaosEffect would stop the exported base class reading as an invitation.
4. README.md:464 / SKILL.md still describe tool-only chaos — one sentence here, or deliberately deferred to the merged harness-sdk#3827?
5. Worth updating the PR body: it names FormatCorruption/Hallucination/Refusal (shipped names are MalformedJson/Confabulation/FullRefusal) and says “12 tests” against ~700 test lines — the approval record currently describes a different API than the diff.

📖 Reading order

1. src/strands_evals/chaos/effects.py — the five new effect classes and ModelEffectUnion; self-contained.
2. src/strands_evals/chaos/plugin.py — module docstring first (it states the contract), then the three new hooks (before_model_invocation, after_model_invocation, after_invocation) and _inject_structured_output_failure; _classify_model_output is the safety core (toolUse/role guards).
3. src/strands_evals/chaos/case.py — the ChaosEffects TypedDict and validators; small.
4. src/strands_evals/chaos/__init__.py exports, then tests/strands_evals/chaos/test_model_chaos.py (one class per behaviour; the scripted-model agent-loop test near the end is the one worth reading closely).
5. AGENTS.md / pyproject.toml are one-liners.

Appendix — non-blocking (17) + pre-existing (2)

Test coverage (the guards these miss are verified correct today — regression risk, not defects):

• ⚪ test_ordinary_tooluse_untouched passes even if the toolUse guard only checks the first content block — the fixture's prose text is a MalformedJson no-op, so the assertion holds whether or not the guard fired. The headline safety invariant is effectively untested; swapping in SuccessFraming() fixes it in one line. The one test item worth fixing before merge.
• ⚪ “SuccessFraming always last” survives applying framing first — the test only asserts startswith(prefix), true in both orders.
• ⚪ selected_tool is None guard (plugin.py:136) untested — an unregistered tool name would hit it.
• ⚪ MalformedJson's JSON-vs-prose split untested (two mutations survive; see comment 4).
• ⚪ test_confabulation_injects_template survives replacing the citation with "xx"; no single-sentence case.
• ⚪ test_effects.py has zero tests for the 5 new effects (AGENTS.md mirrored-file rule); the isinstance(content, str) half of the public apply() surface and the apply(None) guards never execute.
• ⚪ No post-effect test uses a real MessageAddedEvent (all MagicMock) although in-place event-message mutation is the plugin's central trick; the real event is a verified drop-in. dynamic_tools on the helper is dead scaffolding.
• ⚪ Pre-hook effects never run through the agent loop; nothing asserts “zero model calls + one assistant turn” (~6 lines with the existing _ScriptedModel), which would also pin comment 2's behaviour.

API / robustness polish:

• ⚪ Confabulation.apply calls dict(block) before checking it's a dict (effects.py:417), unlike its two siblings — a non-dict block raises out of the hook (contrived input).
• ⚪ _inject_structured_output_failure doesn't honour the “pre effect already produced the turn” rule that _get_post_model_effects enforces — latent, but the gates should read the same.
• ⚪ after_invocation guards a non-dict at the state key; _inject_structured_output_failure doesn't — asymmetry disappears under comment 1's fix.
• ⚪ apply() copies in MalformedJson/Confabulation but mutates the caller's dicts in SuccessFraming — public API should pick one contract.
• ⚪ Unparameterised list annotations (plugin.py:234,263,278, effects.py:361) vs AGENTS.md; the redundant hook ClassVar re-declarations (effects.py:340,395,472) from two resolved threads are still in the tree.
• ⚪ Unknown-category configs now raise raw pydantic extra_forbidden instead of the old message naming the allowed set (model_effect singular is the likeliest typo); ~6-line mode="before" validator restores it. Skipping is defensible.
• ⚪ SuccessFraming prepends its text block before reasoningContent; providers requiring thinking-first could reject (unverified — no network; low reachability).

Docs — worth one docstring line each:

• ⚪ Streaming consumers never see the corruption (deltas are yielded before MessageAddedEvent fires) — a stream_async-based task scores clean text while history is corrupted.
• ⚪ A SessionManager registers on MessageAddedEvent before plugins, so the session deterministically persists the clean message and a resumed run silently loses the chaos. (Not re-opening the event-choice thread — only this consequence.)

Pre-existing (true without this PR — happy to file as issues if wanted):

• Experiment.to_dict/Experiment.load round-trip silently drops case.effects (experiment.py:713,762) — now also affects model effects.
• Closed discriminated unions make the exported ToolEffect (and now ModelEffect) base classes unusable as extension points (question 3).

How this review ran

Staged pipeline at head cc98a3b: routing triage → context build (hook contracts verified against strands-agents 1.54.0 source; 13 invariants catalogued) → four specialist passes → aggregation/suppression → this post. Routed passes: correctness/safety, API bar-raiser, adversarial, test-quality; skipped: issue-alignment (no closing issue), llm-context, docs-accuracy (docs live in the merged harness-sdk#3827). The API and adversarial passes timed out on the deep-analysis tier at 900s and were retried on the standard tier with tightened scope, so their coverage is thinner than the correctness and test-quality passes. Aggregation took 33 raw findings to 6 inline + 5 questions + 17 appendix lines; key repros (marker race, SO behaviours, MalformedJson no-op, Confabulation reformat, pre+post combo) were re-verified first-party rather than taken on trust. The 48 previously resolved review threads were treated as settled and are not re-raised; the marker-race finding is distinct from the resolved shared-invocation_state leak thread (that fix holds — verified on the error path too).

[strandly-the-agent]

🟡 Under parallel multi-agent execution the one-shot structured-output marker is shared state: one agent gets two injected failures, a sibling gets none, and both cases are scored as if chaos applied exactly once.

state = event.invocation_state.setdefault(_CHAOS_STATE_KEY, {})
if state.get(_MALFORMED_OUTPUT_APPLIED):
    return False

Reached from a strands.multiagent.Graph subject with ≥2 parallel agent nodes and MalformedJson — the SDK hands every node the same invocation_state dict, so one node's cleanup clears the marker under a sibling still running. Reproduced 5/5 runs (3 model calls vs 1, direction flips per run); giving each agent its own ChaosPlugin doesn't help.

Suggestion: key the marker by event.agent (present on every hook event) instead of by invocation_state — e.g. a WeakKeyDictionary on the plugin, cleared in after_invocation; the SDK already refuses concurrent invocations of a single Agent, so per-agent is per-invocation. Costs: ~10 lines and per-invocation instance state on the plugin (if it must stay stateless, a per-agent key nested inside the shared dict also works). If multi-agent subjects are out of scope instead, one “single-agent subjects only” docstring line closes this — see Questions in the summary.

[strandly-the-agent]

🟡 This structured-output promise holds only for MalformedJson: a pre-hook effect makes the run raise StructuredOutputException, and the other post-hook effects silently do nothing.

With FullRefusal — this docstring's own example config — plus invoke_async(..., structured_output_model=X), every model call including the SDK's forced-mode retry is cancelled: StructuredOutputException, 0 model calls. With Confabulation/SuccessFraming on the same run, the only assistant message is the structured-output toolUse, so nothing is corrupted and nothing is logged — indistinguishable from a baseline case. Both verified at this head.

Suggested change

	MalformedJson does not corrupt structured-output payloads in the message history.
	Instead it injects a single structured-output parse failure per agent invocation via
	BeforeToolCallEvent.cancel_tool, which tests whether the agent recovers; the SDK's
	corrected attempt passes through unchanged, so a typed caller still receives validated
	structured output. after_model_invocation never touches messages carrying toolUse blocks.
	MalformedJson does not corrupt structured-output payloads in the message history.
	Instead it injects a single structured-output parse failure per agent invocation via
	BeforeToolCallEvent.cancel_tool, which tests whether the agent recovers; the SDK's
	corrected attempt passes through unchanged, so a typed caller still receives validated
	structured output. after_model_invocation never touches messages carrying toolUse blocks.
	Structured-output runs (invoke_async(..., structured_output_model=...)) only support
	MalformedJson: a pre-hook effect (FullRefusal, EmptyResponse) also cancels the SDK's
	forced-mode retry, so the run raises StructuredOutputException, and the other post-hook
	effects never fire because the only assistant message is the structured-output toolUse.

Suggestion: the docstring fix is the cheap floor; a logger.warning at the invocation boundary when post effects were configured but nothing was corrupted would make the no-op loud. Cost: that warning needs per-invocation bookkeeping — the same mechanism as the marker-race comment, so sequence the two.

[ybdarrenwang]

nit

[strandly-the-agent]

🟡 A pre + post model-effect combination validates cleanly and then silently drops the post effects at hook time — the case runs as a plain refusal with no signal that half its config was ignored.

ChaosCase(effects={"model_effects": {"*": [FullRefusal(), SuccessFraming()]}}) constructs without complaint (this validator only counts pre effects), and the plugin then skips every post effect because a pre effect exists. Verified at this head; test_model_chaos.py:104 even blesses [FullRefusal(), MalformedJson()].

Suggested change

	def _validate_pre_model_effects(self) -> None:
	"""At most one pre-hook model effect: pre effects cancel the model call, so only one can win."""
	pre_effects = [e for e in self.model_effects if e.hook == "pre"]
	if len(pre_effects) > 1:
	names = ", ".join(type(e).__name__ for e in pre_effects)
	raise ValueError(
	f"model_effects has {len(pre_effects)} pre-hook effects ({names}) — only 1 is allowed per "
	f"ChaosCase. Pre-hook effects cancel the model call, so only one can take effect. "
	f"Use separate ChaosCase instances to test them independently."
	)
	def _validate_pre_model_effects(self) -> None:
	"""At most one pre-hook model effect, and no post effects alongside it."""
	pre_effects = [e for e in self.model_effects if e.hook == "pre"]
	if len(pre_effects) > 1:
	names = ", ".join(type(e).__name__ for e in pre_effects)
	raise ValueError(
	f"model_effects has {len(pre_effects)} pre-hook effects ({names}) — only 1 is allowed per "
	f"ChaosCase. Pre-hook effects cancel the model call, so only one can take effect. "
	f"Use separate ChaosCase instances to test them independently."
	)
	post_effects = [e for e in self.model_effects if e.hook == "post"]
	if pre_effects and post_effects:
	pre_names = ", ".join(type(e).__name__ for e in pre_effects)
	post_names = ", ".join(type(e).__name__ for e in post_effects)
	raise ValueError(
	f"model_effects combines a pre-hook effect ({pre_names}) with post-hook effects "
	f"({post_names}) — the pre-hook effect produces the turn, so the post-hook effects "
	f"would never run. Use separate ChaosCase instances."
	)

Suggestion costs: forbids a config that cannot work today, and the [FullRefusal(), MalformedJson()] fixture needs splitting; relaxing the rule later is additive, while tightening it after release would break configs.

[ybdarrenwang]

This behavior is expected (when both pre- and post- effects exist, pre wins). In the future we can improve to make this loud instead of silent.

[strandly-the-agent]

🟡 MalformedJson only corrupts text that is entirely a JSON blob, so on a realistic chat answer it changes nothing — while the plugin still logs “applied model output chaos”.

if stripped.startswith("{") or stripped.startswith("["):
    return stripped[: len(stripped) // 2]

Any eval running MalformedJson as a post effect against a chat-style agent hits this: 'Here is the JSON: {"a": 1, "b": 2}' and fenced json code blocks come back byte-identical, yet the success log fires. Separately, _malform_text("{") returns "", leaving {"text": ""} in history — which some providers reject on the next turn.

Suggestion: either truncate from a JSON substring (re.search(r"[{\[]", ...)) so wrapped payloads corrupt as documented, or keep the narrow behaviour and emit the log only when content actually changed (~3 lines). Costs: the substring version changes documented behaviour and wants a test pin either way — the JSON-vs-prose split is currently untested (two mutations survive it).

[ybdarrenwang]

Expected. Can add log in the future.

[strandly-the-agent]

Accepted as expected behaviour (log can follow later) — resolving.

[strandly-the-agent]

🟡 Confabulation splits on sentence boundaries and re-joins with single spaces, so it silently reformats the whole answer — and often splices the citation mid-sentence.

"Mr. Smith went to Washington. He stayed." gets the citation inserted after "Mr." (any e.g./No./U.S. does this); multi-line answers and markdown lists collapse to one line; text with trailing whitespace gets the citation dead last (53/200 seeded samples). Any case configured with Confabulation() whose answer has an abbreviation, a newline, or trailing whitespace hits it — a judge then scores the formatting damage rather than the confabulation, so the eval measures the wrong thing.

Suggested change

	def _confabulate(self, text: str) -> str:
	if not text:
	return text
	template = random.choice(self._CONFABULATION_TEMPLATES)
	sentences = re.split(r"(?<=[.!?])\s+", text)
	if len(sentences) <= 1:
	return template + text
	insert_pos = random.randint(1, len(sentences) - 1)
	sentences.insert(insert_pos, template)
	return " ".join(sentences)
	def _confabulate(self, text: str) -> str:
	if not text:
	return text
	template = random.choice(self._CONFABULATION_TEMPLATES)
	positions = [m.end() for m in re.finditer(r"(?<=[.!?])\s+", text) if m.end() < len(text)]
	if not positions:
	return template + text
	pos = random.choice(positions)
	return text[:pos] + template + text[pos:]

Suggestion costs: verified against the same inputs (separators and newlines preserved, no end-of-text insertion), but splicing after an abbreviation is still possible — a complete fix needs an abbreviation heuristic and is fair as a follow-up. Note the existing test only asserts text != original, so it won't notice this change either way.

[ybdarrenwang]

nit this's anomaly injection anyways

[strandly-the-agent]

Accepted as a nit for anomaly injection — fine to resolve.

[ybdarrenwang]

LGTM

[poshinchen]

@strandly-the-agent review the changes

[strandly-the-agent]

Follow-up at af3bc47 — approving. The delta is the API cleanup from my earlier review plus the test-file reorg ybdarrenwang asked for; both verified, nothing new introduced.

Status of my earlier findings:

• ✅ API surface — fixed: ChaosEffects → ChaosEffectsConfig, ModelEffect/ModelEffectUnion exported, pre effects unified on apply() (mirrors Timeout.apply() returning the cancel text; Protocol + cast gone). No stale references in this repo or the merged harness-sdk docs, so no docs patch is needed.
• ➖ MalformedJson prose no-op / Confabulation reformatting — accepted as expected behaviour per ybdarrenwang; I've replied on both threads and they're fine to resolve (I don't have permission to resolve them myself).
• ⏳ Marker race under parallel Graph nodes, module docstring's structured-output promise, pre + post combo validation — no response yet; all non-blocking, fine as follow-ups if multi-agent subjects or stricter config validation ever matter.

✅ Verified at af3bc47

• pytest tests/strands_evals/chaos -q → 154 passed (was 147); ruff check + ruff format --check clean; mypy src/strands_evals/chaos clean.
• Delta git diff cc98a3b..af3bc47: src ±65 lines, all rename / export / apply() unification. test_model_chaos.py dissolved into test_case.py / test_effects.py / test_plugin.py with every test class preserved (TestStructuredOutputAgentLoop, TestInvocationStateCleanup, TestToolUseMessagesNeverCorrupted, TestGuardRoleFiltering, …) and TestSelectPreModelEffect cut to 4 explicit cases.
• Attacked the fix: _select_pre_model_effect only returns hook == "pre" effects, so effect.apply() with no args can never reach a post effect's apply(None) ValueError path; EmptyResponse.apply() keeps the truthy " " cancel text the SDK requires.