[experiments] Daily Experiment Report — 2026-10-10

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-10

52 experiments analysed across 49 workflows. 26 are 🟢 READY on sample size, but none reached a PROMOTE or REJECT decision: 38 are EXTEND and 14 are INCONCLUSIVE. Run-level outcome charts were skipped; the core decision is the canonical result.

⚡ Quick Stats

Metric	Value
Active experiments	52 (49 workflows)
Ready for analysis	26 (🟡 COLLECTING: 26)
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (no workflow has 2 experiments except the smoke Copilot variants, with no PROMOTE to hold)

🔎 Key findings

• insufficient_observations (24 experiments) blocks every READY experiment with enough samples, for example testqualitysentinel/model_size (174 vs 176) and smokecopilot (183 vs 183). No outcome observations feed the decision layer yet.
• guardrail_unsupported (3 experiments): cicoach/prompt_style, dailysecurityredteam/reasoning_depth and dailyfact/reasoning_depth are READY, but run_success_rate and empty_output_rate are unsupported native metrics.
• unsupported_multi_variant (14 experiments): any experiment with 3 or more variants is INCONCLUSIVE. This includes awfailureinvestigator/tone_variant (221/204/198), deepreport/output_format (4 variants) and dailycompilerquality/output_format. The three model_size experiments also record stray agent and small-agent variants (dailydocupdater, dailycavemanoptimizer, dailydochealer).
• insufficient_samples (9 experiments) are still collecting, for example dailyarchitecturediagram (12/8) and dependabotcampaign (6/6). copilotprnlpanalysis has 0 usable samples.

📊 Full experiments table

Workflow	Experiment	Readiness	Core decision	Reason code
agentperformanceanalyzer	prompt_compression	READY	EXTEND	insufficient_observations
agentpersonaexplorer	sub_agent_strategy	READY	EXTEND	insufficient_observations
architectureguardian	sub_agent_strategy	COLLECTING	EXTEND	insufficient_samples
auditworkflows	audit_decomposition	COLLECTING	EXTEND	insufficient_samples
blogauditor	prompt_style	COLLECTING	EXTEND	insufficient_samples
breakingchangechecker	tone_variant	READY	EXTEND	insufficient_observations
cicoach	prompt_style	READY	EXTEND	guardrail_unsupported
copilotprnlpanalysis	nlp_prompt_style	COLLECTING	EXTEND	insufficient_samples
dailyagentrxtraceoptimizer	sub_agent_strategy	READY	EXTEND	insufficient_observations
dailyarchitecturediagram	detail_level	COLLECTING	EXTEND	insufficient_samples
dailyastrostylelitemarkdownspellcheck	prompt_style	READY	EXTEND	insufficient_observations
dailycachestrategyanalyzer	model_size	COLLECTING	EXTEND	insufficient_samples
dailycommunityattribution	prompt_style	READY	EXTEND	insufficient_observations
dailyfact	reasoning_depth	READY	EXTEND	guardrail_unsupported
dailyfunctionnamer	model_size	COLLECTING	EXTEND	insufficient_observations
dailynews	prompt_style	READY	EXTEND	insufficient_observations
dailyrenderingscriptsverifier	remove_redundant_context_v1	COLLECTING	EXTEND	insufficient_observations
dailysafeoutputoptimizer	log_fetch_strategy	COLLECTING	EXTEND	insufficient_observations
dailysecurityredteam	reasoning_depth	READY	EXTEND	guardrail_unsupported
dataflowprdiscussiondataset	caveman_mode	COLLECTING	EXTEND	insufficient_samples
dependabotcampaign	summary_detail	COLLECTING	EXTEND	insufficient_samples
gpclean	tool_verbosity	READY	EXTEND	insufficient_observations
issuearborist	prompt_style	READY	EXTEND	insufficient_observations
prsouschef	remove_redundant_context_v1	COLLECTING	EXTEND	insufficient_observations
smokeantigravity	sub_agent_strategy	READY	EXTEND	insufficient_observations
smokecopilot	caveman	READY	EXTEND	insufficient_observations
smokecopilot	subagent_model	READY	EXTEND	insufficient_observations
smokecopilotaoaiapikey	caveman	READY	EXTEND	insufficient_observations
smokecopilotaoaiapikey	subagent_model	READY	EXTEND	insufficient_observations
smokecopilotaoaientra	caveman	READY	EXTEND	insufficient_observations
smokecopilotaoaientra	subagent_model	READY	EXTEND	insufficient_observations
smokegemini	sub_agent_strategy	READY	EXTEND	insufficient_observations
smokepi	sub_agent_decomposition	READY	EXTEND	insufficient_observations
smokeproject	prompt_style_test	COLLECTING	EXTEND	insufficient_observations
smoketemporaryid	sub_agent_strategy	COLLECTING	EXTEND	insufficient_observations
testqualitysentinel	model_size	READY	EXTEND	insufficient_observations
typist	tone_style	READY	EXTEND	insufficient_observations
weeklyblogpostwriter	prefetch_strategy	COLLECTING	EXTEND	insufficient_samples
awfailureinvestigator	tone_variant	READY	INCONCLUSIVE	unsupported_multi_variant
copilotagentanalysis	output_format	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailycavemanoptimizer	model_size	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailycodemetrics	output_format	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailycompilerquality	output_format	READY	INCONCLUSIVE	unsupported_multi_variant
dailydochealer	model_size	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailydocupdater	model_size	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailyissuesreport	output_format	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailysemgrepscan	semgrep_output_format	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailysubagentoptimizer	timeout_setting	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
deepreport	output_format	READY	INCONCLUSIVE	unsupported_multi_variant
dependabotgochecker	prompt_style	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
plan	reasoning_depth	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
smokecopilotsubagents	sub_agent_strategy	COLLECTING	INCONCLUSIVE	unsupported_multi_variant

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. Add per-run outcome observations (run success, output length) to the state so the 24 insufficient_observations experiments can resolve. Start with the READY two-variant ones: testqualitysentinel, issuearborist, dailynews and the smokecopilot* family.
2. Support native run_success_rate and empty_output_rate guardrails to unblock cicoach, dailysecurityredteam and dailyfact.
3. Reduce awfailureinvestigator, deepreport and dailycompilerquality to two variants, or add multi-variant support to core analysis. Clean up the stray agent and small-agent variant names in the model_size experiments.

Top eval actions

1. No eval scores were available in this run. Define graders for output-format experiments (ste, prose, structured) that emit verbosity and readability observations.
2. Add a grader for empty or failed outputs, mapped to the empty_output_rate guardrail.
3. Add accuracy-oriented evals for the reasoning_depth and model_size experiments, since they have no quality metric.

Decision-pipeline gaps for next PR

• Native metrics lacking observations: run_success_rate, empty_output_rate, plus primary metrics in the 24 affected experiments.
• Make factorial interaction diagnostics (for example smokecopilot subagent_model × caveman) a normalized core signal.
• Support K≥3 variants in decision logic (Bonferroni-corrected pairwise comparisons against control).

Descriptive window: last 30 runs · No tracking issues were configured, so no issue comments or labels were posted.
Run: 38035975287

Generated by 🧪 daily-experiment-report · copilot · auto · 21.7 AIC · ⌖ 0.93 AIC · ⊞ 16.3K · ◷

• expires on Oct 13, 2026, 12:01 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #67678.