Possible false negatives in disallowed-model judgments for teacher / reasoning-trace distillation

[Picard1701ent]

Hi, thanks for releasing the trajectories and judgments.

I may be misunderstanding the latest policy, so I would appreciate clarification. My understanding is that the current rules disallow importing capabilities from stronger external models, including:

• Locally served teacher models
• Hosted model APIs
• Packaged reasoning traces generated by stronger models

Self-generation and filtering with the assigned model are still allowed, but external-model distillation should be flagged and the run scored as the base model.

However, I found two trajectories that appear inconsistent with this rule.

1. HealthBench / Qwen3-4B-Base

Run:

https://huggingface.co/datasets/aisa-group/PostTrainBench-Trajectories/tree/main/claude_non_api_claude-opus-5_10h_run2/healthbench_Qwen_Qwen3-4B-Base_17419574

The run explicitly states that it locally served Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 as a teacher. That teacher generated:

1. The synthetic conversations
2. The ideal assistant answers
3. Per-example rubrics
4. Self-refined answers

The resulting examples were then filtered and used to fine-tune the assigned Qwen/Qwen3-4B-Base.

The run itself describes the method as “teacher-distilled synthetic data”:

https://huggingface.co/datasets/aisa-group/PostTrainBench-Trajectories/blob/main/claude_non_api_claude-opus-5_10h_run2/healthbench_Qwen_Qwen3-4B-Base_17419574/task/README.md

Despite this, its judgment says disallowed_model: false:

https://huggingface.co/datasets/aisa-group/PostTrainBench-Trajectories/blob/main/claude_non_api_claude-opus-5_10h_run2/healthbench_Qwen_Qwen3-4B-Base_17419574/judgement_gpt5_4.json

The justification appears to check only whether the final model is architecturally derived from the allowed 4B base model. It does not seem to consider who generated the SFT targets.

2. AIME 2025 / Qwen3-4B-Base

Run:

https://huggingface.co/datasets/aisa-group/PostTrainBench-Trajectories/tree/main/claude_non_api_claude-opus-5_10h_run2/aime2025_Qwen_Qwen3-4B-Base_17417418

This run reports three sequential full fine-tunes. Its first stage is explicitly called “broad distillation” and uses approximately 57M tokens of DeepSeek-R1 reasoning traces from:

• open-r1/Mixture-of-Thoughts
• Historical AIME DeepSeek-R1 trace datasets

The second stage additionally uses nvidia/OpenMathReasoning. Its preprocessing code specifically selects rows where:

generation_model == "DeepSeek-R1"

The final submitted model is descended through:

Qwen3-4B-Base → sft1 → sft2 → sft3 → final_model

See:

https://huggingface.co/datasets/aisa-group/PostTrainBench-Trajectories/blob/main/claude_non_api_claude-opus-5_10h_run2/aime2025_Qwen_Qwen3-4B-Base_17417418/task/README.md

The upstream Mixture-of-Thoughts dataset also explicitly describes itself as reasoning traces distilled from DeepSeek-R1:

https://huggingface.co/datasets/open-r1/Mixture-of-Thoughts

Nevertheless, this run is also marked disallowed_model: false:

https://huggingface.co/datasets/aisa-group/PostTrainBench-Trajectories/blob/main/claude_non_api_claude-opus-5_10h_run2/aime2025_Qwen_Qwen3-4B-Base_17417418/judgement_gpt5_4.json

Again, the justification seems limited to final-model architecture and checkpoint lineage rather than training-data provenance.

Questions

1. Under the current rules, should these two runs be classified as external-model distillation?
2. Are public datasets containing DeepSeek-R1, QwQ-32B, or other stronger-model reasoning traces considered disallowed?
3. Does serving a teacher locally instead of calling a hosted API change the ruling?
4. Will the existing judgments be rerun with a provenance-aware check covering training-data generation rather than only final-model identity?
5. If flagged, should the reported improved scores be replaced by the corresponding base-model scores?

This also makes me concerned that similar false negatives may exist in other recently added runs, particularly the Fable runs. I am not claiming that those runs necessarily violate the policy, but I would appreciate access to—or pointers to—the complete Fable trajectories and updated judgments so the same provenance check can be applied consistently.

In particular:

• claude_non_api_max_claude-fable-5_1m__10h_run1
• claude_non_api_max_claude-fable-5_1m__10h_run2

Could the maintainers review these and ideally rerun the updated disallowed-model judge across all trajectories?

Thanks!

[rank-and-file]

Hey,

the current rules allow

• Locally served teacher models
• Packaged reasoning traces generated by stronger models (e.g. from huggingface datasets)

And disallow

• Hosted model APIs

So the described actions in the traces above are allowed :)

Hope that clarifies your comment!