Training data and configs

[PiotrNawrot]

Hey,

Great work! Would it be possible for you to release the training data and configs - I'd like to reproduce your SFT in particular, as a baseline for my research

All the best,
Piotr

[LZhengisme]

Hi Piotr,

Thanks for your interest! We trained using specialized hardware and their customized training stack rather than GPUs, so results might differ slightly when reproduced. Nonetheless,

For the data used in supervised finetuning, we ran a bunch of ablations and found using allenai/tulu-3-sft-mixture alone gave us pretty solid performance. We also added filtered versions (in particular we removed all non-English content and deduplicated data) of OpenCoder-LLM/opc-sft-stage1 and OpenCoder-LLM/opc-sft-stage2 to improve its coding performance. We initially included teknium/OpenHermes-2.5 but after ablations we found the above setting worked pretty well and did not include it in the final SFT data mix.

Below is our training config:

• We swept the learning rate over {1e-5, 2e-5, 3e-5, 5e-5, 5e-6, and 2e-6}; our final checkpoint is at 3e-5 with cosine decay and 300 warmup steps.
• We used AdamW optimizer with betas=(0.9, 0.95) and eps=1e-12; we found much smaller epsilon significantly improved both pre-training and fine-tuning.
• We trained for 2 epochs as we found more epochs lead to somewhat worse performance.
• We used a batch size of 2M bytes after sweeping {1M, 2M} and weight decay of 0.01 on all parameters except RMS norm layers
• We freeze embeddings during fine-tuning. Freezing embeddings or initial several layers generally stabilized training, though this wasn't always consistent. Worth trying if you encounter stability issues :)
• We also include prompt bytes for loss calculation with a loss weight of 0.1, but simply ignoring prompts also yielded similar task perf
• We trained all byte prediction heads during fine-tuning.

Hope this helps a bit. Feel free to reach out if you run into any issues!

Best,
Lin