About the Evaluation Results

[zhongyu0xiao]

I’m very sorry to bother you, but I’ve encountered an issue. As mentioned in your paper regarding the HumanEval results on Fast-dLLM, I tested the open-source Fast-dLLM code under the same configuration described in the paper (L = 512, block = 32). However, my evaluation result is 56.7, whereas the result reported in your paper is 48.78.

I would like to understand the source of this discrepancy. Could you share your evaluation code or provide guidance on how you obtained the reported results? I would greatly appreciate your help, and I’m very much looking forward to your reply.

Configuration details are as follows:

task=humaneval
length=512
block_length=32
steps=$((length / block_length))
model="/root/autodl-tmp/model/Dream-instruct"

# prefix cache+parallel

accelerate launch eval.py --model dream 
--model_args pretrained=${model},max_new_tokens=${length},diffusion_steps=${steps},add_bos_token=true,alg=confidence_threshold,threshold=0.9,use_cache=true,show_speed=True,escape_until=true 
--tasks ${task} 
--batch_size 1 
--output_path evals_results/cache_parallel/humaneval-ns0-${length} --log_samples 
--confirm_run_unsafe_code

The experiment was conducted on an H800, with library versions consistent with Fast-dLLM:
CUDA 12.2, torch==2.5.1+cu124, transformers==4.49.0.

[INV-WZQ]

@zhongyu0xiao

Thank you for sharing your experimental results. I am not entirely sure where the issue arises in your setting, but I can share the configuration I used. For evaluation, I followed the setup provided in Dream. Our setting: torch==2.6.0+cu124, transformers==4.55.0. The inference configuration of Fast-dLLM is shown below (you add the following running code in code for evaluation):

You can try this setting. If you have any further questions or need additional assistance, please feel free to reach out!

Thank you!

[zhongyu0xiao]

@INV-WZQ Thank you, I now understand that you ran the fast-dllm tests based on Dream’s open-source implementation. However, for fast-dllm, since it has publicly released its own evaluation code, my experiments were conducted using that version. There are some differences between the evaluation code released by fast-dllm and that of Dream, and I’m not sure whether you have looked into those.

Following the configuration described in your supplementary materials, I set BS=1, l=512, block_size=32, and T = l / block_size = 16 in the fast-dllm evaluation script and obtained my results. I’m also quite puzzled by the discrepancy, so I plan to run tests on Dream as well.