Training loss becomes 0

[Lelouchrx]

I encountered a strange issue while fine-tuning the model. When I freeze the vision encoder and only unfreeze the visual merger for training, the training loss immediately drops to 0.0 and stays there. However, if I set all visual parameters to requires_grad = False, the loss remains normal.

[waynechu1021]

I encountered a strange issue while fine-tuning the model. When I freeze the vision encoder and only unfreeze the visual merger for training, the training loss immediately drops to 0.0 and stays there. However, if I set all visual parameters to requires_grad = False, the loss remains normal.

We haven't encountered this problem before, but what is your training setup?

[Lelouchrx]

Thanks for the follow-up. Here is our training configuration. We are using DeepSpeed ZeRO-2 to fine-tune the Qwen2.5-VL-3B-Instruct model on a single GPU (via CUDA_VISIBLE_DEVICES=0).

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export PYTHONPATH="./:$PYTHONPATH"
export NCCL_P2P_LEVEL=NVL
CUDA_VISIBLE_DEVICES=0 deepspeed --master_port 25420 src/train/train.py
--deepspeed scripts/zero2.json
--dataset_name data/navida_train_data
--model_name_or_path /HDD_POOL/MODELS/Qwen2.5-VL-3B-Instruct
--num_train_epochs 1
--bf16
--torch_dtype bfloat16
--attn_implementation flash_attention_2
--lr_scheduler_type linear
--gradient_checkpointing True
--per_device_train_batch_size 4
--gradient_accumulation_steps 4
--dataloader_num_workers 32
--dataloader_pin_memory
--learning_rate 2.0e-5
--logging_steps 10
--eval_strategy no
--eval_steps 100
--save_strategy no
--save_steps 12000
--output_dir result/navida_qwen2_5_vl_3b_train_on_r2r_rxr
--report_to wandb
--run_name navida_qwen2_5_vl_3b_r2r_rxr \

我在微调模型时遇到了一个奇怪的问题。当我冻结视觉编码器，只解冻训练时的视觉合并，训练损失立刻降到0.0并保持在那里。但是，如果我把所有视觉参数设为requires_grad = False，损失依然正常。

我们以前没遇到过这个问题，但你们的训练设置是怎样的？

[waynechu1021]

Thanks for the follow-up. Here is our training configuration. We are using DeepSpeed ZeRO-2 to fine-tune the Qwen2.5-VL-3B-Instruct model on a single GPU (via CUDA_VISIBLE_DEVICES=0).

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True export PYTHONPATH="./:$PYTHONPATH" export NCCL_P2P_LEVEL=NVL CUDA_VISIBLE_DEVICES=0 deepspeed --master_port 25420 src/train/train.py --deepspeed scripts/zero2.json --dataset_name data/navida_train_data --model_name_or_path /HDD_POOL/MODELS/Qwen2.5-VL-3B-Instruct --num_train_epochs 1 --bf16 --torch_dtype bfloat16 --attn_implementation flash_attention_2 --lr_scheduler_type linear --gradient_checkpointing True --per_device_train_batch_size 4 --gradient_accumulation_steps 4 --dataloader_num_workers 32 --dataloader_pin_memory --learning_rate 2.0e-5 --logging_steps 10 --eval_strategy no --eval_steps 100 --save_strategy no --save_steps 12000 --output_dir result/navida_qwen2_5_vl_3b_train_on_r2r_rxr --report_to wandb --run_name navida_qwen2_5_vl_3b_r2r_rxr \

我在微调模型时遇到了一个奇怪的问题。当我冻结视觉编码器，只解冻训练时的视觉合并，训练损失立刻降到0.0并保持在那里。但是，如果我把所有视觉参数设为requires_grad = False，损失依然正常。

我们以前没遇到过这个问题，但你们的训练设置是怎样的？

try setting gradient_accumulation_steps =16 or larger? or using multi-gpu and zero3 may help