Not able to generate meaningful results on gsm8k_synthetic_cot #4

Activity

1. casper-hansen commented on Jan 7, 2025

   

   casper-hansen

   on Jan 7, 2025

   Owner

   More actions

   Hi @yongzx, I am aware of this issue. I simply implemented Coconut as it was described in the paper and shared this as an early version, hoping others would benefit. There may be some bugs that are not yet discovered in the modeling code
2. casper-hansen commented on Jan 7, 2025

   

   casper-hansen

   on Jan 7, 2025

   Last edited by casper-hansen

   Owner

   More actions

   In particular, this parts right here is what I am most unsure about and is something I worked a lot on modifying.

   1. The slicing of the last hidden state. Why not feed in the full hidden state?
      

      OpenCoconut/opencoconut/models/qwen2.py

      Line 45 in 5519bf4

      inputs_embeds = outputs.last_hidden_state[:, -1:, :]

   2. Handling caching/masking in with batch size > 1 (I think this could be buggy)

   • OpenCoconut/opencoconut/models/qwen2.py

     Lines 46 to 57 in 5519bf4

     	attention_mask = torch.cat(
     	(
     	attention_mask,
     	torch.ones(
     	(inputs_embeds.shape[0], 1),
     	dtype=attention_mask.dtype,
     	device=attention_mask.device,
     	),
     	),
     	dim=1,
     	)
     	past_key_values = outputs.past_key_values

   • OpenCoconut/opencoconut/models/qwen2.py

     Lines 257 to 290 in 5519bf4

     	for b in range(input_ids.shape[0]):
     	insert_idx = insert_indices[b]
     	new_attention_mask.append(
     	torch.cat(
     	(
     	attention_mask[b, :insert_idx],
     	torch.ones(
     	num_thoughts - 1,
     	dtype=attention_mask.dtype,
     	device=attention_mask.device,
     	),
     	attention_mask[b, insert_idx:],
     	)
     	)
     	)
     	new_labels.append(
     	torch.cat(
     	(
     	labels[b, :insert_idx],
     	torch.full(
     	(num_thoughts - 1,),
     	-100,
     	dtype=labels.dtype,
     	device=labels.device,
     	),
     	labels[b, insert_idx:],
     	)
     	)
     	)
     	# Stack the attention masks and labels along the batch dimension
     	attention_mask = torch.stack(new_attention_mask, dim=0)
     	labels = torch.stack(new_labels, dim=0)

   👍React with 👍1Reacted by thelongestusernameofall
3. yongzx commented on Jan 7, 2025

   

   yongzx

   on Jan 7, 2025

   Author

   More actions

   Gotcha! thank you for being transparent – I will also look into this
4. thelongestusernameofall commented on Jan 15, 2025

   

   thelongestusernameofall

   on Jan 15, 2025

   More actions

   In particular, this parts right here is what I am most unsure about and is something I worked a lot on modifying.

   1. The slicing of the last hidden state. Why not feed in the full hidden state?
      

      OpenCoconut/opencoconut/models/qwen2.py

      Line 45 in 5519bf4

      inputs_embeds = outputs.last_hidden_state[:, -1:, :]

   2. Handling caching/masking in with batch size > 1 (I think this could be buggy)

   • OpenCoconut/opencoconut/models/qwen2.py

     Lines 46 to 57 in 5519bf4

     	attention_mask = torch.cat(
     	(
     	attention_mask,
     	torch.ones(
     	(inputs_embeds.shape[0], 1),
     	dtype=attention_mask.dtype,
     	device=attention_mask.device,
     	),
     	),
     	dim=1,
     	)
     	past_key_values = outputs.past_key_values

   • OpenCoconut/opencoconut/models/qwen2.py

     Lines 257 to 290 in 5519bf4

     	for b in range(input_ids.shape[0]):
     	insert_idx = insert_indices[b]
     	new_attention_mask.append(
     	torch.cat(
     	(
     	attention_mask[b, :insert_idx],
     	torch.ones(
     	num_thoughts - 1,
     	dtype=attention_mask.dtype,
     	device=attention_mask.device,
     	),
     	attention_mask[b, insert_idx:],
     	)
     	)
     	)
     	new_labels.append(
     	torch.cat(
     	(
     	labels[b, :insert_idx],
     	torch.full(
     	(num_thoughts - 1,),
     	-100,
     	dtype=labels.dtype,
     	device=labels.device,
     	),
     	labels[b, insert_idx:],
     	)
     	)
     	)
     	# Stack the attention masks and labels along the batch dimension
     	attention_mask = torch.stack(new_attention_mask, dim=0)
     	labels = torch.stack(new_labels, dim=0)

   the same confusion。
   Maybe we can modify and try.