Question about transformer decoder

[CoinCheung]

Hi,

I am trying to learn about the code, and I find the following line:

MaskFormer/mask_former/modeling/transformer/transformer.py

Line 70 in da3e60d

tgt = torch.zeros_like(query_embed)

The input tgt of the decoder is all zeros, and I see the all-zeros-tensor is used as input in the decoder layer:

MaskFormer/mask_former/modeling/transformer/transformer.py

Line 272 in da3e60d

q = k = self.with_pos_embed(tgt, query_pos)

Here tgt is all-zeros and the query_pos is a learnable embedding, which causes q and k to be non-zero tensor (same tensor in value as query_pos, but the tgt is still all-zeros(used as v). According to the computation rule of qkv attention, if v is all-zeros, the output of qkv would be all-zeros. Thus the self-attention module does not contribute to the model. Am I correct on this?

[bowenc0221]

This is correct only for the first self-attention layer. tgt is no longer zero vector after cross-attention.

[CoinCheung]

Thanks for replying !!! There is another part of code that I cannot understand:

MaskFormer/mask_former/modeling/transformer/transformer.py

Line 75 in da3e60d

return hs.transpose(1, 2), memory.permute(1, 2, 0).view(bs, c, h, w)

If we use default settings of batch_first=False for nn.MultiheadAttention, the above hs tensor should be LNE, where L is sequence length(num of queries here), N is batchsize and E is feature dimension. After the transpose(1,2), hs will become LEN. The batchsize will be the last dimension.
However, according to this line:

MaskFormer/mask_former/modeling/transformer/transformer_predictor.py

Line 130 in da3e60d

outputs_seg_masks = torch.einsum("lbqc,bchw->lbqhw", mask_embed, mask_features)

The output hs should be a 4d tensor ? Would you please tell me what did I miss here ?