natural languageneural networks
I'm currently trying to implement a PyTorch version of the Transformer and had a question.
I've noticed that many implementations apply a mask not just to the decoder but also to the encoder. The official TensorFlow tutorial for the Transformer also states that the Transformer uses something called "MultiHead Attention (with padding masking)."
I'm just confused, why are masks applied to the padding in the encoder sequence?
This is answered in the Attention is All You Need paper by Vaswani et al (see also recording of the talk by one of the co-authors, and those three blogs: here, here, and here).
- How is it possible to mask out illegal connections in decoder multi-head attention?
This is pretty simple. Attention can be defined as
$$ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\Big(\frac{QK^T}{\sqrt{d_k}}\Big)V $$
where $Q$ are queries, $K$ are keys, $V$ are values and $\sqrt{d_k}$ is the scaling constant equal to the square root of the dimension of the keys. The role of the product $QK^T$ is to calculate the similarity matrix between words in $Q$ and $K$ (where each word is a row encoded using embeddings). In the encoder, each $Q,K,V$, comes from the same document. In the decoder, $Q$ comes from target document, while $K,V$ come from source document.
In the Transformer network (and similar ones), there is no direct mechanism that records time dependence. It is recorded indirectly in the embeddings (by summing word embeddings and position embeddings), but at a cost of leaking "future" values when making predictions. Notice that in $QK^T$ we look at similarities between at each word in $Q$ with each word in $K$. To prevent the future leak, we use masking. This is done by performing a pointwise product of $QK^T$ and the upper triangular matrix of ones (illustrated below, image source).
This it zeroes-out the similarities between words and the words that appear after the source words ("in the future"), preventing predictions from depending on knowing the answer before they predict it. Since we remove such information, it cannot be used by the model, and we guarantee that only similarity to the preceding words is considered.
- Is it alright to set some arbitrary max_length for layer normalization?
In the paper, all the inputs and outputs have fixed size of $d_\text{model}$, if this is what you ask. However I can't see why this would be a problem, since what the normalization does is it makes the features have same mean and standard deviation between the layers. So if something was relatively large locally, will be mapped to what is considered large globally. See the Layer normalization paper by Ba et al for details. Moreover, this is applied per feature, so excess zeros have no impact.
I think the paper you quote is just wrong.
Luong only generalizes Bahdanau's equations by replacing a single-layer MLP by a general function score (and shows that dot-product can work equally well as the MLP), but it still scores "similarity" of decoder state and exactly one encoder state.
Best Answer
I hadn't realized this question was unanswered. If I were to attempt to answer my own question, we apply masks to the source data because after the data passes through the Encoder sublayer, there are values for the padding sequences. We don't need nor want the model to attend to these padding sequences, and so we mask them out.
It's slightly different from masking in the decoder in the sense that masking in the decoder takes an additional step of having a "no peeking" mechanism so that our model can't look at future tokens.