I'm trying to understand the transformer architecture for NLP.
The main issue is regarding the attention weights. The same word can have different attention weights in different sentences, right?
Transformers – How Different Attention Weights Occur for the Same Word in Various Sentences
natural languageneural networkstransformers
Related Solutions
We observe these kind of redundancies in literally all neural network architectures, starting from simple fully-connected networks (see diagram below), where same inputs are mapped to multiple hidden layers. Nothing prohibits the network from ending up with same weights in here as well.
We fight this by random initialization of weights. You usually need to initialize all the weights randomly, unless some special cases where initializing with zeros or other values proved to worked better. The optimization algorithms are deterministic, so there is no reason whatsoever why the same inputs could lead to different outputs if all the initial conditions were the same.
Same seems to be true for the original attention paper, but to convince yourself, you can check also this great "annotated" paper with PyTorch code (or Keras implementation if you prefer) and this blog post. Unless I missed something from the paper and the implementations, the weights are treated the same in each case, so there is not extra measures to prevent redundancy. In fact, if you look at the code in the "annotated Transformer" post, in the MultiHeadedAttention class you can see that all the weights in multi-head attention layer are generated using same kind of nn.Linear layers.
A popular method for such sequence generation tasks is beam search. It keeps a number of K best sequences generated so far as the "output" sequences.
In the original paper different beam sizes was used for different tasks. If we use a beam size K=1, it becomes the greedy method in the blog you mentioned.
Best Answer
Yes, and it is not only the case for Transformer but for nearly any deep learning NLP model. Only when treating natural language data as bag-of-words, the sentence is considered as a sum of independent words in the sentence. This is how, for example, a naive Bayes algorithm would consider the sentence. In many cases it would be enough, this is not a bad model. On another hand, in natural language, the context and order of words matter a lot. Different models differently account for the context. Recurrent neural networks do this by considering the previously observed content (recurrence) together with the current word. Architectures like Transformer look at the whole sentence (assuming a sentence-level model) and the positions of the words in the sentence (position embeddings) and then weight them using the attention weights. If they didn't do this, they would be as "dumb" as naive Bayes.