Attention Is All You Need¶

Authors: Ashish Vaswani et al.
Conference: NIPS 2017

Abstract¶

Proposes Transformer - a new network architecture based solely on attention
Dispenses with recurrence and convolutions entirely
More parallelizable and requires significantly less training time
Achieves state-of-the-art results on machine translation tasks:
28.4 BLEU on WMT 2014 English-to-German
41.8 BLEU on WMT 2014 English-to-French
Generalizes well to other tasks like English constituency parsing

Background: Sequence Modeling Challenges¶

Traditional Approaches¶

Recurrent models (LSTM, GRU) process input sequentially
Convolutional models have limited receptive fields
Both struggle with:
Long-range dependencies
Parallelization
Computational efficiency for long sequences

The Need for a New Architecture¶

Key Limitations of Existing Models¶

Recurrent networks: Inherently sequential computation
Convolutional networks: Limited long-range connectivity
Hybrid models: Still rely on recurrence/convolution as primary components

Attention Mechanisms¶

Already used as auxiliary component in sequence models
Allows modeling dependencies without regard to position distance
Not yet used as the primary architectural component

The Transformer: Model Architecture¶

Encoder-decoder structure using stacked self-attention and feed-forward layers

Encoder Architecture¶

Stack of 6 identical layers
Each layer contains two sub-layers:
Multi-head self-attention mechanism
Position-wise fully connected feed-forward network
Residual connections around each sub-layer
Layer normalization: LayerNorm(x + Sublayer(x))
Output dimension: d_model = 512

Decoder Architecture¶

Stack of 6 identical layers
Each layer contains three sub-layers:
Masked multi-head self-attention
Multi-head attention over encoder output
Position-wise fully connected feed-forward network
Residual connections and layer normalization
Masking prevents attending to future positions

Attention Mechanism Basics¶

Definition¶

An attention function maps a query and key-value pairs to an output: - Query (Q), Keys (K), Values (V) are vectors - Output is weighted sum of values - Weights determined by compatibility of query with corresponding key

Two Common Types¶

Additive attention (feed-forward network)
Dot-product attention (scaled in Transformer)

Scaled Dot-Product Attention¶

\[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\]

Computes dot products of query with all keys
Scales by \(\sqrt{d_k}\) to prevent gradient vanishing
Applies softmax to get weights over values
More efficient than additive attention (matrix operations)