Attention Is All You Need
NeurIPS 2017
Proposes the Transformer — a sequence model built entirely on self-attention, eliminating recurrence and convolution. The architecture achieves state-of-the-art machine translation quality while being dramatically faster to train in parallel. It became the architectural foundation for every modern large language model.
- Multi-head self-attention allows the model to jointly attend to information from different representation subspaces
- Positional encoding injects sequence-order information without recurrence, enabling full parallelism
- Encoder-decoder architecture with cross-attention generalises to any seq2seq task
- Achieves 28.4 BLEU on WMT 2014 English-to-German, surpassing all prior ensembles
Transformer
Self-Attention
NLP
Sequence Modeling
Architecture
Deep Learning