Attention Is All You Need
Authors: Vaswani et al. (Google Brain/Research)
Year: 2017
Conference: NIPS 2017
Links:
Crux — My Take
The Transformer architecture brought a paradigm shift by replacing recurrence and convolutions with attention mechanisms. Rather than processing sequences step-by-step like RNNs or through sliding windows like CNNs, attention operates on all positions simultaneously — making the architecture fundamentally more parallelizable from the ground up.
This not only improved translation quality but also resulted in much shorter training times compared to legacy recurrent and convolutional approaches. A model that previously took days to train could now reach state-of-the-art performance in as little as 12 hours — better results, at a fraction of the cost.
Why I Picked This
This groundbreaking study showed that attention alone is sufficient for sequence modeling, setting the stage for the entire modern NLP/NLG landscape.