Attention is all you need: The technology that brought ChatGPT to life

Natural Language Processing (NLP)

Natural language processing, or NLP, is the discipline within artificial intelligence dedicated to teaching computers to understand, interpret, and manipulate human language.

Historically, machines operated strictly under structured programming languages; NLP can be understood as the bridge that allows them to process the world in a manner similar to how humans do. In particular, neural networks and deep learning have driven substantial breakthroughs by enabling the modeling of dense vector representations, capturing the underlying context and semantics of human language.

However, neural networks do not understand raw text, only numbers. Before processing any sentence, a process called tokenization is carried out: text is divided into fragments called tokens, which can be whole words, subwords, or letters. Each token is transformed into a vector of numbers processable by a computational algorithm.

The State of the Art Before 2017

Prior to 2017, the state of the art in NLP relied on architectures designed to process information in a strictly sequential manner. The standard was recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) networks. These architectures ingest text token by token, typically from left to right, maintaining a "hidden state," a sort of short-term memory that updates with each new word, combining the current input with the context of preceding words.

The critical issue with RNNs is that, across long texts, this sequential memory dilutes: upon reaching the end of a paragraph, the network "forgets" the initial context due to a mathematical limitation in its training known as the vanishing gradient. Furthermore, because reading the next token depends on the previous one, parallelizing these networks proves difficult, rendering their scalability limited.

In parallel, computer vision architectures were adapted, such as one-dimensional convolutional neural networks (CNNs) applied to text. CNNs operate like a sliding window that reads blocks of tokens simultaneously. In essence, they were capable of resolving the scalability problem, since they allowed data to be processed in parallel, and they were excellent at detecting local patterns, such as in short phrases. However, CNNs fail when attempting to capture long-range dependencies: they lack a global view; to relate a word at the beginning of the page to one at the end, it is necessary to stack a large number of convolutional layers.

Comparative Table: RNNs vs. CNNs:

Feature	RNNs	CNNs
Processing	Sequential	Parallel
Context	Global (with amnesia)	Local (limited to the filter)
Speed	Slow	Fast
Distant relationships	Difficult (gradient and context)	Require stacking too many layers

In concrete terms, a structural limit existed: an architecture was required that could process massive amounts of data in parallel, while simultaneously being capable of directly modeling dependencies between any pair of words in a sequence.

In 2017, Vaswani et al., engineers and scientists at Google, introduced the Transformer architecture in their foundational paper "Attention Is All You Need" [1], marking a new era in NLP.

Tokenization, Embeddings, and the Context Problem

The first step is understanding how a neural network "reads." In reality, it does not read letters, but instead breaks the text down into small pieces called tokens. A token is a fragment of text that the machine treats as a single unit; for our explanation, let us imagine these tokens as words, although in real models they are sometimes syllables or word pieces.

Each token is then converted into an embedding: a list of numbers (a vector) that acts as the coordinates of that word within a gigantic "map of meanings." On that map, similar words live in nearby regions, and the interesting part is that directions also carry meaning: if from the position of the word man you walk toward the word woman, and then repeat the exact same displacement starting from king, you arrive at queen. The map knows, so to speak, what relationship binds each pair of words together.

However, this map has a flaw that reveals the true problem the Transformer came to solve. Classic embeddings are fixed. An example of why this is a problem is the word taco (or "traffic jam / shoe heel / taco" in Spanish); it occupies a single point on the map, regardless of whether the sentence refers to a traffic jam, a shoe heel, food, or a pool cue. All four meanings share the exact same coordinates.

Think about it with this sentence: "Because she was stuck in the highway taco (traffic jam), she took off her taco shoes (high heels) to rest her feet." A human can resolve the ambiguities in a fraction of a second, because we know the context. The model, on the other hand, saw the exact same point on the map twice, identical in the word stuck and in the word shoes. The embedding, as it was generated, was unaware of context, and therefore could not differentiate between them.

Consequently, the great insight of the Transformer is that a word's embedding should not remain frozen. Each word starts from its generic position, but as the network processes the sentence, its vector shifts progressively toward the region of the map corresponding to that context: the taco from the highway migrates toward traffic jam; the taco from the shoes, toward footwear. It no longer encodes just a word, but its semantic content.

And who decides how much to shift each point and in which direction? That is precisely the attention mechanism [2]: the model computes a "nudge" (which we will call $\Delta E$) for each word, summing influences from surrounding words, assigning more weight to relevant ones and less to irrelevant ones.

The Attention Mechanism (Single-Head → Multi-Head)

:::info Up to this point, we have lied a little: The vector that enters the Transformer is not just the semantic embedding; it also encodes the position of the word in the sentence. We will explain that in the architecture chapter. Moreover, what follows is not exactly what an already trained model does, whose internal calculations are a black box difficult to interpret. It is a step-by-step reconstruction to understand what each component does and for what purpose. :::

Let us begin by looking at the complete formula for attention, just as it appears in the original paper [1], and then build it piece by piece [3]:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right) \cdot V$$

For the following explanation, consider the sentence "The wooden bench was broken" ("El banco de madera estaba roto"). We will use it as a running thread: each word will formulate its question (Query), offer its answer (Key), and at the end of the section we will assemble the full calculation step by step.

Query: The Question Each Word Asks

Imagine that each word, upon reading the sentence, asks itself: "Are there descriptors for me? Is anyone here describing or complementing me?", and that this question is encoded into its own vector, called a Query. And how is it generated? By multiplying the embedding by a table of adjustable numbers called a weight matrix, denoted $W_Q$. If the embedding of "bench" is $E_i$, then $E_i \xrightarrow{W_Q} Q_i$.

The Query is a vector smaller than the embedding (of dimension $d_k$, smaller than that of the original embedding). This reduction is deliberate: it forces each word to condense its question into a few coordinates, without dragging along its entire meaning.

Key: The Answer Each Word Offers

Consequently, if each word asks a question, each word must also be able to answer. For that, another weight matrix exists, $W_K$ (for key): $E_i \xrightarrow{W_K} K_i$. The Key functions like a label on a library shelf: it is not the complete content of the book, but rather what says "I can provide this to you." When the Query of "bench" asks "Are there descriptors for me?", the Key of "wooden" essentially responds: "Yes: material, wood."

The Dot Product Q, K: Measuring Alignment

Now questions must be compared with answers, and here enters the dot product, an operation that measures how "aligned" two arrows are in space: if they point in the same direction, the result is a large positive number; if they are perpendicular, it is zero; if they point in opposite directions, it is negative. Therefore, calculating $Q \cdot K^T$ produces a match table: the cell at row i and column j indicates how much the Key of word j answers the question of word i. Large numbers signify strong correspondence: that query is answered by that key.

Softmax and Scaling

A number like 7 or 1 is not yet a useful instruction: we need to know how much attention to pay to each word, as a percentage. The softmax is the function that converts any list of numbers into a probability distribution that sums to 1, respecting order: whichever had a high score receives a higher percentage. Applied to each row of our match table, we obtain something very interpretable: "The word cat should look 40% at black, 23% at itself, and only 5% at couch."

And why do we divide by $\sqrt{d_k}$ before the softmax? For numerical stability. In high-dimensional spaces, dot products tend to grow large; and with very large numbers, softmax behaves like an "all-or-nothing" switch: it punishes secondary options with near-zero percentages, and training stalls because those near-zero signals provide no gradient to learn from. Dividing by the square root of $d_k$ is, consequently, a "volume knob": it keeps scores in a comfortable range where all options retain some signal.

The Attention Pattern and Masking

The result of $QK^T$, once passed through softmax, is called the attention pattern: the map of who relates to whom within the sentence.

And here masking appears. The primary goal of a Transformer is to predict the next word, and it is beneficial to train it to do so from any point in the sentence: "The" → ?, "The ship" → ?, "The ship of" → ?, and so forth. In this way, a single sentence generates many training examples at once [4]. But there is a risk: that the model "cheats" by looking into the future. To prevent this, cells pointing to subsequent words (the upper triangle of the matrix) are filled with $-\infty$ before applying softmax; $-\infty$ converted into a probability is exactly zero. It is like covering the rest of the line with a piece of cardboard while reading, so as not to skip ahead to the answer.

A relevant fact today: the attention matrix is $N \times N$, meaning it grows quadratically with the number of tokens. With 1,000 tokens, one million relationships must be computed; with 10,000, one hundred million. This is why long texts are expensive to process and why chat applications enforce context limits.

Value: The Material Out of Which Meaning Is Built

So far, we only know how words relate to one another; the step that actually moves the original embedding so it acquires its appropriate qualities is still missing: cat + black → black cat. For this, a third matrix exists, $W_V$: $E_i \xrightarrow{W_V} V_i$. The Value is the "contribution" of each word: what that word can add to the meaning of another. That of "black" contributes "color: black."

Finally, we take the row of the attention pattern corresponding to the word (its attention percentages) and multiply it by the $V_i$, summing everything together. The result is the displacement $\Delta E$: the nudge that transports the original embedding toward its contextual meaning. For instance, if "cat" looks 40% at "black," its $\Delta E$ will be composed 40% of the contribution from "black." Note: the new embedding is formed as $E + \Delta E$, that is, adding on top of the original; this sum is, in fact, the residual connection that will reappear in the architecture.

Important: the matrices $W_Q$, $W_K$, and $W_V$ are not magic, nor do they come out of the box. They are the parameters of the network, the numbers the Transformer must discover during training. Training a model is, in essence, adjusting these tables until the questions, answers, and contributions they produce are useful.

A Step-by-Step Example: "El gato negro dormía en el sillón viejo" ("The black cat slept on the old couch")

Let us see everything together using the sentence "El gato negro dormía en el sillón viejo": it has two noun–adjective pairs ("gato"–"negro" and "sillón"–"viejo") and a verb that links them. We will use 2-dimensional vectors with made-up numbers, but which serve the purpose of the example (in a real model there are hundreds of dimensions and learned values, but the reasoning is identical).

Two clarifications: we omit the connectors "El", "en", and "el" so the tables fit on the page; and we will show the full pattern, without a mask (just as an encoder would compute it) — in a decoder, the "future" cells would be masked with $-\infty$, as seen earlier.

Step 1 - Queries and Keys (in practice, outputs of $W_Q$ and $W_K$):

Word	Q (question)	K (offer)
gato	$[2,\ 1]$ "who describes me?"	$[2,\ 0]$ "main subject"
negro	$[2,\ 0]$ "whom do I describe?"	$[1.9,\ 1]$ "I describe"
dormía	$[2,\ -0.3]$ "who is my subject?"	$[1,\ 1]$ "I provide action"
sillón	$[1,\ 2]$ "who describes me?"	$[0,\ 2]$ "main object"
viejo	$[-0.5,\ 2]$ "whom do I describe?"	$[0.9,\ 2]$ "I describe"

Step 2 - Scores $Q \cdot K^T$, divided by $\sqrt{2} \approx 1.41$:

(question ↓ / offer →)	gato	negro	dormía	sillón	viejo
gato	$2.83$	$3.39$	$2.12$	$1.41$	$2.69$
negro	$2.83$	$2.69$	$1.41$	$0.00$	$1.27$
dormía	$2.83$	$2.47$	$1.20$	$-0.42$	$0.85$
sillón	$1.41$	$2.76$	$2.12$	$2.83$	$3.46$
viejo	$-0.71$	$0.74$	$1.06$	$2.83$	$2.51$

Step 3 - Softmax by rows (the attention percentages):

(question ↓ / offer →)	gato	negro	dormía	sillón	viejo
gato	$23.0\%$	$40.2\%$	$11.3\%$	$5.6\%$	$20.0\%$
negro	$42.0\%$	$36.5\%$	$10.2\%$	$2.5\%$	$8.8\%$
dormía	$48.3\%$	$33.7\%$	$9.5\%$	$1.9\%$	$6.7\%$
sillón	$5.3\%$	$20.5\%$	$10.8\%$	$22.0\%$	$41.3\%$
viejo	$1.4\%$	$6.0\%$	$8.3\%$	$48.8\%$	$35.4\%$

Looking at each row:

• That of "gato": $40.2\%$ to "negro" (its description), $23\%$ to itself, and a touch to the rest.
• That of "negro": looks mostly at "gato" ($42\%$), because an adjective needs to know what it is describing.
• That of "dormía": concentrates on its subject, "gato" ($48.3\%$), which is exactly what a verb needs to know.

The rows for "sillón" and "viejo" mirror the story of "gato" and "negro": the furniture piece and its adjective look at each other ($41.3\%$ and $48.8\%$). And notice the effect of scaling: without dividing by $\sqrt{2}$, "negro" would take nearly $49\%$ of the attention of "gato" instead of $40.2\%$ — the "all-or-nothing" effect we want to avoid, smoothed out.

Step 4 - The Values and the displacement. Let us test the contributions of each word:

Word	V (contribution)
gato	$[0,\ 0]$ "nothing new for itself"
negro	$[1,\ 1]$ "color: black"
dormía	$[0,\ 0.5]$ "action nuance"
sillón	$[0,\ 0]$ "something else: does not contribute to the cat"
viejo	$[0.5,\ -0.5]$ "nuance: old"

The displacement of "gato" is the weighted sum of all contributions, using its attention row:

$$\Delta E_{\text{gato}} = 0.23\begin{bmatrix}0\\0\end{bmatrix} + 0.402\begin{bmatrix}1\\1\end{bmatrix} + 0.113\begin{bmatrix}0\\0.5\end{bmatrix} + 0.056\begin{bmatrix}0\\0\end{bmatrix} + 0.2\begin{bmatrix}0.5\\-0.5\end{bmatrix} \approx \begin{bmatrix}0.50\\0.36\end{bmatrix}$$

That is: the nudge is dominated by "negro", with a touch from the verb and a minimal nuance from "viejo". "Gato" shifts, therefore, toward its full meaning in this sentence: black cat that was sleeping. The remaining rows repeat the same calculation in parallel, and each word ends up with its own contextualized embedding ($E + \Delta E$). Stacking layers refines this effect: the word adjusted by layer 1 re-enters, now better informed, into a new round of attention.

Nuance Regarding the Original Paper

In the paper [1], the attention matrix does not have the same tokens vertically and horizontally: the task was translation, so on one axis go the Spanish tokens and on the other the English tokens. The decoder queries against the outputs of the encoder; this crossing between two different sequences is called cross-attention. What we built above — a word looking at the words of its own sentence — is the particular case called self-attention, which in the 2017 Transformer turns out to be the protagonist.

Multi-Head Attention

In a trained model, each trio $W_Q$, $W_K$, $W_V$ captures one way of interpreting text, and what they actually learn is far more complex to describe — a black box. The Transformer's solution was not to guess that complexity, but to repeat the mechanism in parallel: this is multi-head attention. Each head has its own trio of matrices, proposes its own $\Delta E$, and in the end all proposals are concatenated and projected (with a final matrix $W^O$) to be added to the original embedding.

The complete formula, now translated into words: each head i computes its own attention using its own matrices, and the final output is the concatenation of all heads passed through $W^O$ [1]:

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h) \cdot W^O$$

$$\text{head}_i = \text{Attention}(Q\,W_i^Q,\ K\,W_i^K,\ V\,W_i^V)$$

General Architecture of the Transformer

Encoder and Decoder

An encoder–decoder system passes from an input to a smaller latent space — a compressed internal representation — and from there to an output, seeking to make the relationship between input and output as faithful as possible [5]. That narrow intermediate point is known as a bottleneck, and its purpose is to force abstraction: if you can only "save" the essence, you have to discover what the essence is.

The Complete Structure

The Transformer stacks N identical layers across two towers [1], [4]:

• Encoder: each layer contains a Multi-Head Attention block (the words of the sentence look at each other) and a Feed Forward block — a small neural network applied to each word individually. After each block, there is an Add & Norm: a copy of what entered the block is retrieved (residual connection, the $E + \Delta E$ sum we already saw) and the scale of the numbers is normalized so that nothing explodes or vanishes.
• Decoder: each layer contains first a Masked Multi-Head Attention (it looks at what it has already written, unable to see the future), then a Cross-Attention whose Keys and Values come from the encoder, and finally its Feed Forward and Add & Norm.
• Output: a Linear layer producing raw scores (the logits, not yet normalized) and a final softmax converting them into probabilities over the entire vocabulary.

Experimentation

The paper [1] evaluated the Transformer architecture on the translation task, specifically:

• Data used: WMT 2014 international benchmarks were used, the standard of the era for evaluating translation models.

• English-to-German translation: ~4.5 million sentence pairs.

• English-to-French translation: ~36 million sentence pairs.

• Who did they compete against?: They compared against the state of the art of the time: recurrent networks (RNNs), LSTMs, Google's GNMT, ConvS2S, ByteNet, and ensembles.

• Metric: the Bilingual Evaluation Understudy (BLEU) metric was used, which measures how closely the generated translation resembles reference data. For example, given the sentence "The cat is sleeping on the sofa", and the English-to-Spanish translation task:

• A human reference is available: "El gato está durmiendo en el sofá".

• Model A: "El gato está durmiendo en el sofá" → broad match → high BLEU.

• Model B: "El felino duerme sobre el mueble" → correct meaning but different words → medium BLEU.

• Model C: "El perro está corriendo en el jardín" → incorrect meaning → very low BLEU.

• Additionally, they measured how long it took to train the models.

• The main research question: can a network based solely on attention be used, and does this translate into shorter training times?

Results

Transformer models consistently outperformed the state of the art on both evaluated translation tasks [1], demonstrating superiority in both quality and computational efficiency:

• English-to-German Translation (EN-DE): The Transformer (Big) model established a new state of the art by achieving a BLEU score of 28.4, outperforming the best existing models and ensembles by more than 2.0 BLEU points.
• English-to-French Translation (EN-FR): The same model achieved a BLEU score of 41.8, again surpassing all architectures from previous literature published up to that point.
• Training Cost (FLOPs/Time): The architectures achieved the above results at a remarkably lower computational cost. The base Transformer model was trained in just 3.5 days using 8 P100 GPUs, requiring between one-tenth and one-hundredth of the total training operations (FLOPs) demanded by recurrent (GNMT) or convolutional (ByteNet) architectures.

Why Was It Better Than the State of the Art?

The Transformer performed effectively better than what existed in 2017, but why? The state of the art was RNNs (recurrent neural networks): networks that read text word by word, dragging along and updating an internal memory. That design, which seemed natural, hid two serious limitations:

• Parallelization: In an RNN, to compute word $n+1$ you first need the result of word $n$: there is a chain dependency. This prevents efficiently taking advantage of the computers of the time — and of today —, particularly GPUs: chips specialized in executing thousands of simple calculations simultaneously.
• Long-distance dependencies: In a long text, words very far apart can depend on each other: "The dog that we saw two weeks ago … … … was tired." To know what was tired, information from "dog" must survive the entire journey to "was tired." RNNs tend to forget older words as new content enters: it is the game of broken telephone, where the message passes from person to person and arrives increasingly distorted. The attention mechanism eliminates the game: any word has direct access to any other, no matter how distant.

These two advantages did not merely allow faster training: they allowed training increasingly larger models. That capacity to scale is precisely what gave rise to BERT [6], GPT, and all modern models.

The Current Landscape

In recent times, the Transformer architecture has seen improvements and variations in the following areas:

• Reasoning models: LLMs trained specifically to reason.
• Efficiency improvements: Flash Attention, DeepSeek, and others have designed ways to train/run models at lower cost.
• Harnesses: Expanding model capabilities through unified infrastructure, Claude, Codex, and Pi being some notable exponents.
• Goal-Driven Reinforcement Learning: Integration of policy optimization algorithms (such as PPO) and alignment methods (RLHF or DPO), to maximize strict reward functions in formal and mathematical reasoning domains.

In any case, the Transformer architecture has proven transformational for the field, and its repercussions have reached far beyond natural language processing, establishing a unified foundational architecture for a diverse range of applications.

Why It Matters

The mechanism you have just explored does not live solely inside a paper; it is used in your spell checker, your translator, your search engine, and every customer support chat. Understanding attention is therefore understanding what these systems can and cannot do: they do not "read" like a human; they reconstruct meaning statistically by moving vectors on a map. That distinction is practical judgment: knowing when to trust, when to verify, and why a model can sound confident and still be wrong. It also explains concrete limitations of your digital life: context limits in AI apps exist because the attention matrix grows quadratically — each additional word costs progressively more.

If You Want to Go Deeper

1. The Illustrated Transformer (Jay Alammar) [7] — the classic visual, step-by-step explanation.
2. The 3Blue1Brown video series on Transformers and attention [8], [9] — exceptional visualizations of everything described here.
3. The Annotated Transformer (Harvard NLP) [10] or Andrej Karpathy's video Let's build GPT from scratch [11] — for those who want to see the mechanism translated into code, line by line.

Sources

[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017, pp. 5998–6008. [Online]. Available: https://arxiv.org/abs/1706.03762

[2] D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," in Proc. 3rd Int. Conf. Learning Representations (ICLR), San Diego, CA, USA, 2015. [Online]. Available: https://arxiv.org/abs/1409.0473

[3] S. Cristina, "The attention mechanism from scratch," MachineLearningMastery.com, Jan. 6, 2023. Accessed: Sep. 30, 2026. [Online]. Available: https://machinelearningmastery.com/the-attention-mechanism-from-scratch/

[4] U. Jamil, "Attention is all you need (Transformer) - Model explanation (including math), inference and training," YouTube, May 28, 2023. [Online Video]. Available: https://www.youtube.com/watch?v=bCz4OMemCcA. Accessed: Sep. 30, 2026.

[5] L. Weng, "From autoencoder to beta-VAE," Lil'Log, Aug. 12, 2018. Accessed: Sep. 30, 2026. [Online]. Available: https://lilianweng.github.io/posts/2018-08-12-vae/

[6] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in Proc. 2019 Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Minneapolis, MN, USA, 2019, pp. 4171–4186. [Online]. Available: https://arxiv.org/abs/1810.04805

[7] J. Alammar, "The illustrated transformer," Jun. 27, 2018. Accessed: Sep. 30, 2026. [Online]. Available: https://jalammar.github.io/illustrated-transformer/

[8] G. Sanderson (3Blue1Brown), "Transformers, the tech behind LLMs | Deep Learning Chapter 5," YouTube, Apr. 1, 2024. [Online Video]. Available: https://www.youtube.com/watch?v=wjZofJX0v4M. Accessed: Sep. 30, 2026.

[9] G. Sanderson (3Blue1Brown), "Attention in transformers, step-by-step | Deep Learning Chapter 6," YouTube, Apr. 7, 2024. [Online Video]. Available: https://www.youtube.com/watch?v=eMlx5fFNoYc. Accessed: Sep. 30, 2026.

[10] A. Huang, S. Subramanian, J. Sum, K. Almubarak, S. Biderman, and A. Rush, "The annotated transformer," Harvard NLP, 2022. Accessed: Sep. 30, 2026. [Online]. Available: https://nlp.seas.harvard.edu/annotated-transformer/

[11] A. Karpathy, "Let's build GPT: from scratch, in code, spelled out," YouTube, Jan. 17, 2023. [Online Video]. Available: https://www.youtube.com/watch?v=kCc8FmEb1nY. Accessed: Sep. 30, 2026.