Transformer: From First Principles to a Product Lens

📚 Why does the Transformer matter so much?

The Transformer architecture came out of Vaswani et al.’s 2017 paper “Attention Is All You Need.” It changed NLP completely and is now the backbone behind every major LLM — GPT, BERT, LLaMA, you name it.

The core idea: drop CNNs and RNNs entirely and rely only on the attention mechanism to capture dependencies across a sequence. That single move solved two problems at once:

• Parallelism: RNNs process tokens one at a time; attention lets you process the whole sequence at once, which speeds up training enormously
• Long-range dependencies: RNNs/LSTMs tend to lose the signal over long sequences (vanishing gradients); attention doesn’t have that problem

🏗️ Part 1: The Overall Architecture

1.1 Encoder-decoder structure

The Transformer follows the classic Seq2Seq encoder-decoder pattern:

• Encoder: turns the input sequence (say, a sentence in the source language) into a continuous representation
  • Made of N identical layers stacked on top of each other (N=6 in the original paper)
  • Each layer has a multi-head self-attention sublayer plus a feed-forward sublayer
• Decoder: generates the target sequence based on what the encoder produced
  • Also N identical layers stacked together
  • Each layer has masked multi-head self-attention, encoder-decoder attention, and a feed-forward sublayer
  • The masking: makes sure that when the decoder predicts position i, it can only see outputs before position i — otherwise it would be cheating by looking at the answer

1.2 Core components at a glance

🔍 Part 2: The Core Mechanisms

2.1 Self-attention

The idea: let every word in a sequence look directly at every other word and figure out, dynamically, how relevant each one is.

How it works:

1. Input representation: each input word $x_i$ gets turned into an embedding vector
2. Building Q, K, V: three trainable weight matrices $W_Q, W_K, W_V$ project the input into Query, Key, and Value vectors
3. Attention scores: take the dot product of a Query against every Key — this tells you how much the current position should attend to each other position
4. Scale and normalize: divide by $\sqrt{d_k}$ (otherwise large dot products push the gradients toward zero), then run it through Softmax to get weights
5. Weighted sum: use those weights to combine the Values into the final output

The formula: $$ \text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

Why split into Q, K, V?

• Query: “what is this position looking for”
• Key: “what can this position offer” (a label or index)
• Value: “the actual content this position contributes once it’s attended to”

Why three separate roles? It cleanly separates responsibilities — the Key decides how much attention a position gets, the Value decides what it hands over once attended to, and the Query decides what to look for. That’s a lot more expressive than cramming everything into a single vector.

2.2 Multi-head attention

The idea: instead of looking at a sentence with one pair of eyes, use several pairs of “expert eyes” that each look at it from a different angle at the same time.

How it’s done:

• Project Q, K, V into $h$ smaller subspaces ($h$ is the number of heads — 8, 12, whatever)
• Compute attention independently in each subspace
• Concatenate all the heads’ outputs and pass them through an output projection $W^O$

The formula: $$ \text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O $$ $$ \text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V) $$

Why it helps:

• Division of labor: different heads end up specializing in different things — syntax, semantics, coreference, etc.
• More capacity: catches dependencies at different granularities
• Parallel-friendly: maps perfectly onto GPU compute

A fun fact: nobody tells each head what to specialize in. That division of labor emerges on its own during training, purely from the data. Researchers can visualize what a head learned afterward, but they can’t prescribe it in advance.

2.3 Positional encoding

Why bother? Self-attention has no built-in sense of order — shuffle the words in a sentence and you’d get the exact same output.

The fix: inject position information into the embeddings.

The sine/cosine scheme from the original paper: $$ PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) $$ $$ PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) $$ where $pos$ is the position, $i$ is the dimension index, and $d_{\text{model}}$ is the hidden size.

Why this design works well:

• Values stay in $[-1, +1]$, so they don’t overwhelm the word embedding’s semantics
• It extrapolates to sequences longer than anything seen in training
• It naturally encodes relative position too ($PE_{pos+k}$ can be written as a linear combination of $PE_{pos}$)

Where things have moved since: a lot of newer models (LLaMA, RoBERTa) use learned absolute positional embeddings or rotary position embeddings (RoPE) instead.

2.4 Feed-forward network

Every Transformer layer includes a two-layer fully connected network: $$ \text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2 $$

A few things worth noting:

• Applied independently at each position, with shared parameters
• The inner dimension is usually 4x wider (e.g. $d_{\text{model}}=512$, $d_{ff}=2048$)
• The ReLU non-linearity is what adds expressive power here

One finding worth flagging: making this layer wider tends to help model performance.

2.5 Residual connections and layer normalization

Stacking six or more layers deep creates training-stability problems, and these two tricks are what keep it from falling apart.

Residual connections

What they solve: network degradation — past a certain depth, adding more layers can actually hurt performance instead of helping.

How it works: $$ \text{output} = \text{LayerNorm}(x + \text{Sublayer}(x)) $$

Why it works:

• It gives gradients a “highway” straight back to earlier layers
• The network can learn an identity mapping, so a useless extra layer at worst does nothing rather than hurting things
• Borrowed directly from ResNet

Layer normalization

What it solves: vanishing/exploding gradients, so training stays stable.

How it works: normalize across all the features of a single sample using its mean and variance: $$ \text{LN}(x) = g \cdot \frac{x - \mu}{\sigma} + b $$ where $g$ and $b$ are learned scale and shift parameters, and $\mu$, $\sigma$ are the mean and standard deviation.

Two variants:

• Post-norm: the original setup — normalize after the residual add
• Pre-norm: what most modern models use — normalize before the sublayer, which makes deep networks easier to train

📊 Part 3: A Closer Look at the Key Concepts

3.1 Hidden dimension ($d_{\text{model}}$ / hidden_size)

What it is: the size of the “mental canvas” the model uses internally — how many numbers represent each word inside the model.

An analogy:

• Small hidden dim (128-256): a pocket notebook — room for the essentials only
• Medium (768-1024): a regular desk — enough space to think things through properly
• Large (4096+): a full studio — room to really stretch out

Why it matters: hidden dimension drives both how expressive the model is and how expensive it is to run. The bulk of a Transformer layer’s parameters scale with $d_{\text{model}}$ squared.

3.2 Attention heads

What they are: the parallel units inside multi-head attention, each learning its own pattern of attention within a smaller subspace ($d_{\text{model}}/h$).

How many heads models typically use:

• BERT-base: 12 layers, 12 heads
• BERT-large: 24 layers, 16 heads
• GPT-3: 96 layers, 96 heads
• LLaMA: 32/40/80 layers, 32/40/64 heads depending on size

A more recent trick: Grouped-Query Attention (GQA) lets multiple heads share the same Key-Value projections, which shrinks the KV cache a lot at inference time with barely any quality loss (used in LLaMA 2/3).

3.3 Number of layers

What it means: how “deep” the information refinement goes. Every layer gives the word vectors one more round of talking to each other in context.

The trade-off:

• More layers: better at abstract reasoning, but needs more data, longer training, and more memory
• Fewer layers: trains fast, deploys easily, but struggles with complex tasks

A rule of thumb: performance scales as a power law with layer count, hidden dimension, and data size together.

3.4 What the model’s weights actually are

What’s inside the weights:

• The attention projection matrices ($W_Q, W_K, W_V, W_O$)
• The feed-forward weight matrices ($W_1, W_2$)
• Layer norm’s scale and shift parameters ($g, b$)
• The word embedding matrix

Why are these files so huge? It comes down to parameter count × numeric precision:

• FP32: 4 bytes per parameter
• FP16/BF16: 2 bytes per parameter
• INT8: 1 byte per parameter

Formula: file size ≈ parameter count × bytes per parameter.
Example: LLaMA 3 70B stored in FP16 ≈ $70B \times 2 \text{ bytes} = 140\text{GB}$

Where do the weights come from? They’re learned through gradient descent and backpropagation on massive amounts of data — nobody hand-writes them.

🧮 Part 4: Estimating Parameter Counts

4.1 Parameter count for a single layer

4.2 Estimating the total

Per layer ≈ $4d_{\text{model}}^2 + 8d_{\text{model}}^2 = 12d_{\text{model}}^2$ (assuming $d_{ff}=4d_{\text{model}}$)

Total model size ≈ $L \times 12d_{\text{model}}^2 + V \times d_{\text{model}}$ ($L$ = number of layers, $V$ = vocab size)

Sanity check with LLaMA 7B ($d_{\text{model}}=4096, L=32, V=32000$):
$32 \times 12 \times 4096^2 + 32000 \times 4096 \approx 6.4B + 0.13B = 6.53B$ — close enough to the “7B” name.

4.3 Estimating training cost

The core formula: $C \approx 6PD$

• $C$: total training compute (FLOPs)
• $P$: number of parameters
• $D$: training data size (tokens)

Rule of thumb: a compute-optimal model roughly satisfies $D \approx 20P$

🚀 Part 5: From Pre-training to a Usable Model

5.1 Pre-training

Goal: get the model to pick up general language patterns and world knowledge.

How: run either autoregressive language modeling (predict the next word) or masked language modeling (fill in the blank) on a huge pile of unlabeled text.

What you get: a base model that’s great at continuing text, but hasn’t been taught to follow instructions or match human preferences yet.

5.2 Supervised Fine-Tuning (SFT)

Goal: teach the model to follow instructions and hold a conversation.

Data: high-quality, human-written instruction-response pairs.

In essence: a supervised task where the model learns to reproduce the “correct” answer.

5.3 RLHF (Reinforcement Learning from Human Feedback)

Goal: get the model’s outputs to better match human preferences — safe, helpful, harmless.

The pipeline:

1. Humans rank several model outputs by preference
2. A reward model is trained to learn those preferences
3. Reinforcement learning (e.g. PPO) is used to fine-tune the SFT model against that reward model

5.4 Parameter-efficient fine-tuning (PEFT)

Full fine-tuning is too expensive for large models, so a family of cheaper alternatives showed up:

• LoRA: inject low-rank adapters and only train those
• Adapters: small trainable modules inserted between layers
• Prefix tuning: optimize a continuous prompt vector instead of the model weights

🔧 Part 6: Compression and Speed

Deploying a big model means dealing with efficiency, and there are a few standard tools for that:

6.1 Quantization

The idea: represent weights and activations with lower precision (INT8, INT4, etc.)

Two flavors:

• Post-training quantization (PTQ): quantize after training, no retraining needed
• Quantization-aware training (QAT): simulate quantization error during training — better results, but more expensive

Payoff: half the model size (FP16 → INT8) and faster inference

6.2 Pruning

The idea: remove redundant attention heads, FFN layers, or individual weight connections that don’t matter much.

Two approaches: structured pruning (drop whole heads/layers) vs. unstructured pruning (drop individual weights)

6.3 Knowledge distillation

The idea: train a small “student” model to mimic a large “teacher” model’s output distribution.

Why it helps: the student doesn’t just learn the right answer — it learns the teacher’s “way of thinking” via soft labels.

Where it’s used: shrinking cloud-scale models down to run on phones and IoT devices.

6.4 Parameter sharing

The idea: share parameters across different layers.

Example: ALBERT cuts parameter count dramatically by sharing parameters across layers.

💡 Part 7: A Product Manager’s View of the Transformer

7.1 Key tradeoffs when picking a model

7.2 Cost vs. capability

• Training cost ≈ $6 \times \text{parameters} \times \text{data size}$
• Inference memory ≈ $1.2 \times \text{parameters} \times \text{bytes per parameter}$ (at inference time)
• Training memory: weights + gradients + optimizer state — typically 3-4x what inference needs

7.3 A rough decision checklist

1. What’s the task complexity? Simple classification? Complex reasoning? Creative generation?
2. What are the resource constraints? Cloud or on-device? Latency budget? Cost budget?
3. Where do you start? A general-purpose model API, or an open model you can fine-tune?
4. What’s the data plan? How much labeled data do you need, and how good does it have to be?
5. How will you evaluate it? Model metrics vs. actual user experience metrics.

7.4 Where a model’s capability actually comes from

• Data: the scale, diversity, and quality of the pre-training data sets the ceiling on what the model can know.
• Architecture: the Transformer and its variants are what make parallel computation and long-range dependency modeling practical at this scale.
• Alignment: SFT and RLHF are what bend the model’s behavior toward what people actually want.

References

1. Vaswani et al. (2017). Attention Is All You Need
2. Devlin et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
3. Brown et al. (2020). Language Models are Few-Shot Learners
4. Hoffmann et al. (2022). Training Compute-Optimal Large Language Models
5. Touvron et al. (2023). LLaMA: Open and Efficient Foundation Language Models
6. Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models
7. Sanh et al. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter