Introduction
Overview
The Transformer (Vaswani et al., 2017) replaced recurrence with self-attention, enabling parallel training and becoming the foundation of virtually all modern LLMs.
Technical Deep Dive
Key innovation: all pairwise token interactions in O(n²) attention vs. O(n) sequential RNN steps—tradeoff favors GPUs on parallel matmul. Variants: encoder-only (BERT), decoder-only (GPT), encoder-decoder (T5).
Scaled to 100B+ parameters via data, compute, and architectural tweaks (RoPE, SwiGLU, RMSNorm).
Practical Use Case
Every enterprise copilot today runs on Transformer decoders—understanding attention is prerequisite for debugging context failures.
01Self-Attention
Overview
Self-attention computes a weighted sum of value vectors where weights come from query-key compatibility—each token builds a context-aware representation.
Technical Deep Dive
Per head: Q=XWq, K=XWk, V=XWv; Attention=softmax(QKᵀ/√d_k)V. Multi-head runs h parallel attentions; concat + project. √d_k prevents softmax saturation.
Causal masking in decoders sets future positions to -∞ before softmax.
Practical Use Case
Long doc QA fails when key evidence sits in lost-in-the-middle positions—mitigate with reranking + place critical chunks at context edges.
RLHF & Alignment
Overview
RLHF aligns models with human preferences via reward model training on comparison data, then policy optimization (PPO) to maximize reward while staying near reference model (KL penalty).
Technical Deep Dive
Pipeline: SFT on demonstrations → train reward model R(x,y) from human rankings → RL fine-tune π to maximize R − β·KL(π||π_ref). Alternatives: DPO, KTO (no explicit RM).
Alignment targets helpfulness, harmlessness, honesty—not raw next-token prediction.
Practical Use Case
Enterprise fine-tune: SFT on approved support transcripts + DPO on thumbs-up/down logs—reduces off-brand tone without full RL infrastructure.
02Attention Mechanism
Overview
Queries, keys, and values are learned linear projections of hidden states; multi-head attention runs parallel attention subspaces.
Technical Deep Dive
Each head learns different relational patterns (syntax vs. coreference). Output concatenation + W_O mixes heads. In decoder-only models, causal masking ensures position i attends only to j≤i.
Practical Use Case
Debugging generation errors often traces to attention failing on long-range dependencies—switch to models with longer context or insert retrieval for distant facts.
Embeddings & Tokenization
Overview
Tokenizers (BPE, SentencePiece) segment text into subword units; embedding layer E maps token IDs to ℝ^d vectors combined with positional information.
Technical Deep Dive
Vocabulary size 32k–128k. Special tokens: pad, bos, eos. Byte-level BPE handles rare words and typos. Embedding tables are largest memory component in small models; tied weights with output lm_head reduce parameters.
Practical Use Case
Multilingual helpdesk uses SentencePiece trained on ticket corpus—OOV rate drops 40% vs. word-level tokenization on product codenames.
Positional Encoding
Overview
Transformers have no inherent order; positional encodings (sinusoidal, learned, RoPE, ALiBi) inject sequence position information.
Technical Deep Dive
RoPE (Rotary Position Embedding) rotates Q/K vectors by position-dependent angles—enables relative position generalization and extrapolation beyond training length in some settings.
Practical Use Case
Long-context contracts use RoPE-based models at 128k; legal teams validate performance on clause cross-references at document end.