How to Read This Chapter
Overview
The math behind LLMs and agents is not abstract theory—it directly explains why RAG retrieval works, why training converges, and how RLHF aligns chatbots. Each section gives intuition, a formal equation, symbol definitions, and a production link.
Technical Deep Dive
Reading order: Vectors (retrieval) → Cross-entropy (language modeling) → Gradient descent (training) → Probability (evaluation) → Entropy (uncertainty) → Bayesian (confidence) → RL (RLHF).
Notation: scalars are italic (η), vectors bold or with arrow context (a, b), matrices often capital (W). Subscripts mark time step (t) or token index (i).
Practical Use Case
When debugging an agent, map the symptom to math: wrong documents → cosine similarity; overconfident answers → calibration / cross-entropy; unstable fine-tune → learning rate η and gradient norms.
Vectors & Embeddings
Overview
A vector is an ordered list of numbers in ℝⁿ (n-dimensional real space). Embedding models map words, tokens, or whole documents into these vectors so that semantic similarity becomes geometric closeness—foundation of every vector database in RAG.
Technical Deep Dive
Dot product: a·b = Σᵢ aᵢbᵢ measures alignment of two vectors (larger = more similar direction). Norm: ‖a‖ = √(a·a) is vector length. Cosine similarity: cos(θ) = (a·b)/(‖a‖‖b‖) ∈ [−1, 1] ignores magnitude—ideal when document length varies.
Euclidean distance: ‖a−b‖ used in some indexes; for unit vectors, distance and cosine are monotonically related.
Embedding dimension d: typical 384 (small), 768 (BERT-base), 1536 (OpenAI ada), up to 3072. Higher d can capture finer semantics but costs storage and search time.
L2 normalization: â = a/‖a‖ forces ‖â‖=1 so dot product equals cosine similarity—many APIs return normalized embeddings by default.
Practical Use Case
Enterprise RAG: encode queries and chunks with the same model; retrieve top-k by cosine similarity; rerank with a cross-encoder. If recall is poor, benchmark alternate embedding models on your query–document pairs (MTEB leaderboard is only a starting point).
Gradient Descent
Overview
Neural networks learn by adjusting parameters θ to reduce a loss function L(θ). Gradient descent moves θ in the direction that decreases L fastest—computed via backpropagation.
Technical Deep Dive
Gradient ∇L(θ): vector of partial derivatives [∂L/∂θ₁, …, ∂L/∂θₙ] pointing uphill on the loss surface. We step opposite to the gradient to go downhill.
SGD (Stochastic Gradient Descent): updates on mini-batches—noisy but fast. Adam: adaptive per-parameter learning rates using moving averages of gradient and squared gradient. AdamW: Adam with decoupled weight decay for better generalization.
Learning rate η: step size; too large → divergence; too small → slow training. Schedules: linear warmup then cosine decay are standard for LLM fine-tuning.
Practical Use Case
Fine-tune a ticket-routing head on frozen embeddings: AdamW, η=2×10⁻⁵, 3 epochs, watch validation loss. Full LLM training uses distributed optimizers (ZeRO, FSDP) because θ has billions of dimensions.
Cross-Entropy Loss
Overview
Cross-entropy measures how poorly a predicted probability distribution q matches the true distribution p. It is the standard training objective for classifiers and for next-token prediction in language models.
Technical Deep Dive
For discrete outcomes: H(p, q) = −Σₓ p(x) log q(x). When p is one-hot (true class c), this simplifies to −log q(c)—heavily penalizing low probability on the correct token.
Softmax: converts logits z to probabilities qᵢ = exp(zᵢ)/Σⱼ exp(zⱼ). Numerically stable implementations subtract max(z) before exp.
Perplexity (PPL): exp(average cross-entropy per token)—“effective branching factor.” PPL=20 means the model is as uncertain as choosing uniformly among ~20 tokens each step.
Label smoothing: replaces sharp one-hot p with softened targets—reduces overconfidence and improves calibration.
Practical Use Case
Compare candidate fine-tunes on held-out perplexity and task accuracy. A model with lower training CE but higher PPL on validation is overfitting.
Probability & Statistics
Overview
Probability models uncertainty; statistics summarizes data and quantifies confidence in metrics—essential for A/B testing agents and reporting SLA compliance without fooling yourself with noise.
Technical Deep Dive
P(A|B) — conditional probability: probability of A given B occurred. Bayes' rule: P(A|B) = P(B|A)P(A)/P(B).
E[X] — expectation: average outcome weighted by probability. Var(X) = E[(X−E[X])²]: spread around the mean.
MLE (Maximum Likelihood Estimation): choose parameters θ that maximize P(data|θ)—equivalent to minimizing cross-entropy for many models.
95% confidence interval (CI): range that would contain the true metric in ~95% of repeated experiments—report [72%, 78%] not just “75% accuracy.”
Practical Use Case
Agent rollout: 10,000 tasks, success rate 75% → 95% CI ≈ [74.2%, 75.8%] with large n. If baseline CI overlaps treatment CI, you cannot claim a win yet.
Entropy & Information Theory
Overview
Entropy quantifies unpredictability in a distribution. In ML it connects loss functions, compression, and how “surprised” a model is by each token—core to understanding perplexity and model confidence.
Technical Deep Dive
Shannon entropy: H(X) = −Σₓ p(x) log₂ p(x) measured in bits (use ln for nats). Uniform distribution over N outcomes has maximum entropy log₂(N).
Cross-entropy H(p,q): average bits to encode truth p using code optimized for q. KL divergence: D_KL(p‖q) = H(p,q) − H(p) ≥ 0 (zero iff p=q).
Minimizing cross-entropy on one-hot targets = maximizing log-likelihood of correct class.
Practical Use Case
Monitor entropy of next-token distribution in production: sudden drop may indicate memorization or prompt leakage; spike may indicate distribution shift or adversarial inputs.
Bayesian Reasoning
Overview
Bayesian methods combine prior beliefs with observed evidence to obtain updated (posterior) beliefs—natural framework for calibrated confidence, fraud scoring, and decision-making under uncertainty.
Technical Deep Dive
Posterior ∝ likelihood × prior: P(θ|data) ∝ P(data|θ) P(θ). The prior encodes what we believed before seeing data; the likelihood encodes how well θ explains observations.
Aleatoric uncertainty: irreducible noise in data (e.g. ambiguous user intent). Epistemic uncertainty: ignorance from limited training—reducible with more data.
Calibration: when the model says “80% confident,” it should be correct ~80% of the time—reliability diagrams plot predicted vs actual frequency.
Practical Use Case
Fraud agent outputs score 0.73 meaning P(fraud|evidence) under a calibrated model—investigators sort queue by expected loss, not raw logits.
Reinforcement Learning
Overview
RL trains an agent to maximize cumulative reward by interacting with an environment. RLHF (Reinforcement Learning from Human Feedback) uses RL to align LLMs with human preferences after supervised fine-tuning.
Technical Deep Dive
MDP (Markov Decision Process): tuple (S, A, P, R, γ) — states, actions, transition dynamics, rewards, discount factor.
Policy π(a|s): probability of action a in state s. Value V(s): expected discounted return from s. Q(s,a): expected return taking a in s then following π.
Objective: maximize J = E[Σₜ γᵗ rₜ]. γ∈[0,1] weights future rewards (γ→0 myopic, γ→1 long-horizon).
PPO (Proximal Policy Optimization): policy gradient with clipped updates—stable RLHF workhorse. Reward model R(x,y) trained on human rankings supplies r.
Practical Use Case
Chatbot alignment: SFT on demonstrations → train reward model from thumbs up/down → PPO fine-tune policy with KL penalty to stay near reference model—prevents catastrophic forgetting while improving tone.
Equation Reference
Key formulas with symbol guides and step-by-step intuition. Expand each card for worked explanations.
Symbol guide
| Symbol | Meaning |
|---|---|
a, b | Embedding vectors (e.g. query and document) |
a · b | Dot product — sum of element-wise products |
‖a‖ | L2 norm (Euclidean length) of vector a |
θ | Angle between a and b in ℝⁿ |
cos(θ) | Cosine similarity score used for nearest-neighbor search |
- Compute dot product a·b = Σᵢ aᵢbᵢ (measure of alignment).
- Compute L2 norms ‖a‖ and ‖b‖ (vector lengths).
- Divide: result is 1 when vectors point the same way, 0 when orthogonal, −1 when opposite.
- For unit-normalized embeddings (‖a‖=‖b‖=1), cosine similarity equals the dot product.
Symbol guide
| Symbol | Meaning |
|---|---|
θₜ | Parameter vector at training step t (weights & biases) |
η | Learning rate — hyperparameter controlling step size |
∇L(θₜ) | Gradient of loss L w.r.t. all parameters at θₜ |
L(θ) | Loss function (e.g. cross-entropy) to minimize |
- At step t, compute loss L on a mini-batch.
- Backpropagate to get gradient ∇L(θₜ) — direction of steepest increase in loss.
- Subtract η∇L to move parameters downhill (η controls step size).
- Repeat until validation loss plateaus or early-stopping triggers.
Symbol guide
| Symbol | Meaning |
|---|---|
p | True probability distribution (ground truth) |
q | Model's predicted distribution (softmax output) |
H(p,q) | Cross-entropy — lower is better |
x | Index over classes or vocabulary tokens |
- True distribution p (often one-hot: probability 1 on correct class).
- Model prediction q from softmax over logits.
- Sum −p(x)log q(x) over all classes x — penalizes assigning low q to true outcomes.
- For language modeling, sum over tokens and average for perplexity.
Symbol guide
| Symbol | Meaning |
|---|---|
P(A|B) | Posterior — probability of A given B observed |
P(B|A) | Likelihood — probability of evidence B if A is true |
P(A) | Prior — belief before seeing B |
P(B) | Evidence — marginal probability of B |
Update belief about hypothesis A after observing evidence B. In spam detection, A = “spam”, B = “contains word winner”—posterior P(spam|word) drives the decision.
Symbol guide
| Symbol | Meaning |
|---|---|
H(X) | Entropy of random variable X (uncertainty) |
p(x) | Probability of outcome x |
log₂ | Base-2 log — result in bits |
Average number of bits needed to encode outcomes from distribution p. Fair coin: p(0)=p(1)=0.5 → H=1 bit. Deterministic outcome: H=0.
Symbol guide
| Symbol | Meaning |
|---|---|
θ | Model parameters or hypothesis |
P(θ) | Prior — belief before data |
P(data|θ) | Likelihood — how well θ explains observations |
P(θ|data) | Posterior — updated belief after data |
Posterior is proportional to likelihood times prior. Normalize so probabilities sum to 1. Conjugate priors yield closed-form posteriors (e.g. Beta-Binomial).
Symbol guide
| Symbol | Meaning |
|---|---|
Gₜ | Return — total discounted reward from time t |
rₜ | Reward at step t (human preference score) |
γ | Discount factor ∈ [0,1] — weight on future rewards |
π(a|s) | Policy — probability of action a in state s |
RLHF maximizes expected Gₜ where r comes from a reward model trained on human preferences. PPO constrains policy updates so π does not drift too far from π_ref.