Sources and References

Primary texts — Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning (MIT Press, 2016); Christopher Bishop and Hugh Bishop, Deep Learning: Foundations and Concepts (Springer, 2024). Online resources — Stanford CS231n (CNNs for Visual Recognition); Stanford CS224n (NLP with Deep Learning); fast.ai Practical Deep Learning course; Jay Alammar’s illustrated transformer blog series; Lilian Weng’s blog on diffusion models.

Chapter 1: Feedforward Deep Networks

1.1 From Linear Models to Deep Networks

The story of modern deep learning begins with the simplest computational building block: the linear classifier. Given an input vector \(\mathbf{x} \in \mathbb{R}^d\), a linear model produces an output \(\hat{y} = \mathbf{w}^\top \mathbf{x} + b\) for some weight vector \(\mathbf{w}\) and bias \(b\). While linear models are interpretable and computationally convenient, they cannot represent the highly nonlinear input–output relationships present in natural images, speech, or language. The central insight of deep learning is that by composing many simple nonlinear transformations, a network can learn hierarchical representations of arbitrarily complex data.

The perceptron, introduced by Rosenblatt in 1958, was the first trainable linear classifier. Given a binary label \(y \in \{-1, +1\}\), the perceptron update rule adjusts \(\mathbf{w}\) whenever a training example is misclassified. Although the perceptron convergence theorem guarantees that the rule finds a separating hyperplane if one exists, the model fails on non-linearly separable problems — a limitation famously documented by Minsky and Papert. The solution is to stack multiple layers.

1.2 Activation Functions

A multilayer network inserts a pointwise nonlinearity \(\sigma\) after each linear transformation. Without such nonlinearities, the composition of linear maps is itself linear, and stacking layers would provide no representational benefit. Several activation functions have proven effective in practice.

1.3 The Multilayer Perceptron

A multilayer perceptron (MLP), also called a feedforward neural network, is defined by a sequence of affine transformations interleaved with elementwise nonlinearities. Let \(L\) denote the number of layers, \(\mathbf{h}^{(0)} = \mathbf{x}\) the input, and for \(\ell = 1, \ldots, L\):

\[ \mathbf{a}^{(\ell)} = \mathbf{W}^{(\ell)}\mathbf{h}^{(\ell-1)} + \mathbf{b}^{(\ell)}, \qquad \mathbf{h}^{(\ell)} = \sigma\!\left(\mathbf{a}^{(\ell)}\right). \]

The final layer typically uses a task-specific transformation: a softmax for multiclass classification, a sigmoid for binary classification, or the identity for regression. Denoting the network’s output as \(f(\mathbf{x};\theta)\) where \(\theta\) collects all weights and biases, training minimizes an empirical risk

\[ \mathcal{L}(\theta) = \frac{1}{N}\sum_{i=1}^N \ell\!\left(f(\mathbf{x}_i;\theta),\, y_i\right) \]

over a dataset \(\{(\mathbf{x}_i, y_i)\}_{i=1}^N\). Common loss functions include cross-entropy for classification and mean squared error for regression.

1.4 Universal Approximation

A foundational theoretical result justifies the expressive power of MLPs.

1.5 Computational Graphs and Backpropagation

Training a neural network requires computing gradients of the loss with respect to all parameters. Backpropagation is an efficient algorithm for this purpose, implementing the chain rule of calculus over the computational graph of the network.

A computational graph represents the computation as a directed acyclic graph (DAG) in which nodes correspond to variables (scalars, vectors, or tensors) and edges represent functional dependencies. Each node stores both its forward-pass value and a rule for propagating gradients backward.

For a scalar loss \(\mathcal{L}\) and a chain of composed functions, the chain rule states:

\[ \frac{\partial \mathcal{L}}{\partial \mathbf{a}^{(\ell)}} = \frac{\partial \mathcal{L}}{\partial \mathbf{h}^{(\ell)}} \odot \sigma'\!\left(\mathbf{a}^{(\ell)}\right), \]\[ \frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(\ell)}} = \frac{\partial \mathcal{L}}{\partial \mathbf{a}^{(\ell)}} \left(\mathbf{h}^{(\ell-1)}\right)^\top, \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{b}^{(\ell)}} = \frac{\partial \mathcal{L}}{\partial \mathbf{a}^{(\ell)}}, \]\[ \frac{\partial \mathcal{L}}{\partial \mathbf{h}^{(\ell-1)}} = \left(\mathbf{W}^{(\ell)}\right)^\top \frac{\partial \mathcal{L}}{\partial \mathbf{a}^{(\ell)}}. \]

Here \(\odot\) denotes the elementwise (Hadamard) product. The backward pass proceeds from layer \(L\) to layer \(1\), reusing intermediate activations stored during the forward pass. The total computational cost is \(O(L)\) passes, making backpropagation linear in network depth.

---

Chapter 2: Optimization and Regularization

2.1 Loss Landscapes and Gradient Descent

The loss function \(\mathcal{L}(\theta)\) defines a surface over the high-dimensional parameter space. Unlike convex objectives, neural network loss landscapes are non-convex, containing saddle points, local minima, and flat regions. Empirically, first-order gradient methods find solutions of remarkably good generalization quality, though the theoretical understanding of why remains an active area of research.

Batch gradient descent updates parameters by computing the gradient over the entire dataset:

\[ \theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}(\theta), \]

where \(\eta > 0\) is the learning rate. This is computationally prohibitive for large datasets. Stochastic gradient descent (SGD) approximates the true gradient using a single randomly chosen example, while mini-batch SGD uses a batch of \(B\) examples:

\[ \theta \leftarrow \theta - \frac{\eta}{B} \sum_{i \in \mathcal{B}} \nabla_\theta \ell(f(\mathbf{x}_i;\theta), y_i). \]

The stochastic noise in mini-batch gradients acts as an implicit regularizer, often helping escape sharp minima.

2.2 Momentum and Adaptive Methods

RMSProp addresses the challenge that different parameters may require different effective learning rates. It maintains a running average of squared gradients:

\[ \mathbf{s} \leftarrow \rho\,\mathbf{s} + (1-\rho)\,\left(\nabla_\theta \mathcal{L}\right)^2, \qquad \theta \leftarrow \theta - \frac{\eta}{\sqrt{\mathbf{s} + \varepsilon}} \nabla_\theta \mathcal{L}, \]

where \(\rho \approx 0.9\) and \(\varepsilon\) is a small stabilizing constant.

2.3 Learning Rate Schedules

A fixed learning rate is rarely optimal throughout training. Common schedules include step decay (multiply \(\eta\) by a factor after every \(k\) epochs), cosine annealing (\(\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})(1 + \cos(\pi t / T))\)), and warmup followed by decay (used in transformer training). The linear warmup phase prevents large gradient steps early in training when parameters are randomly initialized.

2.4 Regularization

Regularization encompasses techniques that reduce overfitting by constraining the hypothesis class or introducing noise during training.

L2 regularization (weight decay) adds a penalty \(\frac{\lambda}{2}\|\theta\|_2^2\) to the loss, encouraging small weights and effectively shrinking them toward zero. The gradient update becomes \(\theta \leftarrow (1 - \eta\lambda)\theta - \eta\nabla_\theta \mathcal{L}\). L1 regularization uses \(\lambda\|\theta\|_1\), which induces sparsity by driving many weights to exactly zero.

Layer normalization normalizes across features within a single example rather than across the batch dimension, making it suitable for recurrent networks and transformers where batch statistics are unreliable. For a single activation vector \(\mathbf{h} \in \mathbb{R}^d\):

\[ \text{LN}(\mathbf{h}) = \gamma \odot \frac{\mathbf{h} - \mu}{\sigma} + \beta, \quad \mu = \frac{1}{d}\sum_j h_j, \quad \sigma^2 = \frac{1}{d}\sum_j (h_j - \mu)^2. \]

Early stopping halts training when performance on a held-out validation set begins to degrade, implicitly constraining the effective number of training iterations. It is one of the most practical and widely used regularization strategies.

---

Chapter 3: Convolutional Neural Networks

3.1 The Convolution Operation

Images exhibit strong spatial structure: nearby pixels are correlated, and meaningful patterns (edges, textures, objects) recur across different positions. Convolutional neural networks exploit this structure through three key ideas: local connectivity, weight sharing, and pooling.

The number of learnable parameters in a convolutional layer with kernel size \(f\), \(C_\text{in}\) input channels, and \(C_\text{out}\) output channels is \(C_\text{out} \cdot C_\text{in} \cdot f^2 + C_\text{out}\), far fewer than a fully connected layer over the same spatial extent. Stride \(s\) reduces the output resolution by sampling every \(s\)-th position; padding preserves spatial dimensions by appending zeros around the input border.

3.2 Pooling

Pooling layers reduce the spatial resolution of feature maps, providing a form of spatial aggregation and mild translation invariance. Max pooling takes the maximum value in each local window; average pooling takes the mean. Global average pooling collapses each feature map to a single scalar, and is commonly used in modern architectures before the final classification layer.

3.3 Classic Architectures

LeNet-5 (LeCun et al., 1998) was the first successful CNN, comprising two convolutional layers with average pooling followed by fully connected layers. It achieved near-human performance on handwritten digit recognition.

AlexNet (Krizhevsky et al., 2012) demonstrated the power of deep CNNs on ImageNet, achieving a top-5 error of 15.3% versus 26.2% for the previous best. Its key contributions included ReLU activations, dropout, data augmentation, and training on GPUs. AlexNet used five convolutional layers and three fully connected layers.

VGGNet (Simonyan and Zisserman, 2014) established the principle that deeper networks using only \(3 \times 3\) convolutions outperform shallower networks with larger kernels. A stack of two \(3 \times 3\) convolutions has the same receptive field as one \(5 \times 5\) convolution but fewer parameters. VGG-16 and VGG-19 became standard feature extractors for transfer learning.

3.4 Residual Networks

Very deep networks (beyond ~20 layers) suffered from degradation: training accuracy paradoxically decreases with added depth. He et al. (2016) solved this with residual connections.

ResNets enabled training of networks with 50, 101, and 152 layers, setting records on ImageNet and COCO. The residual connection can be interpreted as unrolling a recurrent computation, as an ensemble of exponentially many shorter paths, or as enabling gradient superhighways through skip connections.

3.5 Efficient Convolutions and Object Detection

Depthwise separable convolutions, used in MobileNet and Xception, factorize a standard convolution into a depthwise convolution (one filter per input channel) followed by a \(1 \times 1\) pointwise convolution. This reduces computation by a factor of roughly \(1/C + 1/f^2\) compared to standard convolutions.

Object detection extends image classification to localizing multiple objects. Faster R-CNN uses a region proposal network (RPN) that slides a small network over the convolutional feature map to propose candidate bounding boxes, then classifies each proposal with a separate head. YOLO (You Only Look Once) frames detection as a single regression problem: the network directly predicts bounding boxes and class probabilities from the full image in one forward pass, enabling real-time detection.

---

Chapter 4: Recurrent Neural Networks

4.1 Sequence Modeling and Unfolded Computation

Many data modalities — text, speech, time series — are inherently sequential. A recurrent neural network processes sequences by maintaining a hidden state \(\mathbf{h}_t\) that summarizes the history up to time \(t\):

\[ \mathbf{h}_t = f\!\left(\mathbf{h}_{t-1},\, \mathbf{x}_t;\, \theta\right). \]

The same parameters \(\theta\) are applied at every time step, a form of weight sharing across time. Unfolding the computation across a sequence of length \(T\) produces a deep directed graph from \(\mathbf{x}_1, \ldots, \mathbf{x}_T\) to a sequence of predictions \(\hat{y}_1, \ldots, \hat{y}_T\).

For a vanilla RNN, the update equations are:

\[ \mathbf{h}_t = \tanh\!\left(\mathbf{W}_h \mathbf{h}_{t-1} + \mathbf{W}_x \mathbf{x}_t + \mathbf{b}\right), \qquad \hat{y}_t = \text{softmax}\!\left(\mathbf{W}_y \mathbf{h}_t + \mathbf{c}\right). \]

4.2 Backpropagation Through Time

Training RNNs requires computing gradients of the loss summed over time steps with respect to shared weights. Backpropagation through time (BPTT) unrolls the network over all \(T\) steps and applies standard backpropagation. The gradient of the loss with respect to \(\mathbf{h}_1\) involves a product of Jacobians:

\[ \frac{\partial \mathbf{h}_T}{\partial \mathbf{h}_1} = \prod_{t=2}^T \frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_{t-1}}. \]

4.3 Long Short-Term Memory

The LSTM (Hochreiter and Schmidhuber, 1997) addresses vanishing gradients by introducing a cell state \(\mathbf{c}_t\) that can carry information across many time steps with minimal interference. Four gates regulate information flow:

4.4 Gated Recurrent Units

The GRU (Cho et al., 2014) simplifies the LSTM by merging the forget and input gates into a single update gate and eliminating the separate cell state:

\[ \mathbf{z}_t = \sigma\!\left(\mathbf{W}_z \left[\mathbf{h}_{t-1}; \mathbf{x}_t\right]\right), \]\[ \mathbf{r}_t = \sigma\!\left(\mathbf{W}_r \left[\mathbf{h}_{t-1}; \mathbf{x}_t\right]\right), \]\[ \tilde{\mathbf{h}}_t = \tanh\!\left(\mathbf{W}_h \left[\mathbf{r}_t \odot \mathbf{h}_{t-1};\, \mathbf{x}_t\right]\right), \]\[ \mathbf{h}_t = (1 - \mathbf{z}_t) \odot \mathbf{h}_{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t. \]

The update gate \(\mathbf{z}_t\) interpolates between the previous hidden state and the new candidate, while the reset gate \(\mathbf{r}_t\) controls how much prior information is used. GRUs have fewer parameters than LSTMs and often perform comparably on language tasks.

4.5 Encoder–Decoder Sequence-to-Sequence Models

For tasks such as machine translation, the input and output sequences may have different lengths. The seq2seq architecture (Sutskever et al., 2014) uses an encoder RNN (LSTM or GRU) to read the input sequence and compress it into a fixed-dimensional context vector \(\mathbf{c} = \mathbf{h}_T^{\text{enc}}\), then a decoder RNN that generates the output sequence token by token, conditioned on \(\mathbf{c}\):

\[ \mathbf{h}_t^{\text{dec}} = f_{\text{dec}}\!\left(\mathbf{h}_{t-1}^{\text{dec}},\, \hat{y}_{t-1},\, \mathbf{c}\right). \]

A key limitation is the information bottleneck: compressing long sequences into a single vector. Attention mechanisms (Chapter 5) resolve this by allowing the decoder to dynamically focus on different encoder positions.

---

Chapter 5: Attention Mechanisms and Transformers

5.1 Scaled Dot-Product Attention

The attention mechanism allows a model to selectively attend to different parts of its input, producing a weighted combination of value vectors based on query–key similarity.

For language models with causal (autoregressive) generation, the attention matrix is masked so that position \(i\) cannot attend to positions \(j > i\), setting those entries to \(-\infty\) before the softmax.

5.2 Multi-Head Attention

5.3 Positional Encoding

Self-attention is permutation equivariant: it treats the input as a set, not a sequence. To inject positional information, the original Transformer adds a sinusoidal positional encoding to each token embedding:

\[ PE_{(pos, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d_\text{model}}}\right), \]\[ PE_{(pos, 2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d_\text{model}}}\right). \]

This encoding is deterministic, not learned, and allows the model to attend by relative position since \(PE_{pos+k}\) is a linear function of \(PE_{pos}\). Rotary positional embeddings (RoPE) and learned absolute embeddings are popular alternatives in modern architectures.

5.4 The Full Transformer Block

The full Transformer (Vaswani et al., 2017) stacks \(N\) such blocks. Its \(O(n^2 d)\) self-attention cost in sequence length \(n\) is a key scalability challenge addressed in Chapter 6.

5.5 BERT

BERT (Devlin et al., 2019) applies bidirectional transformer encoders to language understanding. It is pre-trained on two objectives:

Masked Language Modeling (MLM): 15% of input tokens are randomly masked, and the model predicts them from context. Unlike autoregressive language models, BERT can use both left and right context for each masked token.

Next Sentence Prediction (NSP): Given two segments A and B, BERT predicts whether B follows A in the original corpus. A special \([CLS]\) token prepends the input, and its final representation is used for classification tasks.

BERT is fine-tuned on downstream tasks (sentiment analysis, question answering, named entity recognition) by appending a task-specific head and training end-to-end on labeled data.

5.6 GPT and Autoregressive Language Modeling

GPT (Radford et al., 2018) uses a stack of transformer decoder blocks (causal self-attention only) trained as an autoregressive language model:

\[ p(x_1, x_2, \ldots, x_T) = \prod_{t=1}^T p(x_t \mid x_1, \ldots, x_{t-1}). \]

The model is trained by maximizing the log-likelihood over a large text corpus, with the same parameters used for all positions. GPT-2 (2019) scaled to 1.5B parameters and demonstrated that language models can generate coherent long-form text. GPT-3 (Brown et al., 2020) with 175B parameters introduced in-context learning: by conditioning on a few examples in the prompt (few-shot learning), the model performs competitively on many tasks without any gradient updates. Scaling laws observed that loss decreases as a power law in model size, dataset size, and compute, providing principled guidance for scaling decisions.

---

Chapter 6: Performers and Efficient Transformers

6.1 The Quadratic Bottleneck

Standard self-attention scales as \(O(n^2)\) in both time and memory with respect to sequence length \(n\), since the attention matrix \(QK^\top / \sqrt{d_k}\) is \(n \times n\). For long sequences (documents, high-resolution images, genomic sequences), this is prohibitive. Efficient transformer variants approximate or replace the full attention matrix.

6.2 Performers and FAVOR+

The Performer (Choromanski et al., 2021) approximates the full softmax attention kernel using random feature maps, achieving \(O(n)\) attention complexity.

6.3 Sparse Attention Patterns

Sparse transformers (Child et al., 2019) restrict each token to attend only to a local window and a set of stride-sampled global positions. The attention pattern is a union of local and strided components, reducing complexity to \(O(n\sqrt{n})\). Longformer combines a sliding window attention with global tokens (e.g., \([CLS]\)) that attend to the full sequence, scaling to documents of thousands of tokens.

BigBird adds random attention edges to local and global patterns, enabling \(O(n)\) attention while retaining theoretical properties of full attention (Turing-completeness).

6.4 Flash Attention

Flash Attention (Dao et al., 2022) achieves exact (not approximate) attention while reducing memory from \(O(n^2)\) to \(O(n)\) by restructuring the computation to exploit GPU SRAM hierarchy. Rather than materializing the full \(n \times n\) attention matrix in HBM (high-bandwidth memory), Flash Attention tiles the computation across blocks that fit in SRAM, fusing the softmax normalization with the matrix product. This yields a 2–4× wall-clock speedup in practice, and has become the standard attention implementation in modern LLM training frameworks.

---

Chapter 7: Autoencoders and Variational Autoencoders

7.1 Deterministic Autoencoders

An autoencoder is a neural network trained to reconstruct its input via a compressed bottleneck representation. It consists of an encoder \(f_\phi : \mathcal{X} \to \mathcal{Z}\) and a decoder \(g_\theta : \mathcal{Z} \to \mathcal{X}\), trained to minimize a reconstruction loss:

\[ \mathcal{L}(\phi, \theta) = \mathbb{E}_{\mathbf{x}}\!\left[\left\|g_\theta(f_\phi(\mathbf{x})) - \mathbf{x}\right\|^2\right]. \]

When the bottleneck dimension \(\dim(\mathcal{Z}) < \dim(\mathcal{X})\), the encoder is forced to learn a compressed representation. Deterministic autoencoders learn a deterministic mapping and can suffer from discontinuous or irregular latent spaces, making them unsuitable for generation.

7.2 The Variational Autoencoder

The Variational Autoencoder (VAE; Kingma and Welling, 2014) frames representation learning as probabilistic inference, endowing the latent space with a regular, continuous structure suitable for generation.

7.3 The Evidence Lower Bound

Training the VAE maximizes a lower bound on the log-marginal likelihood. By Jensen’s inequality applied to the concavity of the logarithm:

\[ \log p_\theta(\mathbf{x}) \geq \mathbb{E}_{q_\phi(\mathbf{z}|\mathbf{x})}\!\left[\log p_\theta(\mathbf{x} \mid \mathbf{z})\right] - D_{\text{KL}}\!\left(q_\phi(\mathbf{z}|\mathbf{x}) \,\|\, p(\mathbf{z})\right). \]

This lower bound, called the ELBO (Evidence Lower BOund), decomposes into a reconstruction term (expected log-likelihood of the data given the latent code) and a regularization term (KL divergence between the approximate posterior and the prior). For Gaussian \(q_\phi\) and prior \(p\), the KL divergence has the closed form:

\[ D_{\text{KL}}\!\left(\mathcal{N}(\mu, \sigma^2 I) \| \mathcal{N}(0, I)\right) = \frac{1}{2}\sum_{j}\!\left(\mu_j^2 + \sigma_j^2 - \log \sigma_j^2 - 1\right). \]

7.4 The Reparameterization Trick

Naive Monte Carlo estimation of the reconstruction term gradient with respect to \(\phi\) fails because \(\mathbf{z}\) is sampled from \(q_\phi\). The reparameterization trick expresses \(\mathbf{z} = \mu_\phi(\mathbf{x}) + \sigma_\phi(\mathbf{x}) \odot \boldsymbol{\varepsilon}\) where \(\boldsymbol{\varepsilon} \sim \mathcal{N}(0,I)\), making \(\mathbf{z}\) a deterministic function of \(\phi\) and a noise variable independent of \(\phi\). Gradients can then be backpropagated through \(\mathbf{z}\) with respect to \(\phi\).

---

Chapter 8: Deep Generative Models

8.1 Generative Adversarial Networks

A Generative Adversarial Network (GAN; Goodfellow et al., 2014) frames generative modeling as a two-player minimax game between a generator \(G\) and a discriminator \(D\).

8.2 Wasserstein GAN

The original GAN objective can produce vanishing gradients when the generator is far from the data distribution (since \(D_{\text{JS}}\) saturates to \(\log 2\)). The Wasserstein GAN (WGAN; Arjovsky et al., 2017) uses the Earth Mover’s distance instead:

\[ W(p_{\text{data}}, p_G) = \sup_{\|f\|_L \leq 1} \mathbb{E}_{\mathbf{x} \sim p_{\text{data}}}\!\left[f(\mathbf{x})\right] - \mathbb{E}_{\mathbf{z}}\!\left[f(G(\mathbf{z}))\right], \]

where the supremum is over 1-Lipschitz functions. The critic (discriminator without sigmoid) must satisfy the Lipschitz constraint, enforced either by weight clipping (original WGAN) or gradient penalty (WGAN-GP):

\[ \mathcal{L}_{\text{GP}} = \lambda\, \mathbb{E}_{\hat{\mathbf{x}}}\!\left[\left(\|\nabla_{\hat{\mathbf{x}}} D(\hat{\mathbf{x}})\|_2 - 1\right)^2\right], \]

where \(\hat{\mathbf{x}}\) is sampled uniformly along lines between real and generated samples. WGAN provides meaningful loss curves that correlate with sample quality.

8.3 Conditional GANs and Image Translation

A Conditional GAN (Mirza and Osindero, 2014) conditions both generator and discriminator on auxiliary information \(\mathbf{y}\) (class label or another image):

\[ \min_G \max_D\; \mathbb{E}\!\left[\log D(\mathbf{x}, \mathbf{y})\right] + \mathbb{E}\!\left[\log\!\left(1 - D(G(\mathbf{z}, \mathbf{y}), \mathbf{y})\right)\right]. \]

pix2pix (Isola et al., 2017) learns image-to-image translation (e.g., semantic labels to street scenes) by conditioning the generator on a full input image, using a U-Net generator and PatchGAN discriminator that classifies \(N \times N\) patches as real or fake. CycleGAN (Zhu et al., 2017) learns unpaired image translation by training two generators \(G: X \to Y\) and \(F: Y \to X\) with a cycle consistency loss \(\|F(G(x)) - x\|_1 + \|G(F(y)) - y\|_1\).

Common GAN training instabilities include mode collapse (the generator produces only a few distinct samples), oscillations, and sensitivity to hyperparameters. Spectral normalization, progressive growing, and better loss functions (hinge loss, relativistic loss) mitigate these issues.

---

Chapter 9: Deep Reinforcement Learning

9.1 Markov Decision Processes

The state value function \(V^\pi(s) = \mathbb{E}_\pi\!\left[G_t \mid S_t = s\right]\) and action-value function \(Q^\pi(s,a) = \mathbb{E}_\pi\!\left[G_t \mid S_t = s, A_t = a\right]\) satisfy the Bellman equations:

\[ V^\pi(s) = \sum_a \pi(a|s)\sum_{s'} P(s'|s,a)\!\left[R(s,a) + \gamma V^\pi(s')\right], \]\[ Q^\pi(s,a) = \sum_{s'} P(s'|s,a)\!\left[R(s,a) + \gamma \sum_{a'} \pi(a'|s') Q^\pi(s',a')\right]. \]

9.2 Q-Learning and Deep Q-Networks

Q-learning is an off-policy temporal difference algorithm that estimates the optimal action-value function:

\[ Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha\!\left[r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)\right]. \]

DQN (Mnih et al., 2015) approximates \(Q(s,a;\theta)\) with a deep neural network. Two key innovations enable stable training:

Experience replay: Transitions \((s, a, r, s')\) are stored in a replay buffer and sampled uniformly for training, breaking temporal correlations and improving data efficiency.

Target network: A periodically updated copy \(\hat{Q}(s,a;\theta^-)\) provides stable regression targets, preventing oscillations from rapidly changing targets. The loss is:

\[ \mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}}\!\left[\left(r + \gamma \max_{a'} \hat{Q}(s',a';\theta^-) - Q(s,a;\theta)\right)^2\right]. \]

9.3 Policy Gradient Methods

Policy gradient methods directly optimize a parameterized policy \(\pi_\theta(a|s)\). The REINFORCE algorithm (Williams, 1992) uses the policy gradient theorem:

\[ \nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[\sum_t \nabla_\theta \log \pi_\theta(a_t|s_t)\, G_t\right], \]

estimating the expectation via Monte Carlo rollouts. A baseline \(b(s_t)\) is subtracted to reduce variance: \(G_t - b(s_t)\).

Actor-critic methods combine a policy network (actor) with a value network (critic) that estimates \(V^\pi(s)\) or \(Q^\pi(s,a)\), using the advantage \(A(s,a) = Q(s,a) - V(s)\) as a lower-variance gradient signal.

Proximal Policy Optimization (PPO; Schulman et al., 2017) constrains policy updates to remain close to the current policy using a clipped surrogate objective:

\[ \mathcal{L}^{\text{CLIP}}(\theta) = \mathbb{E}_t\!\left[\min\!\left(r_t(\theta)\hat{A}_t,\; \text{clip}(r_t(\theta), 1-\varepsilon, 1+\varepsilon)\hat{A}_t\right)\right], \]

where \(r_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\theta_\text{old}}(a_t|s_t)\) is the probability ratio. Clipping prevents excessively large updates, providing stability without the computational overhead of KL-constrained trust region methods.

---

Chapter 10: Graph Neural Networks

10.1 Graphs and the Need for Specialized Architectures

Standard deep learning architectures (MLPs, CNNs, RNNs) assume inputs with fixed grid or sequence structure. Many real-world datasets are naturally represented as graphs: social networks, molecular structures, citation networks, knowledge graphs. A graph \(G = (V, E)\) consists of a set of nodes \(v \in V\) with feature vectors \(\mathbf{x}_v \in \mathbb{R}^d\) and edges \((u,v) \in E\). Graph neural networks (GNNs) learn node or graph-level representations by propagating information along edges.

10.2 Message Passing Framework

10.3 Graph Convolutional Networks

10.4 GraphSAGE and Graph Attention Networks

GraphSAGE (Hamilton et al., 2017) uses inductive learning by sampling a fixed-size neighborhood and aggregating with a learned aggregator (mean, max, or LSTM):

\[ \mathbf{h}_v^{(k)} = \sigma\!\left(\mathbf{W}^{(k)} \cdot \text{Concat}\!\left(\mathbf{h}_v^{(k-1)},\; \text{Aggregate}\!\left(\left\{\mathbf{h}_u^{(k-1)}\right\}_{u \in \mathcal{N}_s(v)}\right)\right)\right), \]

where \(\mathcal{N}_s(v)\) is a sampled subset of neighbors.

GNNs have achieved state-of-the-art results in molecular property prediction (predicting quantum chemical properties of small molecules), protein structure analysis, drug discovery, and recommendation systems. A key challenge is over-smoothing: with many GNN layers, node representations converge to indistinguishable vectors; in practice 2–4 layers are most effective.

---

Chapter 11: Diffusion Models

11.1 The Forward Diffusion Process

Diffusion models (Ho et al., 2020; Song and Ermon, 2020) define a generative model by learning to reverse a gradual noising process. The forward process \(q\) progressively adds Gaussian noise to a data sample \(\mathbf{x}_0\) over \(T\) steps according to a fixed variance schedule \(\{\beta_t\}_{t=1}^T\):

11.2 The Reverse Process and ELBO

The reverse process is a learned Markov chain that denoises from \(\mathbf{x}_T\) back to \(\mathbf{x}_0\):

11.3 DDIM and Deterministic Sampling

Denoising Diffusion Implicit Models (Song et al., 2021) derive a non-Markovian forward process with the same marginals as DDPM, yielding deterministic sampling trajectories. Given \(\mathbf{x}_t\), the DDIM update is:

\[ \mathbf{x}_{t-1} = \sqrt{\bar{\alpha}_{t-1}} \underbrace{\frac{\mathbf{x}_t - \sqrt{1-\bar{\alpha}_t}\,\epsilon_\theta(\mathbf{x}_t,t)}{\sqrt{\bar{\alpha}_t}}}_{\text{predicted }\mathbf{x}_0} + \sqrt{1-\bar{\alpha}_{t-1}}\,\epsilon_\theta(\mathbf{x}_t,t). \]

Crucially, DDIM allows generation with far fewer steps (e.g., 50 rather than 1000) without significant quality loss, and enables a deterministic encoding–decoding useful for image editing.

11.4 Classifier-Free Guidance

Classifier-free guidance (Ho and Salimans, 2022) enables conditional generation without a separate classifier. A single model is trained both unconditionally (by randomly dropping the conditioning signal \(\mathbf{c}\) with probability \(p_{\text{uncond}}\)) and conditionally. At inference time, the effective score is:

\[ \tilde{\epsilon}_\theta(\mathbf{x}_t, \mathbf{c}) = \epsilon_\theta(\mathbf{x}_t, \varnothing) + w\!\left(\epsilon_\theta(\mathbf{x}_t, \mathbf{c}) - \epsilon_\theta(\mathbf{x}_t, \varnothing)\right), \]

where \(w > 1\) is the guidance scale. Larger \(w\) produces samples more faithful to the conditioning signal at the cost of diversity.

11.5 Score-Based Generative Models

Score-based models (Song and Ermon, 2019) learn the score function \(\nabla_\mathbf{x} \log p(\mathbf{x})\) — the gradient of the log-density with respect to the input — using denoising score matching. Samples are generated by running Langevin dynamics:

\[ \mathbf{x}_{i+1} = \mathbf{x}_i + \frac{\delta}{2}\nabla_\mathbf{x} \log p(\mathbf{x}_i) + \sqrt{\delta}\,\mathbf{z}, \quad \mathbf{z} \sim \mathcal{N}(0,I). \]

Song et al. (2021) unified score-based and diffusion models under a continuous-time stochastic differential equation (SDE) framework, enabling new samplers and exact likelihood computation.

---

Chapter 12: Large Language Models

12.1 From Pre-Training to Instruction Following

GPT-3 demonstrated remarkable few-shot capabilities but remained a general-purpose text predictor. Instruction tuning (Wei et al., 2022) fine-tunes a pre-trained LLM on a large collection of tasks formatted as natural language instructions, dramatically improving zero-shot generalization. Models such as FLAN-T5 and instruction-tuned GPT-4 follow novel instructions without explicit demonstrations.

Reinforcement Learning from Human Feedback (RLHF; Christiano et al., 2017; Ouyang et al., 2022) aligns LLMs with human preferences through three stages:

Direct Preference Optimization (DPO; Rafailov et al., 2023) bypasses explicit reward model training by directly optimizing the preference objective, simplifying the RLHF pipeline.

12.2 Chain-of-Thought Prompting

Standard prompting asks LLMs to produce answers directly. Chain-of-thought (CoT; Wei et al., 2022) prompting elicits multi-step reasoning by including worked examples with intermediate reasoning steps in the few-shot prompt. Models prompted with CoT dramatically improve performance on arithmetic, commonsense, and symbolic reasoning benchmarks. Zero-shot CoT appends the phrase “Let’s think step by step” to the query, prompting reasoning without demonstrations. Self-consistency samples multiple reasoning chains and aggregates by majority voting, improving reliability.

12.3 Emergent Abilities and Scaling Laws

Emergent abilities are capabilities that appear abruptly above a model scale threshold without continuous improvement below it (Wei et al., 2022). Examples include multi-step arithmetic, multi-lingual translation, and chain-of-thought reasoning. Whether emergence is a genuine phase transition or an artifact of evaluation metrics remains debated.

12.4 Mixture of Experts

Mixture of Experts (MoE) replaces the dense feed-forward sublayer in each transformer block with a collection of \(E\) expert networks \(\{f_e\}_{e=1}^E\) and a router that routes each token to the top-\(k\) experts:

\[ \text{MoE}(\mathbf{x}) = \sum_{e \in \text{Top}_k(G(\mathbf{x}))} G_e(\mathbf{x})\, f_e(\mathbf{x}), \]

where \(G(\mathbf{x}) = \text{softmax}(\mathbf{x} W_g + \text{noise})\). With \(E\) experts and \(k=2\), the model activates only \(2/E\) of its parameters for each token, allowing parameter count to scale without proportional compute increase. Switch Transformer (Fedus et al., 2022) and Mixtral-8x7B (Mistral, 2023) demonstrated that MoE architectures achieve competitive quality at significantly lower inference cost.

12.5 Long Context and Retrieval Augmentation

Modern LLMs extend context windows to 128K or more tokens using techniques like RoPE scaling, ALiBi positional biases, and sliding window attention. Retrieval-Augmented Generation (RAG; Lewis et al., 2020) decouples parametric knowledge (stored in weights) from non-parametric knowledge (retrieved at inference time from an external corpus). A retriever (often a bi-encoder dense retrieval model) fetches relevant documents given the query; the generator LLM conditions on retrieved text to produce the final answer. RAG improves factual accuracy, enables access to up-to-date information, and reduces hallucinations without requiring costly retraining.