Session 3 - Neural Networks

Neural Networks

Christian Cabrera Jojoa

Assistant Research Professor

Department of Computer Science and Technology

University of Cambridge

chc79@cam.ac.uk

Session 3 - Neural Networks

Last Time

Session 3 - Neural Networks

Last Time

Linear regression

Regression Fit
Linear regression fit.

Linear classifiers

SVM Separating Hyperplanes
SVM Separating Hyperplanes - Cyc, Public domain, via Wikimedia Commons.

Hypothesis space and loss

Hypothesis Space for Regression
Example functions described using a linear model.

Session 3 - Neural Networks

Last Time

Gradient descent

Gradient Descent Algorithm
Gradient Descent Algorithm - Jacopo Bertolotti, CC0, via Wikimedia Commons.

is the learning rate

Underfitting and overfitting

Overfitting
Overfitting - ThirdOrderLogic, CC BY 4.0 , via Wikimedia Commons

too simple or too complex

The perceptron

Perceptron Architecture
Perceptron Architecture.

A linear classifier

Session 3 - Neural Networks

Neural Networks

Session 3 - Neural Networks

Neural Networks

Neural networks extend the perceptron by introducing multiple layers of interconnected neurons (i.e., multi-layer perceptron), enabling the learning of complex, non-linear relationships in data.

Session 3 - Neural Networks

Neural Networks

Perceptron Architecture
Perceptron Architecture.

From simple linear regression to the perceptron:


Session 3 - Neural Networks

Neural Networks

Hypothesis Space for Regression

From simple linear regression to the perceptron:


Session 3 - Neural Networks

Neural Networks

XOR Problem
XOR Problem - Andreas Maier, CC BY 4.0 , via Wikimedia Commons

XOR is not linearly separable (Minsky & Papert, 1969)


Perceptron with a general activation :


Basis functions : a richer hypothesis space

Session 3 - Neural Networks

Neural Networks

Richer hypothesis spaces:

Basis Functions
Basis Functions - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Neural Network
Neural Network Architecture.

Multilayer perceptron (MLP)



  • Fix the number of basis functions in advance
  • Basis functions are adaptive and their parameters can be updated during training

Training is costly but inference is cheap

Session 3 - Neural Networks

Neural Networks

Neural Network
Neural Network Architecture.

Multilayer perceptron (MLP)



  • Input Layer: Receives the input features
  • Hidden Layers: Process information through weighted connections
  • Output Layer: Produces the final prediction
  • Activation Functions: Introduce non-linearity
Session 3 - Neural Networks

Neural Networks

Neural Network
Neural Network Architecture.

Multilayer perceptron (MLP)



The Universal Approximation theorem states that a neural network with one hidden layer can approximate any continuous function on a compact subset of , given sufficient neurons in the hidden layer because neural networks form complex decision boundaries through the combination of linear transformations and non-linear activation functions.

Session 3 - Neural Networks

Neural Networks

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).

Feedforward Neural Network:

  • Tasks: regression and classification

  • Data: fixed-length feature vectors such as tabular records and engineered features

  • Key idea: fully connected layers learn the basis functions from data

Limitation: ignores spatial and temporal structure in the input

Session 3 - Neural Networks

Neural Networks

Feedforward Neural Network:

First, recall the general form of a linear model with nonlinear basis functions:


where is a nonlinear activation function (e.g., identity for regression, sigmoid for classification), are basis functions, and are weights.

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Feedforward Neural Network:

First, recall the general form of a linear model with nonlinear basis functions:


where is a nonlinear activation function (e.g., identity for regression, sigmoid for classification), are basis functions, and are weights.


In neural networks, the basis functions themselves are parameterized and learned. The network is constructed as a sequence of transformations.

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Feedforward Neural Network:

1. Linear combination of inputs (first layer):

where and are the weights from input to hidden unit , is the bias for hidden unit .

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Feedforward Neural Network:

1. Linear combination of inputs (first layer):

where and are the weights from input to hidden unit , is the bias for hidden unit .

2. Basis function of the inputs or nonlinear activation (hidden layer):

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Feedforward Neural Network:

2. Basis function of the inputs or nonlinear activation (hidden layer):


The designer chooses (sigmoid, tanh, or ReLU). Every hidden unit uses that same curve.

and set the centre and the width of .


Training learns these weights. The curve stays the one the designer chose.

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Feedforward Neural Network:

3. Linear combination of hidden activations (output layer):

where and are the weights from hidden unit to output unit , is the bias for output unit . are the learned features. scales them and does not move the curves.

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Feedforward Neural Network:

3. Linear combination of hidden activations (output layer):

where and are the weights from hidden unit to output unit , is the bias for output unit . are the learned features. scales them and does not move the curves.

The activations is transformed using an appropriate activation function to produce the final network outputs . For example, for regression problems or for binary classification problems.

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Feedforward Neural Network:

The overall network function combines these stages. For sigmoidal output unit activation functions, takes the form:

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Feedforward Neural Network:

The overall network function combines these stages. For sigmoidal output unit activation functions, takes the form:


The bias parameters can be absorbed:

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Feedforward Neural Network:

The overall network function combines these stages. For sigmoidal output unit activation functions, takes the form:


The bias parameters can be absorbed:


are continuous functions. The neural network is differentiable with respect to the parameters .

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

An activation function is a fixed curve applied to one unit's score. On a hidden unit, builds the basis function . On an output unit, it turns into the prediction .

Session 3 - Neural Networks

Neural Networks

Sigmoid Function:


Range:

Derivative:


Logistic Curve

Logistic Curve - Qef, Public domain, via Wikimedia Commons.

Hyperbolic Tangent:


Range:

Derivative:


Hyperbolic Tangent

Hyperbolic Tangent - Geek3, CC BY-SA 3.0 , via Wikimedia Commons.

ReLU (Rectified Linear Unit):


Range:

Derivative:


Ramp Function

Ramp Function - Qef, Public domain, via Wikimedia Commons.
Session 3 - Neural Networks

Neural Networks

Choosing :

  • Nonlinear: a linear makes the network a linear model

  • Derivative: training follows the slope of . A step is flat almost everywhere

  • Saturation: sigmoid and tanh flatten in the tails. ReLU keeps slope 1 for positive scores, which helps deeper networks

  • Range: tanh is centred at 0. A sigmoid hidden unit is always positive

Output activation follows the task. Identity for regression, sigmoid for two classes, softmax for more than two classes.

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Training Process:


Given a training set of example input-output pairs


Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Training Process:


Given a training set of example input-output pairs



Each pair was generated by an unknown function :

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Training Process:


Given a training set of example input-output pairs



Each pair was generated by an unknown function :


We want to find a hypothesis that minimises the error function:

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Training Process (regression problem):


Giving a probabilistic interpretation to the network outputs:



where is the precision (i.e. inverse variance ).

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Training Process (regression problem):


Giving a probabilistic interpretation to the network outputs:



where is the precision (i.e. inverse variance ).

For an i.i.d. training set, the likelihood function corresponds to:

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Training Process (regression problem):


For an i.i.d. training set, the likelihood function corresponds to:

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Training Process (regression problem):


For an i.i.d. training set, the likelihood function corresponds to:


Maximising the likelihood function is equivalent to minimising the sum-of-squares error function given by:

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Training Process (binary classification):

The network output is:

We use a single target variable such that denotes class 1 and denotes class 2.

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Training Process (binary classification):

The network output is:

We use a single target variable such that denotes class 1 and denotes class 2. We interpret as the conditional probability distribution of targets given inputs:

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Training Process (binary classification):

The network output is:

We use a single target variable such that denotes class 1 and denotes class 2. We interpret as the conditional probability distribution of targets given inputs:

For an i.i.d. training, the error function is the cross-entropy error:

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).
Session 3 - Neural Networks

Neural Networks

Gradient Descent Algorithm
Gradient Descent Algorithm - Jacopo Bertolotti, CC0, via Wikimedia Commons.
Session 3 - Neural Networks

Neural Networks

Gradient Descent Algorithm
Gradient Descent Algorithm - Jacopo Bertolotti, CC0, via Wikimedia Commons.

We choose any starting point and then compute an estimate of the gradient. We then move a small amount in the steepest downhill direction, repeating until we converge on a point in the weight space with (local) minima loss.


Gradient Descent Algorithm:
Initialize w
repeat
    for each w[i] in w
        Compute gradient: g = ∇Loss(w[i])
        Update weight:   w[i] = w[i] - α * g
until convergence

The size of the step is given by the parameter , which regulates the behaviour of the gradient descent algorithm. This is a hyperparameter of the regression model we are training, usually called learning rate.

Session 3 - Neural Networks

Neural Networks

Error backpropagation is an efficient algorithm for computing gradients in neural networks using the chain rule of calculus.

Session 3 - Neural Networks

Neural Networks

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).

Error Backpropagation Algorithm:

Session 3 - Neural Networks

Neural Networks

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).

Error Backpropagation Algorithm:

1. Forward Pass to compute all activations and outputs for an input vector.

Session 3 - Neural Networks

Neural Networks

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).

Error Backpropagation Algorithm:

1. Forward Pass to compute all activations and outputs for an input vector.

2. Error Evaluation for all outputs using:

Session 3 - Neural Networks

Neural Networks

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).

Error Backpropagation Algorithm:

1. Forward Pass to compute all activations and outputs for an input vector.

2. Error Evaluation for all outputs using:

3. Backward Pass: for each hidden unit in the network using:

Session 3 - Neural Networks

Neural Networks

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).

Error Backpropagation Algorithm:

1. Forward Pass to compute all activations and outputs for an input vector.

2. Error Evaluation for all outputs using:

3. Backward Pass: for each hidden unit in the network using:

4. Derivatives Evaluation for each parameter using:

Session 3 - Neural Networks

Neural Networks

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).

Gradient Descent Update Rule:

Session 3 - Neural Networks

Neural Networks

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).

Gradient Descent Update Rule:

Session 3 - Neural Networks

Neural Networks

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).

Gradient Descent Update Rule:


Session 3 - Neural Networks

Neural Networks

Two-layer Neural Network
Two-layer Neural Network - (Bishop, 2006).

Gradient Descent Update Rule:




Where is the learning rate.

Session 3 - Neural Networks

Neural Networks

Python implementation: scikit-learn

  • hidden_layer_sizes: number of hidden layers and units per layer

  • alpha: the penalty , not the learning rate

  • Defaults: ReLU activation and the Adam optimiser

from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

model = make_pipeline(
    StandardScaler(),
    MLPClassifier(
        hidden_layer_sizes=(8,),   # one hidden layer, 8 units
        alpha=1e-4,                # L2 penalty
        max_iter=3000,
        random_state=42,
    ),
)
model.fit(X_train, y_train)
print(model.score(X_test, y_test))
Session 3 - Neural Networks

Neural Networks

Python implementation: PyTorch

  • Forward pass: model(X)

  • Backpropagation: loss.backward()

  • Update: optimiser.step() applies 

import torch
from torch import nn

model = nn.Sequential(
    nn.Linear(D, 8),   # W(1), b(1)
    nn.Tanh(),         # h(.)
    nn.Linear(8, 1),   # W(2), b(2)
)
loss_fn = nn.BCEWithLogitsLoss()   # sigmoid + cross-entropy
optimiser = torch.optim.SGD(model.parameters(), lr=0.1)

for epoch in range(100):
    optimiser.zero_grad()
    loss = loss_fn(model(X).squeeze(1), y)
    loss.backward()
    optimiser.step()
Session 3 - Neural Networks

Deep Learning

Session 3 - Neural Networks

Deep Learning

Deep learning extends neural networks by using multiple hidden layers to learn hierarchical representations of data, enabling the automatic discovery of complex features.

Session 3 - Neural Networks

Deep Learning

Deep Neural Network
Deep Neural Network with multiple hidden layers - QuantuMechaniX8, CC0, via Wikimedia Commons
Session 3 - Neural Networks

Deep Learning

Deep Neural Network
Deep Neural Network with multiple hidden layers - QuantuMechaniX8, CC0, via Wikimedia Commons
Session 3 - Neural Networks

Deep Learning


Key Characteristics:


  • Multiple Hidden Layers: 3+ layers for deep architectures
  • Hierarchical Features: Each layer learns increasingly abstract representations
  • Automatic Feature Learning: No manual feature engineering required
  • Representation Learning: Learns useful representations from raw data
Deep Neural Network
Deep Neural Network with multiple hidden layers - QuantuMechaniX8, CC0, via Wikimedia Commons
Session 3 - Neural Networks

Deep Learning


Vanishing and Exploding Gradients: In deep networks, gradients can become very small (vanishing) or very large (exploding) during backpropagation:

Deep Neural Network
Deep Neural Network with multiple hidden layers - QuantuMechaniX8, CC0, via Wikimedia Commons
Session 3 - Neural Networks

Deep Learning


Vanishing and Exploding Gradients: In deep networks, gradients can become very small (vanishing) or very large (exploding) during backpropagation:

  • Proper Weight Initialization: Xavier/Glorot initialization
  • Batch Normalization: Normalize activations during training
  • Modern Optimizers: Adam, RMSprop with adaptive learning rates

  • Regularisation: Dropout, , early stopping, and augmentation techniques

  • Network Architectures: Different architectures for different problems and data types
Deep Neural Network
Deep Neural Network with multiple hidden layers - QuantuMechaniX8, CC0, via Wikimedia Commons
Session 3 - Neural Networks

Deep Learning

Weight Initialization:


Xavier/Glorot Initialization:



He Initialization (for ReLU):



Where and are the number of input and output neurons respectively.

Deep Neural Network
Deep Neural Network with multiple hidden layers - QuantuMechaniX8, CC0, via Wikimedia Commons
Session 3 - Neural Networks

Deep Learning

Batch normalisation:





Where (i.e., scale) and (i.e., shift) are learnable parameters, and is a small constant for numerical stability.

Deep Neural Network
Deep Neural Network with multiple hidden layers - QuantuMechaniX8, CC0, via Wikimedia Commons
Session 3 - Neural Networks

Deep Learning

Modern Optimisers:


Optimisers train models by efficiently navigating the loss landscape to find optimal parameters. They help in accelerating convergence, avoiding local minima, and improving generalisation.

  • Stochastic Gradient Descent (SGD): Updates parameters using a subset of data and reduces computation time.
  • Adam: Adaptively adjust the learning rate for each parameter.
Gradient Descent Algorithm
Gradient Descent Algorithm - Jacopo Bertolotti, CC0, via Wikimedia Commons.
Session 3 - Neural Networks

Deep Learning

Regularization Techniques:


Dropout: Randomly deactivate neurons during training


Regularization: Add penalty to loss function


Early Stopping: Stop training when validation loss increases


Data Augmentation: Create additional training examples through transformations

Session 3 - Neural Networks

Deep Learning

Deep Neural Network
Deep Neural Network with multiple hidden layers - QuantuMechaniX8, CC0, via Wikimedia Commons

Architecture

  • Number of layers : depth of the network

  • Hidden units : width of each hidden layer

  • Activation : sigmoid, tanh, or ReLU

Training

  • Learning rate : size of each gradient descent step

  • Epochs and batch size : passes over the data and examples per update

  • Optimiser: SGD or Adam

  • Regularisation strength : weight of the complexity penalty

Session 3 - Neural Networks

Neural Network Architectures

Session 3 - Neural Networks

Recurrent Neural Networks

Unfolded RNN
Unfolded recurrent neural network - fdeloche, CC BY-SA 4.0 , via Wikimedia Commons

  • Tasks: forecasting, sequence classification, language modelling, and speech recognition

  • Data: time series, text, audio, and sensor signals

  • Key idea: a hidden state carries memory from one time step to the next

Limitation of the MLP: sequences have an order and a variable length, and an MLP expects a fixed-size input

Session 3 - Neural Networks

Recurrent Neural Networks

Unfolded RNN
Unfolded recurrent neural network - fdeloche, CC BY-SA 4.0 , via Wikimedia Commons

Example: the next character

  • Sequence: the 11 characters in "el temblor "

  • Each step: one character, as a one-hot vector

  • Hidden state: memory of the characters already read

  • Task: the next character, m

Session 3 - Neural Networks

Recurrent Neural Networks

Python implementation: PyTorch

A window of characters to the next character

  • Layers: nn.RNN, then a score for each character

  • Clipping: clip_grad_norm_ before each update

from torch import nn

class SequenceModel(nn.Module):
    def __init__(self, n_chars, n_hidden):
        super().__init__()
        self.rnn = nn.RNN(n_chars, n_hidden, batch_first=True)
        self.head = nn.Linear(n_hidden, n_chars)

    def forward(self, x):                 # x: (batch, T, n_chars)
        out, h_T = self.rnn(x)            # tanh hidden state
        return self.head(h_T[-1])         # scores for the next character

# in the training loop
nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
Session 3 - Neural Networks

Autoencoders

Autoencoder
Autoencoder schema - Michela Massi, CC BY-SA 4.0 , via Wikimedia Commons

  • Tasks: dimensionality reduction, denoising, anomaly detection, and generation

  • Data: unlabelled inputs such as images, signals, and feature vectors

  • Key idea: learn to reconstruct the input through a small latent code

Session 3 - Neural Networks

Autoencoders

Autoencoder
Autoencoder schema - Michela Massi, CC BY-SA 4.0 , via Wikimedia Commons

Example: handwritten digit images

  • Input: one image, with no label

  • Latent code: a short vector

  • Output: the reconstructed image

  • Task: reconstruction, such as spotting an unusual digit from a large error

Session 3 - Neural Networks

Autoencoders

Python implementation: PyTorch

An autoencoder for vectors of size

  • Latent code: smaller than

  • Loss: squared error between the input and the reconstruction

  • Target: the input itself, so no labels are needed

from torch import nn

class Autoencoder(nn.Module):
    def __init__(self, D, M, latent_dim):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(D, M), nn.ReLU(), nn.Linear(M, latent_dim))
        self.dec = nn.Sequential(nn.Linear(latent_dim, M), nn.ReLU(), nn.Linear(M, D))

    def forward(self, x):
        z = self.enc(x)
        return self.dec(z)

def ae_loss(x_hat, x):
    return ((x - x_hat) ** 2).sum()
Session 3 - Neural Networks

Variational Autoencoders

Variational Autoencoder
Variational autoencoder structure - EugenioTL, CC BY-SA 4.0 , via Wikimedia Commons

  • Tasks: generating new samples and learning a continuous latent space

  • Data: unlabelled inputs such as images, signals, and feature vectors

  • Key idea: the encoder outputs a distribution over the latent code

Session 3 - Neural Networks

Variational Autoencoders

Variational Autoencoder
Variational autoencoder structure - EugenioTL, CC BY-SA 4.0 , via Wikimedia Commons

Example: the same digit images

  • Input: one image, with no label

  • Latent code: a sample from a learned distribution

  • Decode: a new image from a drawn code

  • Task: generating new digits

Session 3 - Neural Networks

Autoencoders

Python implementation: PyTorch

A VAE for vectors of size

  • log_var: for numerical stability

  • KL term: closed form for Gaussians

  • Added here: mu, log_var, and the KL term

import torch
from torch import nn

class VAE(nn.Module):
    def __init__(self, D, M, latent_dim):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(D, M), nn.ReLU())
        self.mu = nn.Linear(M, latent_dim)
        self.log_var = nn.Linear(M, latent_dim)
        self.dec = nn.Sequential(nn.Linear(latent_dim, M), nn.ReLU(), nn.Linear(M, D))

    def forward(self, x):
        h = self.enc(x)
        mu, log_var = self.mu(h), self.log_var(h)
        z = mu + torch.exp(0.5 * log_var) * torch.randn_like(mu)   # reparameterisation
        return self.dec(z), mu, log_var

def vae_loss(x_hat, x, mu, log_var):
    rec = ((x - x_hat) ** 2).sum()
    kl = -0.5 * (1 + log_var - mu ** 2 - log_var.exp()).sum()
    return rec + kl
Session 3 - Neural Networks

Graph Neural Networks

Message Passing
Message passing neural network - NickDiCicco, CC BY-SA 4.0 , via Wikimedia Commons

  • Tasks: node classification, link prediction, and graph classification

  • Data: graphs such as social networks, molecules, and road or sensor networks

  • Key idea: each node updates its representation with messages from its neighbours

Session 3 - Neural Networks

Graph Neural Networks

Message Passing
Message passing neural network - NickDiCicco, CC BY-SA 4.0 , via Wikimedia Commons

Example: a sensor network, such as seismic stations

  • Nodes: sensors

  • Edges: nearby sensors are connected

  • Node features: recent readings

  • Task: node regression, such as the shaking at each station

Session 3 - Neural Networks

Graph Neural Networks

Python implementation: PyTorch

A GCN layer in plain PyTorch

  • PyTorch Geometric: GCNConv implements this layer

import torch
from torch import nn

class GCNLayer(nn.Module):
    def __init__(self, n_in, n_out):
        super().__init__()
        self.linear = nn.Linear(n_in, n_out)          # W(l)

    def forward(self, H, A):                          # H: (N, n_in), A: (N, N)
        A_tilde = A + torch.eye(A.shape[0])           # self-loops
        D_inv_sqrt = torch.diag(A_tilde.sum(dim=1).pow(-0.5))
        A_hat = D_inv_sqrt @ A_tilde @ D_inv_sqrt     # degree normalisation
        return torch.relu(A_hat @ self.linear(H))
Session 3 - Neural Networks

Convolutional Neural Networks

Typical CNN
Typical CNN - Aphex34, CC BY-SA 4.0 , via Wikimedia Commons

  • Tasks: image classification, object detection, and segmentation

  • Data: grid-structured inputs such as images, video frames, spectrograms, and maps

  • Key idea: local connectivity, shared weights, and pooling

Session 3 - Neural Networks

Convolutional Neural Networks

Typical CNN
Typical CNN - Aphex34, CC BY-SA 4.0 , via Wikimedia Commons

Example: small colour photographs

  • Input: one RGB image of size 32 times 32

  • Local patterns: an edge or a texture can appear anywhere

  • Shared weights: one kernel is reused across the image

  • Task: classification into 10 classes, such as airplane, car, or bird

Session 3 - Neural Networks

Convolutional Neural Networks

Python implementation: PyTorch

RGB images of size and 10 classes

  • Training loop: the same as for the MLP

  • Loss: CrossEntropyLoss applies the softmax

from torch import nn

model = nn.Sequential(
    nn.Conv2d(3, 32, kernel_size=3, padding=1),   # 32 feature maps
    nn.ReLU(),
    nn.MaxPool2d(2),                              # 32x32 to 16x16
    nn.Conv2d(32, 64, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.MaxPool2d(2),                              # 16x16 to 8x8
    nn.Flatten(),
    nn.Linear(64 * 8 * 8, 10),                    # 10 classes
)
loss_fn = nn.CrossEntropyLoss()
Session 3 - Neural Networks

Conclusions

Machine Learning Pipeline
Data Science Process
Session 3 - Neural Networks

Conclusions

Linear regression

Regression Fit
Linear regression fit.

Linear classifiers

SVM Separating Hyperplanes
SVM Separating Hyperplanes - Cyc, Public domain, via Wikimedia Commons.

Hypothesis space and loss

Hypothesis Space for Regression
Example functions described using a linear model.

Session 3 - Neural Networks

Conclusions

Gradient descent

Gradient Descent Algorithm
Gradient Descent Algorithm - Jacopo Bertolotti, CC0, via Wikimedia Commons.

is the learning rate

Underfitting and overfitting

Overfitting
Overfitting - ThirdOrderLogic, CC BY 4.0 , via Wikimedia Commons

too simple or too complex

The perceptron

Perceptron Architecture
Perceptron Architecture.

A linear classifier

Session 3 - Neural Networks

Conclusions

Feedforward Network

Feedforward (MLP)

Tasks: regression and classification

Data: feature vectors

RNN

Recurrent (RNN, LSTM, GRU)

Tasks: forecasting, language, speech

Data: sequences

Autoencoder

Autoencoders and VAEs

Tasks: compression, denoising, generation

Data: unlabelled inputs

GAN

GANs and diffusion

Tasks: generating images and audio

Data: unlabelled samples

GNN

Graph neural networks

Tasks: node, link, and graph prediction

Data: graphs

CNN

CNNs and ResNets

Tasks: classification, detection, segmentation

Data: images and grids

Session 3 - Neural Networks

Many Thanks!

chc79@cam.ac.uk

_script: true

This script will only execute in HTML slides

_script: true