Instead of assuming deterministic relationships, we model the regression problem with the likelihood function:

The functional relationship between and is given as:

Where is Gaussian noise with zero mean and variance .

We want to find that approximates the unknown function and generalises well.

For linear regression, we have:

Where is Gaussian noise with zero mean and variance . We seek the parameters .

So, given a training set of i.i.d input-output pairs:

The likelihood factorises according to:

The likelihood function tells us how likely the observed data is given the specific parameters:

We estimate by maximising the likelihood.

Polynomial basis:

Gaussian basis:

Sigmoid basis:

If we apply a probabilistic interpretation, we need to maximise the likelihood of:

After a similar process (See Chapter 3 in Bishop, 2006), the loss function becomes:

And the normal equation solution:

The design matrix is constructed as:

By using basis functions , we can capture complex, non-linear relationships between the input features and the target variable. The design matrix essentially acts as a bridge, allowing us to apply linear techniques to problems that are inherently non-linear in nature. We can design the matrix and evaluate which design works better using cross-validation. This is essentially feature engineering.

The perceptron was simulated by Frank Rosenblatt in 1957 on an IBM 704 machine as a model for biological neural networks. The neural network was invented in 1943 by McCulloch & Pitts.

By 1962 Rosenblatt published the "Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms".

The Perceptron machine is named Mark I. It is a special-purpose hardware that implemented the perceptron supervised learning for image recognition.

The Perceptron machine is named Mark I. It is a special-purpose hardware that implemented the perceptron supervised learning for image recognition.

• Input layer: An array of 400 photocells (20x20 grid) named "sensory units" or "input retina".
• Hidden Layer: 512 perceptrons named "association units" or "A-units".
• Output Layer: 8 perceptrons named "response units" or "R-units".

The perceptron is a linear classifier model (i.e., linear discriminant), with hypothesis space defined by all the functions of the form:

The function is similar to the function previously defined. is given by a step function of the form:

We want to find :

We need to derive the perceptron criterion. We know we are seeking parameters vector such that features in in class :

And for features in in class :

Using , all features will satisfy:

The loss function is:

The update rule for a missclassified input is:

Feedforward Neural Network:

First, recall the general form of a linear model with nonlinear basis functions:

where is a nonlinear activation function (e.g., identity for regression, sigmoid for classification), are basis functions, and are weights.

Feedforward Neural Network:

First, recall the general form of a linear model with nonlinear basis functions:

where is a nonlinear activation function (e.g., identity for regression, sigmoid for classification), are basis functions, and are weights.

In neural networks, the basis functions themselves are parameterized and learned. The network is constructed as a sequence of transformations.

Feedforward Neural Network:

1. Linear combination of inputs (first layer):

where and are the weights from input to hidden unit , is the bias for hidden unit .

Feedforward Neural Network:

1. Linear combination of inputs (first layer):

where and are the weights from input to hidden unit , is the bias for hidden unit .

2. Nonlinear activation (hidden layer):

where is a nonlinear activation function (e.g., sigmoid, tanh, ReLU).

Feedforward Neural Network:

1. Linear combination of inputs (first layer):

where and are the weights from input to hidden unit , is the bias for hidden unit .

2. Nonlinear activation (hidden layer):

where is a nonlinear activation function (e.g., sigmoid, tanh, ReLU).

Feedforward Neural Network:

3. Linear combination of hidden activations (output layer):

where and are the weights from hidden unit to output unit , is the bias for output unit .

Feedforward Neural Network:

3. Linear combination of hidden activations (output layer):

where and are the weights from hidden unit to output unit , is the bias for output unit .

Optionally, the output activations can be further transformed using an appropriate activation function to produce the final network outputs . For example, for regression problems or for binary classification problems.

Feedforward Neural Network:

The overall network function combines these stages. For sigmoidal output unit activation functions, takes the form:

Feedforward Neural Network:

The overall network function combines these stages. For sigmoidal output unit activation functions, takes the form:

The bias parameters can be absorbed:

Feedforward Neural Network:

The overall network function combines these stages. For sigmoidal output unit activation functions, takes the form:

The bias parameters can be absorbed:

are continuous functions. The neural network is differentiable with respect to the parameters .

Sigmoid Function:

Range:

Derivative:

Sigmoid Function:

Range:

Derivative:

Hyperbolic Tangent:

Range:

Derivative:

Sigmoid Function:

Range:

Derivative:

Hyperbolic Tangent:

Range:

Derivative:

ReLU (Rectified Linear Unit):

Range:

Derivative:

Training Process:

Given a training set of N example input-output pairs

Training Process:

Given a training set of N example input-output pairs

Each pair was generated by an unknown function :

Training Process:

Given a training set of N example input-output pairs

Each pair was generated by an unknown function :

We want to find a hypothesis that minimises the error function:

Training Process (regression problem):

Giving a probabilistic interpretation to the network outputs:

where is the precision (i.e. inverse variance ).

Training Process (regression problem):

Giving a probabilistic interpretation to the network outputs:

where is the precision (i.e. inverse variance ).

For an i.i.d. training set, the likelihood function corresponds to:

Training Process (regression problem):

Training Process (regression problem):

As we saw in previous sessions, maximising the likelihood function is equivalent to minimising the sum-of-squares error function given by:

Training Process (binary classification):

The network output is:

We use a single target variable such that denotes class 1 and denotes class 2.

Training Process (binary classification):

The network output is:

We use a single target variable such that denotes class 1 and denotes class 2. We interpret as the conditional probability distribution of targets given inputs:

Training Process (binary classification):

The network output is:

We use a single target variable such that denotes class 1 and denotes class 2. We interpret as the conditional probability distribution of targets given inputs:

For an i.i.d. training, the error function is the cross-entropy error:

We choose any starting point and then compute an estimate of the gradient. We then move a small amount in the steepest downhill direction, repeating until we converge on a point in the weight space with (local) minima loss.

Gradient Descent Algorithm:
Initialize w
repeat
    for each w[i] in w
        Compute gradient: g = ∇Loss(w[i])
        Update weight:   w[i] = w[i] - α * g
until convergence

The size of the step is given by the parameter α, which regulates the behaviour of the gradient descent algorithm. This is a hyperparameter of the regression model we are training, usually called learning rate.

Error Backpropagation Algorithm:

Error Backpropagation Algorithm:

1. Forward Pass: Compute all activations and outputs for an input vector.

Error Backpropagation Algorithm:

1. Forward Pass: Compute all activations and outputs for an input vector.

2. Error Evaluation: Evaluate the error for all the outputs using:

Error Backpropagation Algorithm:

1. Forward Pass: Compute all activations and outputs for an input vector.

2. Error Evaluation: Evaluate the error for all the outputs using:

3. Backward Pass: Backpropagate errors for each hidden unit in the network using:

Error Backpropagation Algorithm:

1. Forward Pass: Compute all activations and outputs for an input vector.

2. Error Evaluation: Evaluate the error for all the outputs using:

3. Backward Pass: Backpropagate errors for each hidden unit in the network using:

4. Derivatives Evaluation: Evaluate the derivatives for each parameter using:

Gradient Descent Update Rule:

Gradient Descent Update Rule:

Gradient Descent Update Rule:

Gradient Descent Update Rule:

Where is the learning rate.

Key Characteristics:

• Multiple Hidden Layers: 3+ layers for deep architectures
• Hierarchical Features: Each layer learns increasingly abstract representations
• Automatic Feature Learning: No manual feature engineering required
• Representation Learning: Learns useful representations from raw data

Vanishing and Exploding Gradients: In deep networks, gradients can become very small (vanishing) or very large (exploding) during backpropagation:

Vanishing and Exploding Gradients: In deep networks, gradients can become very small (vanishing) or very large (exploding) during backpropagation:

• Proper Weight Initialization: Xavier/Glorot initialization
• Batch Normalization: Normalize activations during training
• Modern Optimizers: Adam, RMSprop with adaptive learning rates
• Regularisation: Dropout, L2, early stopping, and augmentation techniques
• Network Architectures: Different architectures for different problems

Weight Initialization:

Xavier/Glorot Initialization:

He Initialization (for ReLU):

Where and are the number of input and output neurons respectively.

Input normalisation:

Where (i.e., scale) and (i.e., shift) are learnable parameters, and is a small constant for numerical stability.

Modern Optimisers:

Optimisers train models by efficiently navigating the loss landscape to find optimal parameters. They help in accelerating convergence, avoiding local minima, and improving generalisation.

• Stochastic Gradient Descent (SGD): Updates parameters using a subset of data and reduces computation time.
• Adam: Adaptively adjust the learning rate for each parameter.
• ...

Regularization Techniques:

Dropout: Randomly deactivate neurons during training