Christian Cabrera Jojoa
Assistant Research Professor
Department of Computer Science and Technology
University of Cambridge
chc79@cam.ac.uk
Given a training set of example input-output pairs
Each pair was generated by an unknown function
The goal is to discover a function that approximates the true function .
The regression problem involves predicting a continuous numerical value. Regression models approximate a function that maps input features to a continuous output.
The hypotheses space includes linear functions of continuous-valued inputs and outputs.
The simplest example is "fitting a straight line". The model learns the coefficients
Finding the that best fits the data is called linear regression.
Finding the values of the weights and that minimise the empirical loss (Squared Error).
Given the
We want to find
We know that the loss is minimised when its partial derivatives with respect to and are zero.
Solving the partial derivatives
We choose any starting point and then compute an estimate of the gradient and move a small amount in the steepest downhill direction, repeating until we converge on a point in the weight space with (local) minima loss.
Gradient Descent Algorithm: Initialize w randomly repeat for each w[i] in w Compute gradient: g = ∇Loss(w[i]) Update weight: w[i] = w[i] - α * g until convergence
The size of the step is given by the parameter , which regulates the behaviour of the gradient descent algorithm. This is a hyperparameter of the regression model we are training, usually called learning rate.
Following with our "straight line" example, the update rule for and is
The loss function is represented as a composition of functions.
We need to differentiate the loss function step by step using the chain rule to compute the gradients.
Applying the chain rule to the loss function
Applying this to and
The update rule for each weight in the "straight line" example is (folding 2 into )
For training examples
These updates constitute the batch gradient descent. For the "straight line", this gradient descent is deterministic.
An epoch is defined as one complete pass through the entire dataset during the training process. Multiple epochs are needed for the model to converge to an optimal solution. Not enough epochs could generate underfitting. Too many epochs could generate overfitting. Another hyperparameter for the model.
Batch Gradient Descent
The algorithm updates using the entire dataset in each iteration.
import numpy as np def batch_gradient_descent(X, y, alpha=0.01, epochs=1000): """ X: numpy array of shape (n_samples, n_features) y: numpy array of shape (n_samples,) alpha: learning rate epochs: number of passes over the data Returns: w (numpy array of shape (n_features,)) """ n_samples, n_features = X.shape w = np.zeros(n_features) for epoch in range(epochs): grad = [0] * len(w) for i in range(n_samples): x_i = X[i] y_i = y[i] prediction = dot(w, x_i) error = prediction - y_i for j in range(len(w)): grad[j] += error * x_i[j] for j in range(len(w)): grad[j] /= len(n_samples) w[j] -= alpha * grad[j] return w
Stochastic Gradient Descent
The algorithm updates after computing the gradient for each training example.
import numpy as np def stochastic_gradient_descent(X, y, alpha=0.01, epochs=1000): """ X: numpy array of shape (n_samples, n_features) y: numpy array of shape (n_samples,) alpha: learning rate epochs: number of passes over the data Returns: w (numpy array of shape (n_features,)) """ n_samples, n_features = X.shape w = np.zeros(n_features) for epoch in range(epochs): for i in range(n_samples): x_i = X[i] y_i = y[i] prediction = np.dot(w, x_i) error = prediction - y_i for j in range(n_features): w[j] -= alpha * error * x_i[j] return w
Mini-batch Gradient Descent
The algorithm updates after computing the gradient for a small batch of training examples (batch size ). The batch size is another hyperparameter.
import numpy as np def mini_batch_gradient_descent(X, y, alpha=0.01, epochs=1000, batch_size=32): """ X: numpy array of shape (n_samples, n_features) y: numpy array of shape (n_samples,) alpha: learning rate epochs: number of passes over the data batch_size: number of samples per batch Returns: w (numpy array of shape (n_features,)) """ n_samples, n_features = X.shape w = np.zeros(n_features) for epoch in range(epochs): indices = np.arange(n_samples) np.random.shuffle(indices) X_shuffled = X[indices] y_shuffled = y[indices] for start in range(0, n_samples, batch_size): end = start + batch_size X_batch = X_shuffled[start:end] y_batch = y_shuffled[start:end] y_pred = X_batch @ w error = y_pred - y_batch grad = X_batch.T @ error / len(X_batch) w -= alpha * grad return w
Linear Regression
Training Dataset
Hypothesis Space: All possible linear functions of continuous-valued inputs and outputs.
Hypothesis:
Loss Function:
Cost Function:
Analytical Solution:
Gradient Descent Algorithm:
Initialize w randomly repeat for each w[i] in w Compute gradient: g = ∇Loss(w[i]) Update weight: w[i] = w[i] - α * g until convergence
Hyperparameters: Learning rate, number of epochs, and batch size.
Multivariate regression extends the simple linear model to handle multiple input features. The model learns a function that maps multiple input variables to a continuous output.
In these problems, each example is a n-element vector. The hypotheses space now includes linear functions of multiple continuous-valued inputs and a single continuous output.
We want to find the that best fits the data.
In vector notation:
Where is the feature vector, is the parameters vector, and is a bias term.
The loss function for multivariate regression is:
In matrix notation:
Where is the target vector, is the feature matrix, and is the weight vector.
Given the Loss function:
Taking the derivative with respect to and setting to zero:
This leads to the normal equation:
Gradient Descent for Multivariate Regression
The gradient of the loss function is:
Update rule for :
Where is the learning rate.
Underfitting occurs when our hypothesis space is too simple to capture the true function
Even the best hypothesis in will have high error because:
where is some small positive number.
Overfitting occurs when our hypothesis space is too complex, leading to:
Low empirical loss but high generalisation loss:
The hypothesis memorizes the training examples instead of learning the true function .
Regularisation transforms the loss function into a cost function that penalizes complexity to avoid overfitting
Choosing the complexity function depends on the hypotheses space . A good example for polynomials will be a function that returns the sum of the squares of coefficients.
Given the new cost function
The best hypothesis is the one that minimises the cost:
where is a hyperparameter that controls the trade-off between fitting the data and model complexity and serves as a conversion rate.
Alternatives to mitigate overfitting are feature selection, hyperparameter tuning, splitting dataset in training, validation, and test data (e.g., cross validation algorithm).
Linear classifiers are models that separate data into classes using a linear decision boundary. These classifiers are functions that can decide if an input (i.e., vectors of numbers) belong to a specific class or not.
A decision boundary is a line (or a surface in higher dimensions) that separates data into classes.
The hypothesis is the result of passing a linear function through a threshold function:
The optimal hypothesis is the one that minimises the loss function:
Where is the hypothesis space.
Partial derivatives and gradient methods do not work. Instead, we apply perceptron learning rule.
Perceptron update rule:
The rule is applied one example at a time, choosing examples at random (as in stochastic gradient descent).
An alternative to the discrete threshold function is the logistic function, which is smooth and differentiable.
An alternative to the discrete threshold function is the logistic function, which is smooth and differentiable. The process of finding the optimal hypothesis is called logistic regression.
Following the gradient descent algorithm, the update rule is:
The loss function is represented as a composition of functions:
We need to differentiate the loss function using the chain rule to compute the gradients:
The resulting update rule after solving the partial derivatives:
The gradient descent algorithm is applied using this update rule.
Gradient Descent for Logistic Regression
The perceptron is one of the earliest and simplest artificial neural network models for binary classification.
The perceptron was simulated by Frank Rosenblatt in 1957 on an IBM 704 machine as a model for biological neural networks. The neural network was invented in 1943 by McCulloch & Pitts.
By 1962 Rosenblatt published the "Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms".
The Perceptron machine is named Mark I. It is a special-purpose hardware that implemented the perceptron supervised learning for image recognition.
The perceptron is a linear classifier model (i.e., linear discriminant), with hypothesis space defined by all the functions of the form:
The function is similar to the function previously defined. is given by a step function of the form:
We want to find :
We need to derive the perceptron criterion. We know we are seeking parameters vector such that features in in class :
And for features in in class :
Using , all features will satisfy:
The loss function is:
The update rule for a missclassified input is:
The perceptron learning algorithm is similar to the stochastic gradient descent.
Initialise weights w randomly repeat for each training example (x, y) Compute prediction: y_pred = f(w·Φ(x)) if y_pred ≠ y then Update weights: w = w + αΦ(x)y until no misclassifications or max iterations
import numpy as np class Perceptron: def __init__(self, learning_rate=0.01, max_iterations=1000): self.learning_rate = learning_rate self.max_iterations = max_iterations self.weights = None self.bias = None def fit(self, X, y): n_samples, n_features = X.shape self.weights = np.zeros(n_features) self.bias = 0 for _ in range(self.max_iterations): misclassified = 0 for i in range(n_samples): prediction = self.predict(X[i]) if prediction != y[i]: self.weights += self.learning_rate * (y[i] - prediction) * X[i] self.bias += self.learning_rate * (y[i] - prediction) misclassified += 1 if misclassified == 0: break def predict(self, x): return 1 if np.dot(self.weights, x) + self.bias >= 0 else 0
Python Implementation
XOR Problem:
The perceptron cannot learn the XOR function because it's not linearly separable.
(Minsky & Papert, 1969)
Other limitations:
Hyperparameters defined in this lecture
Learning rate : size of each gradient descent step
Epochs: number of complete passes through the dataset
Batch size : training examples per mini-batch update
Regularisation strength : weight of the complexity penalty in the cost function
Max iterations: stopping limit for the perceptron learning algorithm
_script: true
This script will only execute in HTML slides