Feedforward Neural Network:

The overall network function combines these stages. For sigmoidal output unit activation functions, takes the form:

The bias parameters can be absorbed:

are continuous functions. The neural network is differentiable with respect to the parameters .

Sigmoid Function:

Range:

Derivative:

Hyperbolic Tangent:

Range:

Derivative:

ReLU (Rectified Linear Unit):

Range:

Derivative:

Training Process:

Given a training set of N example input-output pairs

Each pair was generated by an unknown function :

We want to find a hypothesis that minimises the error function:

We choose any starting point and then compute an estimate of the gradient and move a small amount in the steepest downhill direction, repeating until we converge on a point in the weight space with (local) minima loss.

Gradient Descent Algorithm:
Initialise w
repeat
    for each w[i] in w
        Compute gradient: g = ∇Loss(w[i])
        Update weight:   w[i] = w[i] - α * g
until convergence

The size of the step is given by the parameter α, which regulates the behaviour of the gradient descent algorithm. This is a hyperparameter of the regression model we are training, usually called learning rate.

Error Backpropagation Algorithm:

1. Forward Pass: Compute all activations and outputs for an input vector.

2. Error Evaluation: Evaluate the error for all the outputs using:

3. Backward Pass: Backpropagate errors for each hidden unit in the network using:

4. Derivatives Evaluation: Evaluate the derivatives for each parameter using:

Gradient Descent Update Rule:

Where is the learning rate.

There is not a straight single answer to determine the "right" hyperparameter values.

There is not a straight single answer to determine the "right" hyperparameter values. But, experts follow similar steps:

1. Become one with the data
2. Set up the end-to-end training/evaluation skeleton
3. Start with a simple model. For most of the tasks, a fully-connected neural network with one hidden layer.
4. Implement a complex model that overfits and regularise
5. Tune the hyperparameters
6. Continue training

There is not a straight single answer to determine the "right" hyperparameter values. But, experts follow similar steps:

1. Become one with the data
2. Set up the end-to-end training/evaluation skeleton
3. Start with a simple model. For most of the tasks, a fully-connected neural network with one hidden layer.
4. Implement a complex model that overfits and regularise
5. Tune the hyperparameters
6. Continue training

See more in Karpathy's recipe, and Tobin's lecture.

In the process, you should combine the methods previously discussed like preprocessing, feature engineering, normalising inputs, parameters initialisation, etc.

In the process, you should combine the methods previously discussed like preprocessing, feature engineering, normalising inputs, parameters initialisation, etc.

The agent aims to maximise the rewards from its actions. Rewards can be immediate or sparse.

The agent aims to maximise the rewards from its actions. Rewards can be immediate or sparse.

Providing a reward signal is easier than providing labelled examples (i.e., supervised learning).

The agent aims to maximise the rewards from its actions. Rewards can be immediate or sparse.

Providing a reward signal is easier than providing labelled examples (i.e., supervised learning).

The RL framework is composed of:

• Agent: The learner and decision maker
• Environment: The world in which the agent operates
• State: Current situation of the environment
• Action: What the agent can do
• Reward: Feedback from the environment

The RL framework is composed of:

• Agent: The learner and decision maker
• Environment: The world in which the agent operates
• State: Current situation of the environment
• Action: What the agent can do
• Reward: Feedback from the environment

The environment is stochastic, meaning that the outcomes of actions taken by the agent in each state are not deterministic.

A MDP is a 4-tuple:

A MDP is a 4-tuple:

Where:


is a set of states with initial state
is a set of actions in each state
is a transition model that tells the probability of reaching , if the agent is in and performs action
is the reward function that tells the reward for every transition from to through

The transition model describes the outcome of each action in each state. Since the outcome is stochastic, we write:

The transition model describes the outcome of each action in each state. Since the outcome is stochastic, we write:

Transitions are Markovian: The probability of reaching from depends only on and not on the history or earlier states.

Uncertainty once again brings MDPs closer to reality when compared against deterministic approaches.

From every transition the agent receives a reward:

From every transition the agent receives a reward:

The agent wants to maximise the sum of the received rewards (i.e., utility function):

The utility function depends on a sequence of states and actions named the environment history.

The solution for this problem is called a policy, which specifies what the agent should do for any state that the agent might reach. A policy is a mapping from states to actions that tells the agent what to do in each state:

Deterministic Policy:

The solution for this problem is called a policy, which specifies what the agent should do for any state that the agent might reach. A policy is a mapping from states to actions that tells the agent what to do in each state:

Deterministic Policy:

Stochastic Policy:

where is the probability of taking action in state .

The quality of the policy in a given state is measured by the expected utility of the possible environment histories generated by that policy. We can compute the utility of state sequences using additive (discounted) rewards as follows:

where is the discount factor that determines the importance of future rewards.

The quality of the policy in a given state is measured by the expected utility of the possible environment histories generated by that policy. We can compute the utility of state sequences using additive (discounted) rewards as follows:

where is the discount factor that determines the importance of future rewards.

The expected utility of executing the policy , starting in state , is given by:

is with respect to the probability distribution over state sequences determined by and .

We can compare policies at a given state using their expected utilities:

is with respect to the probability distribution over state sequences determined by and .

We can compare policies at a given state using their expected utilities:

is with respect to the probability distribution over state sequences determined by and .

The goal is to select the policy that maximises the expected reward:

The policy recommends an action for every state in the sequence starting in state .

The utility function allows the agent to select actions by using the principle of maximum expected utility. The agent chooses the action that maximises the reward for the next step plus the expected discounted utility of the subsequent step:

The utility function allows the agent to select actions by using the principle of maximum expected utility. The agent chooses the action that maximises the reward for the next step plus the expected discounted utility of the subsequent step:

The utility of a state is the expected reward for the next transition plus the discounted utility of the next state, assuming that the agent chooses the optimal action. The utility of a state
is given by:

This is called the Bellman Equation, after Richard Bellman (1957).

Another important quantity is the action-utility function or Q-function, which is the expected utility of taking a given action in a given state:

Another important quantity is the action-utility function or Q-function, which is the expected utility of taking a given action in a given state:

relates to the utility function as follows:

Another important quantity is the action-utility function or Q-function, which is the expected utility of taking a given action in a given state:

relates to the utility function as follows:

Then we have:

Another important quantity is the action-utility function or Q-function, which is the expected utility of taking a given action in a given state:

The Q-function tells us how good it is to take action in state .

Another important quantity is the action-utility function or Q-function, which is the expected utility of taking a given action in a given state:

The Q-function tells us how good it is to take action in state .

The optimal policy can be extracted from as follows:

• Knows transition model and reward function
• Can simulate outcomes before taking actions
• Value Iteration, Policy Iteration

• Unknown transition model and reward function
• Cannot simulate outcomes
• Q-Learning, DQN

Policy Iteration is a model-based reinforcement learning algorithm that alternates between policy evaluation and policy improvement to find the optimal policy.

Policy Iteration is a model-based reinforcement learning algorithm that alternates between policy evaluation and policy improvement to find the optimal policy.

Finds the optimal policy through iterative refinement:

• Requires a model of the environment (transition and reward functions)
• Guaranteed to converge to the optimal policy
• Works with finite state and action spaces

Policy Iteration consists of two main phases that alternate until convergence:

Policy Iteration consists of two main phases that alternate until convergence:

1. Policy Evaluation: Compute the value function for the current policy:

Policy Iteration consists of two main phases that alternate until convergence:

1. Policy Evaluation: Compute the value function for the current policy:

2. Policy Improvement: Update the policy to be greedy with respect to the current value function:

Policy Iteration Algorithm:

The algorithm iteratively improves the policy by alternating between evaluation and improvement steps until convergence to the optimal policy.

Policy Iteration Algorithm:

The algorithm iteratively improves the policy by alternating between evaluation and improvement steps until convergence to the optimal policy.

The optimal policy maximizes the expected cumulative reward by selecting actions that lead to the highest Q-values.

Policy iteration guarantees convergence to the optimal policy through the principle of policy improvement.

Policy Iteration Limitations:

Policy Iteration Limitations:

• Model Dependency: Requires complete knowledge of the environment's transition and reward functions.
• Computational Cost: Policy evaluation can be expensive for large state spaces, requiring iterative computation.
• Discrete Spaces: Designed for finite state and action spaces, not suitable for continuous environments.
• Memory Requirements: Needs to store value functions and policies for all states.

At each step, the agent updates its estimate of the Q-function for a state and action using the observed reward and the maximum estimated value of the next state. The update rule is defined as follows:

where:
is the learning rate
is the discount factor
is the observed reward
is the next state after taking action in

Q-Learning Algorithm:
Initialize Q(s, a) arbitrarily for all s, a
Repeat (for each episode):
    Initialize state s
    Repeat (for each step of episode):
        a ← π(s)
        // Act according to a
        r ← = R(s, a, s')
        Q(s, a) ← update(Q, s, a, s', r)
        s ← s'
    until s is terminal

Q-Learning Algorithm:
Initialize Q(s, a) arbitrarily for all s, a
Repeat (for each episode):
    Initialize state s
    Repeat (for each step of episode):
        a ← π(s)
        // Act according to a
        r ← = R(s, a, s')
        Q(s, a) ← update(Q, s, a, s', r)
        s ← s'
    until s is terminal

The agent must balance exploring new actions to discover their value and exploiting known actions to maximize reward.

Q-Learning Algorithm:
Initialize Q(s, a) arbitrarily for all s, a
Repeat (for each episode):
    Initialize state s
    Repeat (for each step of episode):
        a ← π(s)
        // Act according to a
        r ← = R(s, a, s')
        Q(s, a) ← update(Q, s, a, s', r)
        s ← s'
    until s is terminal

The agent must balance exploring new actions to discover their value and exploiting known actions to maximize reward.

ε-Greedy Policy:

Q-Learning Algorithm:
Initialize Q(s, a) arbitrarily for all s, a
Repeat (for each episode):
    Initialize state s
    Repeat (for each step of episode):
        a ← π(s)
        // Act according to a
        r ← = R(s, a, s')
        Q(s, a) ← update(Q, s, a, s', r)
        s ← s'
    until s is terminal

The agent must balance exploring new actions to discover their value and exploiting known actions to maximize reward.

ε-Greedy Policy:

Q-Learning Limitations:

Q-Learning Limitations:

• Scalability: Q-learning struggles with large state spaces as it requires a Q-value for every state-action pair, leading to high memory usage.
• Generalisation: Q-learning does not generalise well to unseen states since it relies on a discrete Q-table.
• Continuous State Spaces: Q-learning is not suitable for environments with continuous state spaces as it requires discretisation, which can lead to loss of information.
• Sample Inefficiency: Q-learning can be sample-inefficient, requiring many interactions with the environment to learn an optimal policy.

Deep Q-Networks (DQN) extend Q-learning to environments with large or continuous state spaces. It replaces the Q-table using neural networks to approximate the Q-function:

Deep Q-Networks (DQN) extend Q-learning to environments with large or continuous state spaces. It replaces the Q-table using neural networks to approximate the Q-function:

Deep Q-Networks (DQN) extend Q-learning to environments with large or continuous state spaces. It replaces the Q-table using neural networks to approximate the Q-function:

A main network approximates and are its trainable parameters. The input of the network is the state and the output is a Q-value per possible action.

Deep Q-Networks (DQN) extend Q-learning to environments with large or continuous state spaces. It replaces the Q-table using neural networks to approximate the Q-function:

A main network approximates and are its trainable parameters. The input of the network is the state and the output is a Q-value per possible action.

DQNs store past experiences in a replay buffer. During training a minibatch of experiences are selected. A target network is used to compute the target Q-values during updates.

Value-function

Model-based with guaranteed convergence for finite and discrete problems.

Q-function

Model-free and simple for small and discrete problems.

Neural Network

Model-free and complex for large and continuous problems.

AlphaGo Zero combines key RL concepts:

• Self-play environment: Agent plays against itself (no human data needed)
• Policy network: Learns π(s) → probability distribution over actions
• Value network: Learns V(s) → probability of winning from state s
• Monte Carlo Tree Search: Uses policy/value to guide search
• Temporal difference learning: Updates based on game outcomes
• Experience replay: Stores and learns from self-play games