Batch ingestion

• Bounded data, processed on a schedule
• ETL steps wired into an orchestrated flow
• Governance travels with it: audit log, schema contract

Stream ingestion

• Unbounded events, processed as they arrive
• Publish / subscribe decouples producers and consumers
• A partitioned log gives parallelism and replay

Analytics is its own stage of the pipeline

• Visualise: charts and maps that humans can read
• Dashboard: several views for one decision audience
• Model: predict or summarise with limits and governance

From processing to decision

• Assess: trust the curated layer that defines the schema, coverage, bias
• Address: answer the decision question with labelled evidence

Choose the chart for the question

• Trend over time? a line chart
• Compare categories? a bar chart
• Relationship between two variables? a scatter plot
• Distribution? a histogram or box plot

Honest charts

• Start axes at a sensible baseline; do not exaggerate
• Label units, the source, and the date
• Show uncertainty when it matters

A misleading chart is a governance failure, not a style choice.

Evidence layers

• Official layer: the result we stand behind
• Context layer: monitoring and exploration, clearly labelled not official

Governance controls

• Different audiences see different panels
• Suppress small groups to protect people
• State the source and refresh date on every panel

Two layers, one figure

• Each panel names its layer
• The context panel is marked not official
• The title carries the source

from plotly.subplots import make_subplots
import plotly.graph_objects as go

dash = make_subplots(rows=2, cols=1, subplot_titles=[
    "Official layer: observed vs predicted",
    "Context layer: monitoring (not official)",
])
dash.add_trace(go.Scatter(x=y_val, y=pred, mode="markers"),
               row=1, col=1)
dash.add_trace(go.Bar(x=["events stored"], y=[n_context]),
               row=2, col=1)
dash.update_layout(
    title_text="Evidence dashboard - official + context",
)

A Machine Learning algorithm is a set of instructions that are used to train a machine learning model. It defines how the model learns from data and makes predictions or decisions. Linear regression, decision trees, and neural networks are examples of machine learning algorithms.

A Machine Learning model is a program that is trained on a dataset and used to make predictions or decisions. The goal is to create a trained model that can generalise well to new, unseen data. For example, a trained model could predict house prices based on new input features.

A Machine Learning algorithm uses the training process that goes from a specific set of observations to a general rule (i.e., induction). This process adjusts the Machine Learning model internal parameters to minimise prediction errors. For example, in a linear regression model, the algorithm adjusts the slope and intercept to minimise the difference between predicted and actual values.

The parameters of Machine Learning models are adjusted according to the training data (i.e., seen data). For example, when training a model to predict house prices, the training data would include features like square footage, number of bedrooms, and location, along with their actual sale prices. The training data consists of vectors of attribute values.

In classification problems, the prediction (i.e., model's output) is one of a finite set of values (e.g., sunny/cloudy/rainy or true/false). In the regression problems, the model's output is a number.

Predictions can deviate from the expected values. Prediction errors are quantified by a loss function that indicates to the algorithm how far the prediction is from the target value. This difference is used to update the machine learning model's internal parameters to minimise the prediction errors. Common examples include Mean Squared Error for regression problems and Cross-Entropy for classification problems.

Three types of feedback can be part of the inputs in our training process:

• Supervised Learning: The model is trained on labeled data to learn a mapping between inputs and the corresponding labels, so the model can make predictions on unseen data.
• Unsupervised Learning: The model is trained on unlabeled data, and it must find patterns in the data. The goal is to identify hidden structures or groupings in the data.
• Reinforcement Learning: The model learns by interacting with an environment and receiving rewards or penalties. The goal is to learn a policy that maximizes the reward.

Linear Regression

Training Dataset

Hypothesis Space: All possible linear functions of continuous-valued inputs and outputs.

Hypothesis:

Loss Function:

Cost Function:

Linear Regression

Analytical Solution:

Gradient Descent Algorithm:

Initialize w randomly
repeat
    for each w[i] in w
        Compute gradient: g = ∇Loss(w[i])
        Update weight:   w[i] = w[i] - α * g
until convergence

Hyperparameters: Learning rate, number of epochs, and batch size.

Perceptron update rule:

The rule is applied one example at a time, choosing examples at random (as in stochastic gradient descent).

Why go beyond linear?

• Linear / logistic: fast, interpretable, limited shape
• Small MLP: more flexible, harder to explain
• Training is costly (many weights); inference is cheap once trained

Architecture (feedforward)

• Input layer: features (e.g. department, month, year)
• Hidden layer(s): weighted sum + nonlinear activation
• Output layer: prediction (e.g. employment rate)

Information flows forward: inputs → hidden units → output.

Training a neural network

• Feedforward: pass inputs forward layer by layer, applying weights and activation functions, to compute the final prediction
• Loss Function: same idea as before compare the prediction with the actual value using the loss function
• Backpropagation applies the chain rule to compute gradients through the layers efficiently

Hyperparameters: hidden size, learning rate, max iterations.

Precision vs explainability

• The neural network may be more accurate
• The linear model is easier to explain to a decision-maker
• For a public decision, explainability often wins

No model removes uncertainty — there is no Laplace demon. We report what the model supports and what it cannot.

Train, validate, test

• Train: fit the model
• Validation: choose between models
• Test: judge once, at the end

from sklearn.model_selection import train_test_split

# hold out the test set first
X_rest, X_test, y_rest, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42)

# split the rest into train and validation
X_train, X_val, y_train, y_val = train_test_split(
    X_rest, y_rest, test_size=0.25, random_state=42)

Start simple: a linear model

• Transparent and fast
• Report R² and MAE on validation

from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score, mean_absolute_error

linear = LinearRegression().fit(X_train, y_train)
pred = linear.predict(X_val)

print("R2 :", r2_score(y_val, pred))
print("MAE:", mean_absolute_error(y_val, pred))

More flexible: a small neural network

• Scale the inputs first (learn scale on train only)
• An MLP can fit non-linear shapes

from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPRegressor

scaler = StandardScaler().fit(X_train)
mlp = MLPRegressor(hidden_layer_sizes=(8,),
                   max_iter=500, random_state=42)
mlp.fit(scaler.transform(X_train), y_train)
pred = mlp.predict(scaler.transform(X_val))

Choose, then judge once

• Pick the model with lower validation error
• Report test error only at the end

# choose on validation, judge once on test
if mae_linear <= mae_mlp:
    model, name = linear, "linear"
else:
    model, name = mlp, "mlp"

print("chosen:", name)
print("test R2:", r2_score(y_test, model.predict(X_test_ready)))

• Analytics turns ingested data into evidence
• Dashboards are governance: layers, audiences, suppression
• Data address can include prediction models
• Supervised learning

Notebooks and templates

• Lecture 5 - batch and stream ingestion with Prefect and Kafka
• Lecture 6 - analytics and visualisation on curated aggregates
• Week 3 group notebook
• Week 3 reflection template

Deliverables

• Group presentation in our next session: ~20 min (slides + live demo) and ~10 min Q&A.
• Zip file with code that implements the pipeline, PDF project report, and governance files.
• Individual reflection on final project

Templates

• Project report template (Word)
• Presentation template (6 slides)
• Final project reflection template (Word)