• Scalability: Elastic compute and storage at scale
• Availability: Fault tolerance and replication
• Latency: Network bandwidth and distributed execution
• Security: identity, and access control
• Interoperability: Service interfaces (APIs) for clients and pipelines
• Ethics, privacy, and fairness

• Distributed file systems and batch engines (Hadoop era)
• In-memory and stream processing (Spark, Flink, Kafka)
• Cloud object storage, warehouses, and lakehouse platforms
• Orchestration, catalogues, and observability tools

Most investment targets scale and speed...

Data governance is the formulation of policy to optimize, secure, and leverage information as an enterprise asset by aligning the objectives of multiple functions. (Khatri & Brown, 2010)

Existing efforts: GDPR-style regulation, public-sector frameworks such as the UK Data and AI Ethics Framework (transparency, accountability, fairness, privacy, trade-offs), catalogues, lineage, IAM. Still less mature than the technical stack, often reactive after harm.

Governance after consequences

• Public hearings follow harm, leaks, and election controversies
• Policy and platform changes arrive late in the pipeline
• Reactive fixes differ from designing requirements in from the start
• Frameworks such as the UK Data and AI Ethics Framework aim to bridge principles and practice earlier

"Surveillance capitalism unilaterally claims human experience as free raw material for translation into behavioral data." (Zuboff, 2019)

Problem first, not ethics last

• Ethics is often taught as a final checklist before deployment
• In this course we place ethics, privacy, and fairness at the start of the pipeline
• Next: define what these terms mean and where they apply

Universal Declaration of Human Rights (UDHR)

"We put forth the doctrine of universal human rights as a globally salient set of values for explicit value alignment in responsible AI." (Prabhakaran et al., 2022)

First-order challenges — two types

Afroogh et al., 2024 split value challenges facing AI systems:

• Ethical–legal: fairness, privacy, accountability — moral and legal rights and duties
• Multi-dimensional: trust, transparency, explainability — social acceptance, intelligibility, and confidence in the system

Both types are first-order: they shape how data projects affect people directly.

Map of the debate (Mittelstadt et al., 2016)

• Discrimination and unfair treatment of groups
• Autonomy and loss of meaningful choice
• Non-transparency and opaque decisions
• Privacy and unwanted exposure
• Plus procedural, psychological, and economic harms

Trade-offs as decisions are not easy

First-order values can pull in opposite directions. For example when responding to a disaster: tracing phones may support fair rescue reach but challenge privacy. (Afroogh et al., 2024)

Public-sector guidance asks teams to document tensions, consult human-rights frameworks, and revisit choices as the project evolves. (UK Data and AI Ethics Framework)

Privacy (ethical–legal)

Control over appropriate flows of personal information in context. Not only secrecy, but who receives what, when, and for what purpose.

(Nissenbaum, 2004)

UDHR Art. 12 (Prabhakaran et al., 2022)

Fairness (ethical–legal)

Just and equitable treatment across groups. In algorithms, asks whether similar people are treated similarly and whether historical bias is reproduced.

Linked to non-discrimination (UDHR) and justice in (Mittelstadt et al., 2016)

Accountability (ethical–legal)

Clear answer to "who is responsible for the way the system works and for harm it causes?" Includes documentation, audit trails, and remedial action.

(Prabhakaran et al., 2022)

Trust (multi-dimensional)

Confidence that a data system is competent, honest, and used for legitimate purposes. Distrust grows with opaque systems, security failures, and unfair outcomes. Linked to privacy and fairness in (Afroogh et al., 2024)

Transparency (multi-dimensional)

Making methods, data limits, and uncertainty visible to stakeholders. Can support accountability, but full openness may conflict with privacy.

Harm type «non-transparency» in (Mittelstadt et al., 2016)

Explainability (multi-dimensional)

Ability to describe how a system reached an output. Needed when «black box» systems affect high-stake decisions and when auditors or affected people must understand limits.

Closely tied to transparency and accountability in (Afroogh et al., 2024)

Consent (in practice)

Meaningful agreement to how data is collected and used. Secondary statistics and linked map data rarely re-ask each respondent. Autonomy harm in (Mittelstadt et al., 2016). Purpose limitation in (UK Data and AI Ethics Framework)

Transversal

Ethics, privacy, and fairness apply across the whole pipeline, not only in analytics or the brief.

Data quality carries ethical requirements

• Missing, biased, or mis-labelled data can harm groups silently
• Assess is where we inspect structure, coverage, and risk before analytics
• Veracity is a Big Data dimension with direct policy consequences

Quasi-identifiers in GEIH

• Department, age, sex, education, occupation combinations
• List sensitive columns before any publishable aggregate

CANDIDATOS_QI = ["DPTO", "MPIO", "AREA", "P6040", "P6240", "P6160"]
presentes = [c for c in CANDIDATOS_QI if c in df.columns]
tabla_qi = pd.DataFrame({
    "columna": presentes,
    "valores_unicos": [df[c].nunique() for c in presentes],
})

Disclosure preview

• Count rows in small department × age cells
• Alert when a cell has very few respondents

work = df[["DPTO", "P6040"]].copy()
work["banda_edad"] = pd.cut(work["P6040"], bins=[0, 17, 29, 44, 59, 120])
celdas = work.groupby(["DPTO", "banda_edad"], observed=True).size()
celda_min = celdas.min()

Join survey and map data

• Overpass query for POIs by department
• Compare with GEIH sample counts
• State limits: coverage, consent, ecological fallacy

query = """
[out:json][timeout:60];
area(3601380130)->.a;
node["amenity"="school"](area.a);
out;
"""
poi_count = len(overpass_post(query).json()["elements"])

Lecture 2

• Ethics as a crucia requirement
• Human-rights at the center
• Transversal governance
• Data Assess

Lecture 3

• Data formats
• Data structures
• Storage

Notebooks and templates

• Lecture 1 - access and manifest
• Lecture 2 - ethics walkthrough
• Week 1 group notebook
• Week 1 reflection template

Group zip by Monday

• Executed week 1 group notebook
• manifest.json
• Map points by department file

Individual reflection by Tuesday

• Covers Lecture 1 and Lecture 2 prompts (R1–R5)
• Includes ungraded week feedback (R6)
• Use the week 1 reflection template on the course page

Saturday 6 June, 07:00 to 13:00 Colombia time

• Shared block for Lecture 1 and Lecture 2
• Reading discussion, group notebooks, project clinic, plenary
• Full timetable: Week 1 session plan (PDF)