AI Specialization Program

 WEEK 2 Lesson 3 

Title: Data Cleaning & Transformations (Pipelines Mindset)

Lesson Objectives:

By the end of this lesson, students will be able to:

 ✔ Handle missing values properly

✔ Decide when to drop vs impute data

✔ Convert incorrect data types

✔ Remove duplicates safely

✔ Create derived (feature) columns

✔ Build reusable transformation workflows

✔ Think like a production data engineer

  e

Lesson Overview

Real-world data is never clean.

Before Machine Learning begins, data must be:

Complete

Consistent

Correctly formatted

Structured for reuse

Professional AI engineers spend 60–80% of their time cleaning data — not building models.

This lesson introduces the Pipelines Mindset, teaching students how to clean data systematically and reproducibly, not randomly.

1. Why Data Cleaning Matters

Messy data causes:

Wrong predictions

Model bias

Training instability

Deployment failures

Example problems:

Problem

Example

Missing values

Age = NaN

Wrong type

Price stored as text

Duplicates

Same customer twice

Inconsistent format

Lagos / lagos / LAGOS

AI models learn patterns — including bad ones.

 Garbage In = Garbage Out.

2. Handling Missing Values

Missing values appear as:

Python:

NaN

None

Detect Missing Data

Python:

df.isnull().sum()

Strategy 1 — Drop Missing Rows

Use when data loss is small.

Python:

df = df.dropna()

Simple

Can remove useful information.

Strategy 2 — Fill (Impute) Values

Fill with Mean

Python:

df["age"] = df["age"].fillna(df["age"].mean())

Fill with Median (better for outliers)

Python:

df["salary"] = df["salary"].fillna(df["salary"].median())

Fill with Category

Python:

df["city"] = df["city"].fillna("Unknown")

Engineer Thinking Rule

Situation

Action

Few missing rows

Drop

Important feature

Impute

Categorical data

Fill label

Large missing portion

Investigate source

3.Type Conversion

Reposi3. Type Conversion

Data often loads incorrectly.

Example:

"50000" → string instead of number

Check Types

Python:

df.dtypes

Convert Types

Python:

df["price"] = df["price"].astype(float)

Convert dates:

Python:

df["date"] = pd.to_datetime(df["date"])

Why This Matters

Models cannot learn from text pretending to be numbers.

4. Removing Duplicates

Duplicates distort statistics and model learning.

Detect duplicates

Python:

df.duplicated().sum()

Remove duplicates

Python:

df = df.drop_duplicates()

Real-World Example

Duplicate customers may cause:

Biased predictions

Overfitting

Revenue miscalculations

5. Creating Derived Columns (Feature Engineering Basics)

Derived columns create new useful information.

Example dataset:

salary    |   age

60000   |    30

Create New Feature

Python:

df["salary_per_age"] = df["salary"] / df["age"]

Convert Categories

Python:

df["is_senior"] = df["age"] > 50

Why Derived Features Matter.

Better features → Better models.

Often feature engineering beats algorithm choice.

6. The Pipelines Mindset (VERY IMPORTANT) 

Beginners clean data randomly.

Professionals build repeatable pipelines.

A pipeline means:

Same cleaning steps applied every time automatically.

Example Cleaning Workflow

Python:

def clean_data(df):

    df = df.drop_duplicates()

    df["age"] = df["age"].fillna(df["age"].median())

    df["salary"] = df["salary"].astype(float)

    df["city"] = df["city"].fillna("Unknown")

    df["salary_per_age"] = df["salary"] / df["age"]

    return df

Use Pipeline.

Python:

df = clean_data(df)

Why Pipelines Are Powerful

• Reproducible

• Production-ready

• Easy deployment

• Team collaboration

• Prevents human error

This is how real AI companies operate.

7. Engineering Best Practices

Always:

• Inspect data first

• Clean before modeling

• Document transformations

• Keep raw data unchanged

• Write reusable functions

Golden Rule:

Never manually edit datasets — automate cleaning.

Mini Practice Exercise

Students should:

1. Load dataset

2. Detect missing values

3. Fill numeric columns using median

4. Remove duplicates

5. Convert one column type

6. Create a derived feature

7. Real-World Connection

In production AI systems:

• Cleaning scripts run automatically

• Data pipelines execute daily

• Models retrain using cleaned data

• Data cleaning is part of engineering, not preparation.

Lesson Summary

Students learned how to:

• Handle missing values

• Choose drop vs impute strategies

• Convert incorrect data types

• Remove duplicates

• Create derived features

• Build reusable cleaning pipelines

Lesson Outcome

By the end of Lesson 3, students can:

✔ Systematically clean messy datasets

✔ Prepare structured data for Machine Learning

✔ Apply a pipeline mindset used in industry AI systems

✔ Week 2 Progression

Lesson 1 → NumPy Foundations ✔ 

Lesson 2 → Pandas Exploration ✔

Lesson 3 → Data Cleaning & Pipelines ✔

Next:

Lesson 4 → Feature Engineering Foundations

Week 2 — Lesson 4 is where students make a BIG leap:

Feature Engineering for Machine Learning (Industry Methods)

This is where they start thinking like real ML engineers.

Powered by Soft AI Africa | Training the Next Generation of AI Leaders in Africa.