Preparing features

The road ahead: machine learning

Meet the data: invented student records

import pandas as pd
st = pd.read_csv("students.csv")
print(st.head())
print(st.isna().sum())

   hours  attendance  prior_gpa  dept passed
0    NaN          55       3.28   CSE    yes
1    NaN          88       2.37   EEE    yes
2    9.3          81       3.52  Math    yes
3    2.7          83       3.52   EEE     no
4    3.6          78       3.44   CSE     no
hours         8
attendance    0
prior_gpa     6
dept          0
passed        0
dtype: int64

Missing values: drop or fill?

print(len(st), len(st.dropna()))
filled = st.copy()
for col in ["hours", "prior_gpa"]:
    filled[col] = filled[col].fillna(filled[col].median())
print(filled.isna().sum().sum())

150 136
0

One-hot encoding: categories become 0/1 columns

A model cannot multiply the word CSE. One-hot encoding gives each category its own yes/no column.

One-hot encoding in pandas

pd.get_dummies creates the 0/1 columns · dtype=int shows 0 and 1 instead of False/True · Every category gets its own column · In pipelines we use OneHotEncoder instead (later)

dummies = pd.get_dummies(filled["dept"], prefix="dept", dtype=int)
print(dummies.head(4))

   dept_CSE  dept_EEE  dept_Math
0         1         0          0
1         0         1          0
2         0         0          1
3         0         1          0

By hand: why scale matters for k-nearest neighbours

Distance from student A (hours 2, attendance 90) to B (hours 10, attendance 85) and C (hours 3, attendance 60).

# Unscaled: attendance dominates.
#   C looks far, B looks close, although
#   B studies 8 hours more than A!
# After scaling both columns to similar ranges,
#   hours matter again and C is nearer.

Watch it: scaling changes the neighbours

StandardScaler: mean 0, standard deviation 1

Each column: subtract its mean, divide by its std · Every feature now has a similar range · Fit the scaler on training data only · Then transform both training and test data

from sklearn.preprocessing import StandardScaler
num = filled[["hours", "attendance", "prior_gpa"]]
scaled = StandardScaler().fit_transform(num)
print(num.mean().round(1).to_dict())
print(scaled.mean(axis=0).round(2) + 0.0, scaled.std(axis=0).round(2))

{'hours': 6.1, 'attendance': 70.0, 'prior_gpa': 3.0}
[0. 0. 0.] [1. 1. 1.]

Data leakage: the silent killer

Watch it: leakage happening, then prevented

A pipeline does every step, in the right order

Numbers: fill missing with the median, then scale · Categories: one-hot encode · Then the model · Inside cross-validation, each step is fitted on training folds only

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
X = st[["hours", "attendance", "prior_gpa", "dept"]]
y = st["passed"]
prep = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"),
                          StandardScaler()),
     ["hours", "attendance", "prior_gpa"]),
    ("cat", OneHotEncoder(), ["dept"])])
model = make_pipeline(prep, KNeighborsClassifier())
print(round(cross_val_score(model, X, y, cv=5).mean(), 3))

0.893

Does scaling really matter? Measure it.

def build(scale):
    num_steps = [SimpleImputer(strategy="median")]
    if scale:
        num_steps.append(StandardScaler())
    prep = ColumnTransformer([
        ("num", make_pipeline(*num_steps),
         ["hours", "attendance", "prior_gpa"]),
        ("cat", OneHotEncoder(), ["dept"])])
    return make_pipeline(prep, KNeighborsClassifier())
for scale in [False, True]:
    s = cross_val_score(build(scale), X, y, cv=5).mean()
    print("scaled" if scale else "unscaled", round(s, 3))

unscaled 0.687
scaled 0.893

Common feature-preparation mistakes

Read the last line of an error first. It names the problem.

KNeighborsClassifier().fit(filled[["hours", "dept"]], filled["passed"])

ValueError: could not convert string to float: 'CSE'

KNeighborsClassifier().fit(st[["hours"]], st["passed"])

ValueError: Input X contains NaN.

OneHotEncoder().fit_transform(st["dept"])

ValueError: Expected a 2-dimensional container but got <class 'pandas.Series'> instead. Pass a DataFrame containing a si ...

Today's toolkit

Homework 12

Due before Lecture P2 · [date]