Classifier Lab · Lecture 10

k-NN by hand: let the nearest neighbours vote

Click anywhere on the plane to move the new point ?. The circle grows until it holds k neighbours; they vote, and the majority wins.

k =

Clicking the plane moves ? adds an A point adds a B point

What to notice

Distance = √((x₁ − x₂)² + (y₁ − y₂)²): straight-line distance on the plane.
Same data, different k, different answer: k = 1 says B, k = 3 says A.
With an even k the vote can tie. scikit-learn then picks the class that comes first (A).

Try this

Homework 10.1: predict (4, 4) with k = 1, 3 and 5 by hand, then check here. Find a spot where k = 1, 3 and 5 all give different answers… or prove it cannot happen with two classes. Add one B point so that the slide's answer for k = 3 changes.

Decision boundaries: where the model's answer changes

Every pixel is coloured by the class the model would predict there. Models are trained on the lecture's training split of the iris data (70%), using only two measurements so we can draw them.

Measurements

Model

k =

show test flowers (hollow circles)

training accuracy

test accuracy

setosaversicolorvirginicafilled = training flower, hollow = test flower

What to notice

k = 1 draws jagged islands around single flowers: almost 100% on training data, worse on test data. (Not quite 100% with sepals: some flowers have identical measurements but different species.)
Large k smooths the regions; very large k ignores real detail.
A tree cuts the plane with straight horizontal and vertical lines, one question at a time.

Try this

On the sepal measurements, find the k with the best test accuracy. Homework 10.4: compare a tree of depth 2 and depth 10. Describe the difference in one sentence. Why are the petal measurements so much easier for every model?

Read the tree: follow the questions for one flower

A tree trained on all four measurements (the lecture's split). Move the sliders to describe a flower and watch its path light up.

max_depth

What to notice

Each box asks one yes/no question. Yes (≤) goes left, no goes right.
The coloured bar shows the training flowers that reached the box; leaves answer with the majority.
Nobody typed these rules: the tree chose the questions that made each side as pure as possible.

Try this

Slide 11: check the three flowers. Do you get 0, 1 and 2? With depth 2, does sepal length ever change the answer? Why not? Press "a random test flower" until the tree gets one wrong. Which leaf was it in?

Compare the models: complexity, and the scale of the features

All four measurements, the lecture's split. Training accuracy always rewards complexity; test accuracy tells the truth.

k-NN: accuracy as k grows

k =

Decision tree: accuracy as depth grows

max_depth =

Change the units of one measurement (cm → mm is × 10)

Featuremultiply by

k-NN (k = 5), test accuracy

tree (depth 3), test accuracy

What to notice

k = 1 and deep trees score 100% on training data: they memorise.
k-NN measures distance, so a feature with bigger numbers dominates the vote. Trees only compare one feature with a threshold, so units do not matter.

Try this

Which k and which depth would you choose? Use the test curve only after deciding how (Lecture 11 shows the honest way). Multiply sepal width by 100. What happens to k-NN, and why?

Works offline: no internet needed. The data and the trained models are built into this page. Every number was calculated with scikit-learn and checked, so it matches what you will see in Jupyter or Colab.