Evaluation Lab · Lecture 11

Underfit and overfit: follow the trend, not the noise

The 15 noisy points from slide 3 (made with np.random.default_rng(1)). Raise the polynomial degree and watch the training error and the error on 200 new points from the same process.

degree =

Sample

show the true pattern (sin) show the 200 new points

Error as the curve gets more flexible

training error (MSE)

error on new points (MSE)

What to notice

Training error only ever goes down as the degree rises. It cannot tell you when to stop.
The error on new points falls, then rises again: that turn is where overfitting starts.
A degree-14 curve passes through all 15 points: zero training error, wild everywhere else.

Try this

Find the degree with the lowest error on new points. Is it the same for another sample? Compare degree 12 on rng(1) and rng(2). Why do the curves differ so much while degree 3 barely changes?

Choose depth honestly: cross-validation first, the test set once

Breast cancer data, the lecture's split. Choose max_depth using 5-fold cross-validation on the training data. Only then score the test set, once.

max_depth =

5-fold cross-validation for this depth

Accuracy against depth

show the test curve (for discussion after choosing)

What to notice

Each round, one fifth of the training data validates and four fifths train. Every row validates exactly once.
The five scores differ: one split alone would give a lucky or unlucky number.
Choosing on the test set and then reporting it is cheating: the score looks better than the model really is.

Try this

Homework 11.4: for depth 2, which fold scores lowest and which highest? Why do they differ? Tick "show the test curve". Which depth would someone pick if they cheated? How much better would their reported score look?

Precision and recall: the spam model's mistakes

The Lecture 6 spam model on its 1,034 test messages. Move the threshold and watch the two kinds of mistake trade against each other.

spam if P(spam) >

By hand (homework 11.1)

TPFPFNTN

Every threshold at once: precision against recall

Each dot on the curve is one threshold. Moving right catches more spam; moving down loses more real messages.

What to notice

Precision: of the messages flagged as spam, how many really were spam? Recall: of all real spam, how much did we catch?
Lowering the threshold raises recall and usually lowers precision. You cannot have both for free.
For a spam filter, losing a real message (a false positive) is the worse mistake, so watch precision.

Try this

Check slide 12: at 0.5, do you get 0.959, 0.893 and 0.925? Find the lowest threshold that still has precision 100%. How much spam slips through there? Do homework 11.1 on paper first, then type the numbers here to check.

The accuracy trap: when 95% is a failure

100 people are screened for a disease. Change how rare the disease is and how good the test is, and see what accuracy hides.

sick people =

Model

accuracy

recall (sick found)

precision

The 100 people

sick, foundsick, missedhealthy, false alarmhealthy, cleared

Which metric matters? (slide 13)

What to notice

When one class is rare, "always say the common class" gets a high accuracy and is useless.
Recall for the rare class shows the failure immediately: 0 sick people found.
A real test can have lower accuracy than the lazy one and still be far more useful.

Try this

With 5 sick people, find a real test whose accuracy is below 95% but whose recall is above 80%. Set sick people to 50. Is accuracy still misleading for the lazy model?

Works offline: no internet needed. The data and the trained models are built into this page. Every number was calculated with scikit-learn and numpy and checked, so it matches what you will see in Jupyter or Colab.