Spam Model Lab · Lecture 6

Split: a study set and an exam set

The model learns from the training set and is graded on the test set it has never seen. Change the split and see what changes.

test_size = 0.2

random_state stratify=y

All 5,169 messages, one square each

train ham train spam test ham test spam

What to notice

The test messages are scattered at random through the data.
The same random_state gives exactly the same split every time; another number gives another exam.
stratify=y keeps the spam share the same in both parts. Without it, the share drifts a little.

Try this

Untick stratify and change random_state. How far does the spam share in the test set move? Set test_size to 0.5. What do you gain, and what does the model lose?

Words to numbers: what CountVectorizer builds

A model only understands numbers. fit learns the vocabulary from the training messages; transform counts words using that vocabulary only.

Training messages for fit (one per line) New message for transform

The count table (bag of words)

In the real model

Look up a word

What to notice

One row per message, one column per word, each cell a count. Word order is lost.
Words never seen in training (like "tomorrow") get no column: they are ignored.
One-letter words ("a", "u") are skipped by default, and punctuation is dropped.

Try this

Add a third training message. How many columns does the table have now? Make a new message that becomes a row of all zeros. Look up "u" and "ur" in the real model. Why is one missing?

Train and evaluate: an honest score

Multinomial Naive Bayes trained in your browser on the split from tab 1, giving the same numbers as scikit-learn. Then judge it on the test set, and compare with doing nothing.

spam if P(spam) > 0.50

smoothing alphadefault 1.0

Model accuracy

Always say ham

Error rate

Confusion matrix (click a cell to see its messages)

P(spam) for every test message

Messages in this cell

What to notice

Rows are the truth, columns the prediction. The diagonal is right; the other two cells are the two kinds of mistake.
Lowering the threshold catches more spam but sends more real messages to the spam folder.
The score depends on the split: another random_state gives a slightly different number.

Try this

Read the 5 false positives. Why might the model have flagged them? Find a threshold with zero false positives. How much spam slips through then? Change random_state in tab 1 and come back. Write down the three accuracies.

Try your own: what does the model see?

Type any message. Each word the model knows pushes towards spam or towards ham; words it never saw are ignored.

message

P(spam)

surely hamsurely spam

Evidence word by word

What did it learn? Strongest clue words in this model

What to notice

The model starts from the share of spam in training (about 13%), then each known word adds evidence.
"150p" and "tone" are clues because the data is from the UK around 2011, not because they are universally spammy.
A modern parcel scam slips through: it looks nothing like 2011 spam.
"gt" and "lt" are strong ham clues only because the raw data contains <#> placeholders in normal messages: a data quirk, not language.

Try this

Fool the model (homework 6, part 4): write a spam message it calls ham. Write a normal message it calls spam. Which words did it? Change one word in the parcel scam so that it gets caught.

Works offline: no internet needed. The whole dataset is built into this page. Every number is calculated the same way as pandas and scikit-learn and was checked against the real libraries, so it matches what you will see in Jupyter or Colab.