Real Data Lab · Lecture 5

Look inside: rows, columns and single messages

The full SMS Spam Collection is built into this page: 5,572 real messages. Look before you model.

after df = df.drop_duplicates()

Show

n = 5

df.shape

matching rows

Read one message: df.loc[row, "message"]

row =

The table

What to notice

The numbers on the left are the index: row labels, not positions.
After drop_duplicates() the index has gaps, so df.loc[7] may no longer exist.
Long messages are cut with ... on screen only; the full text is still there.

Try this

Read 5 spam messages in full. Write down 3 patterns you notice (homework 5, question 2). Tick drop_duplicates and look for row 7. What does pandas say? Find all messages containing "lol". Are any of them spam?

Word clues: is this word a good sign of spam?

For any word, see what share of spam and of normal (ham) messages contain it, exactly as str.contains and groupby compute it.

word =

after drop_duplicates() regex=False (search the text literally)

Share of messages containing it

Your comparison (best spam clue first)

Examples that contain it

What to notice

A good clue is common in spam and rare in ham. Being common in spam is not enough.
str.contains finds text inside words too: "free" also matches "freedom".
By default the text is a pattern (a regular expression): "." means any character, "(" is an error. Tick regex=False to search literally.

Try this

Find a word that beats "free": a larger spam share with a ham share still under 2%. Try "u" and "ok". Why is a high share in ham useful too? Type "(" with and without regex=False. Explain the difference.

Group and compare: spam messages are longer

groupby("label") splits the table into a spam group and a ham group, then summarises each. Then try the simplest possible model: a single length rule.

after drop_duplicates()

Message length by label

bins = 40

A one-number model: can message length alone find spam?

Rule

t / lo = 100

Accuracy of the rule

Always say "ham"

Spam caught

Ham wrongly flagged

What to notice

The two groups overlap: many ham messages are long, some spam is short.
The mean is pulled up by a few very long ham messages; the median is less affected.
A clear difference in averages is not enough to classify: no "longer than" rule beats always saying ham. Lecture 6 uses every word instead.

Try this

Press "Find the best rule" for "length > t". Explain why the best t means "never say spam". Switch to the band rule. How much does it beat 87.4% by, and how many ham messages does it wrongly flag? Why is 87% so hard to beat? (Hint: look at value_counts.)

Duplicates: the same text, sent many times

Spammers send one text to many people, and some everyday replies repeat too. See what repeats, and why duplicates make a test unfair.

Most repeated messages

Why it matters: a split with duplicates

Split the messages at random: 80% to study (train), 20% for the exam (test). Count how many exam messages also appear, word for word, in the study set.

drop duplicates first

Test messages

Already seen in training

What to notice

duplicated() marks the 2nd, 3rd, … copy; the first copy is kept.
With duplicates, some exam questions were already in the study notes: the score looks better than it really is.
Dropping duplicates makes the test honest. Lecture 12 calls this problem leakage.

Try this

Make several splits. Roughly how many test messages leak each time? Are the leaked messages mostly spam or ham? Why might that be?

Works offline: no internet needed. The whole dataset is built into this page. Every number is calculated the same way as pandas and matplotlib and was checked against the real libraries, so it matches what you will see in Jupyter or Colab.