Lecture 4.2 · Interactive lab

Word Counting Lab

Split messages into words, count them in a dictionary, and build a spam score learned from examples

Introduction to Data Science and Machine Learning
Md Mahbub E Noor · University of Barishal
All labs

Split and clean: from a message to a list of words

Type a message and watch each step. The words at the end are exactly what Python would count.

Step by step

The words

len(words)
different words

What to notice

  • .split() cuts at any run of spaces; punctuation stays stuck to the word.
  • Without .lower(), Win and win are two different words.
  • Same colour = same word. These are the words a dictionary will count.

Try this

Dictionary builder: count every word, one at a time

This is count_words from the lecture. Each step handles one word: look up its count (or 0), add one, store it back.

counts (the dictionary)

Trace

Printed by print(counts)

What to notice

  • A new word enters the dictionary with count 1; a word seen before goes up by 1.
  • counts[word] fails with KeyError for a word not seen yet. counts.get(word, 0) gives 0 instead.
  • The dictionary keeps words in the order they were first seen.

Try this

Spam vs normal: which words are clues?

Count the words in spam examples and in normal examples separately. A word that is common in one group but rare in the other is a strong clue.

Word counts in both groups

Spam clue ← → normal clue (top words)

What to notice

  • free appears only in spam: a strong spam clue. you appears only in normal messages.
  • now appears in both, so it tells us little.
  • These clues were learned from examples; nobody typed a rule.

Try this

Spam score: a filter learned from examples

Each word adds its spam count and subtracts its normal count. Test the score on new messages, then teach it more examples and watch it improve (or not).

spam_score(message)

Test on new messages the filter has never seen

Spam caught
Spam missed
False alarms
Correct

What to notice

  • The score is a number, not just yes or no: bigger means stronger spam evidence.
  • Words never seen in training add 0: the filter knows only what its examples taught it.
  • Teach many spam examples and few normal ones, and every word starts to look like spam. Balancing fixes part of this. Lecture 6 does it properly.

Try this

Works offline: no internet needed. The Python shown here runs in your browser with the same rules as Python 3, so results and error messages match what you will see in Jupyter.