Seeing data

The road ahead: machine learning

Mean and median: two kinds of average

import pandas as pd
marks = pd.Series([55, 62, 68, 70, 71, 74, 78, 99])
print(marks.mean())
print(marks.median())
marks2 = pd.Series([55, 62, 68, 70, 71, 74, 78, 300])
print(marks2.mean())
print(marks2.median())

72.125
70.5
97.25
70.5

By hand: mean and standard deviation

Standard deviation measures how far values typically are from the mean. Fill each column, then add up the last one.

values = [2, 4, 4, 4, 5, 5, 7, 9]
# mean = sum / count = 40 / 8 = 5
# variance = sum of squared gaps / count
#          = 32 / 8 = 4
# standard deviation = square root = 2

describe(): the whole summary at once

values = pd.Series([2, 4, 4, 4, 5, 5, 7, 9])
print(values.std().round(2))
print(values.describe().round(2))

2.14
count    8.00
mean     5.00
std      2.14
min      2.00
25%      4.00
50%      4.50
75%      5.50
max      9.00
dtype: float64

Histogram: the shape of one column

A histogram groups values into bins and counts each bin. Spam and normal messages have very different length shapes.

Codecopy

import matplotlib.pyplot as plt
df = pd.read_csv("sms.tsv", sep="\t", header=None,
                 names=["label", "message"])
df["length"] = df["message"].str.len()
ham = df[df["label"] == "ham"]["length"]
spam = df[df["label"] == "spam"]["length"]
plt.hist(ham, bins=40, alpha=0.6, label="ham")
plt.hist(spam, bins=40, alpha=0.6, label="spam")
plt.xlabel("message length (characters)")
plt.ylabel("number of messages")
plt.legend()
plt.show()

Watch it: how a histogram is built

Bar chart: counts per category

Use a bar chart to compare counts of categories. Always label the axes.

Codecopy

counts = df["label"].value_counts()
plt.bar(counts.index, counts.values,
        color=["tab:blue", "tab:orange"])
plt.ylabel("number of messages")
plt.title("Class imbalance in the SMS data")
plt.show()

Scatter plot: two columns at once

Each dot is one student. The pattern shows how marks change with hours studied (a small invented example).

Codecopy

hours = [1, 2, 3, 4, 5, 6, 7, 8]
marks = [42, 48, 55, 60, 58, 70, 75, 81]
plt.scatter(hours, marks)
plt.xlabel("hours studied per week")
plt.ylabel("exam marks")
plt.show()
r = pd.Series(hours).corr(pd.Series(marks))
print(round(r, 3))

Output

0.983

Correlation: how strongly two columns move together

Correlation r runs from -1 to +1. Close to +1: rise together. Close to -1: one rises as the other falls. Near 0: no straight-line pattern.

import numpy as np
rng = np.random.default_rng(0)
x = rng.uniform(0, 10, 60)
ys = {"positive": 2 * x + rng.normal(0, 2, 60),
      "none": rng.normal(10, 4, 60),
      "negative": -2 * x + rng.normal(0, 2, 60)}
fig, axes = plt.subplots(1, 3, figsize=(10, 3))
for ax, (name, y) in zip(axes, ys.items()):
    r = np.corrcoef(x, y)[0, 1]
    ax.scatter(x, y, s=12)
    ax.set_title(f"{name}: r = {r:.2f}")
plt.show()

Correlation: how strongly two columns move together: the result

Watch it: correlation as the points tighten

Correlation is not causation

Box plot: spread and outliers at a glance

The box holds the middle 50% of values, the line is the median, and dots are unusual values (outliers).

Codecopy

plt.boxplot([ham, spam], labels=["ham", "spam"])
plt.ylabel("message length (characters)")
plt.show()
print(ham.median(), spam.median())

Output

52.0 149.0

Common mistakes with charts and statistics

Read the last line of an error first. It names the problem.

plt.scatter([1, 2, 3], [4, 5])

ValueError: x and y must be the same size

print(df["label"].mean())

TypeError: Cannot perform reduction 'mean' with string dtype

plt.hist(df["lenght"])

KeyError: 'lenght'

Today's toolkit

Homework 7

Due before Lecture 8 · [date]