Real data is messy: blanks, “N/A,” a 999 that means “didn’t answer,” the same city spelled three ways. Cleaning it isn’t janitorial work — it’s a series of judgment calls, and every one changes the answer.
No reading — this page is the reading. Runestone’s datasets come pre-cleaned, which is exactly why we need this lesson.
Big Data. Week 1 got data in and asked it questions. Week 2 is about trust: what you had to do to the data first, and how far you can push the answer.
Today’s practice set has every classic problem in 18 rows: three kinds of blank, a code pretending to be a number, and one city spelled four ways.
Data cleaning is finding and handling values that would make a result wrong. The usual suspects:
| Problem | Looks like | Typical rules |
|---|---|---|
| Missing data | blank, N/A, a space | drop the row · fill with a value · leave it and skip it per question |
| Placeholder codes | 999, -1, 0 | treat as missing, not as a number |
| Inconsistent text | Brea, brea, BREA, Brea | normalize: trim spaces, one capitalization |
| Outliers | a value far from the rest | check it: impossible → error; unusual but real → keep |
| Duplicates | the same record twice | remove the extra copy |
# Rule: blank, "N/A", or spaces → missing. 999 → missing. Missing rows are skipped. hours = [] for row in rows: h = row["hours"].strip() # " " becomes "" if h == "" or h == "N/A" or h == "999": continue # skip this row hours.append(int(h)) city = row["city"].strip().title() # "brea " → "Brea"
18 rows from a summer-jobs survey: id, city, hours_per_week. Flip each rule and watch the answers
move. Same data, different decisions, different results.
Find every problem in the sandbox’s raw rows. For each, write a rule and one sentence of why. Then compare with a neighbor: different rules? Different answers? Both defensible?
Five questions. The recurring shape: which of these could distort the result?
Missing data is rarely random. People skip questions they find uncomfortable; clinics with fewer staff leave more fields blank; forms that only accept certain name formats reject the rest. Drop the incomplete rows and you may have quietly dropped a whole group — then published a result about “everyone.”