Introduction: Surveys Are Everywhere

PSCI 3802: Political Polling

Marc Trussler

When we say “polling”…

We think of this:

Election polls are the visible tip of the iceberg. They hide how much of what we know about the world comes from surveys.

It’s actually everywhere

  • The unemployment rate: two surveys (households + businesses), not a count of every job.
  • “38% of the US has a college degree”: a survey. The Census only asks sex, age, race, home ownership.
  • Even great admin records (COVID doses by race) need a survey the moment you want a rate.

Why do elections feel different?

Because we get to see the truth.

  • We argue for weeks about whether the election polls were “accurate.”
  • Is the unemployment rate accurate? We have no idea.
  • There is no “Unemployment Election” to benchmark it against.

The error is always there. Elections are the rare case where we get to see it.

What makes a survey a survey?

According to Groves:

A survey is a systematic method for gathering information from a sample of entities for the purpose of constructing a quantitative description of the attributes of the larger population of which the entities are members.

  • Systematic: a defined, repeatable method
  • Quantitative: attributes measured systematically
  • Sample → population: use a part to understand the whole

A census (counting everyone) is not a survey. The population is the point.

1936: the Literary Digest disaster

“More data is better”: 10 million postcards, 2.4 million replies.

  • Prediction: Alf Landon wins, 55%
  • Reality: FDR wins 61%, loses only 8 electoral votes

Two problems we’ll spend the semester on:

  • Sample-frame bias: their lists (phone + car owners) skewed Republican
  • Non-response bias: Landon supporters mailed their cards back at higher rates

Gallup and quota sampling

The “fix” targeted only the frame: set subgroup targets, fill them.

  • Smaller, better-targeted data beat the Digest’s 2.4 million
  • Called 1936, 1940, 1944; the pollsters became stars

But quotas let interviewers choose who filled them…

1948: Dewey Defeats Truman

Interviewers avoided the very poor and very rich (“the unshaven and sweaty”). Truman won.

Even with the right quotas, human choice in who gets interviewed introduces bias.

The resolution: random sampling

Stop hand-picking respondents to hit targets.

Let chance choose them.

Let’s play God: build a population where we know the truth, then sample from it.

Playing God: a known population

50% male, 60% white, 25% rich, 55% college, vote split 50/50 (traits correlated).

           male           white            rich 
          0.501           0.599           0.251 
        college vote.republican 
          0.551           0.501 

Shrink the sample

A random sample recovers the population in expectation, even a tiny one.

           male white  rich college vote.republican
n = 50000 0.502 0.601 0.253    0.55           0.505
n = 1000  0.524 0.601 0.263    0.56           0.505
n = 100   0.490 0.630 0.200    0.54           0.530

Noisier as it shrinks, but no systematic tilt.

Size doesn’t matter

A random sample of 1,000 works as well in China (1.4B) as in Canada (40M).

The soup analogy: stir the pot well (that’s the randomness), and one spoonful tells you the seasoning. The size of the pot is irrelevant.

How polls go wrong: biased sampling

Break the randomness: let college grads be likelier to be interviewed (20% vs 15%).

           male           white            rich 
          0.479           0.613           0.249 
        college vote.republican 
          0.622           0.489 
  • We overstate the college share (here by ~7 points)…
  • …and because traits are correlated, the Republican estimate falls too, always in the same direction.

A tilt on one variable quietly biases the ones you actually care about.

Takeaway

  • The only way to understand a mass public is a survey.
  • A survey is only as good as its sampling.
  • Random selection buys unbiased inference from a tiny sample.
  • The moment selection depends on who people are, bias creeps into every estimate.

The population is the point.