14 Review II
As with the pre-midterm review, knowledge of the contents of this review chapter is not necessarily sufficient to ensure success on the final. This summary is here as a reference guide to what we’ve covered, and allows me to emphasize the things that I find to be important.
14.1 Non Response and Weighting
The Big Picture
Non-response is probably the most important source of error in moden polling, given rapidly declining response rates. With coverage error it was plausible that few people were covered, and if they were not covered, that they had similar opinions to the people included. We definitely have non-response, and people who don’t respond are definitely different than those who do. Survey weighting can help us with ignorable non-response, but is not a panacea.
Things to think about
- Unit vs Item non-response
- Response, contact, and cooperation rates.
- Equations for non-response bias.
- When survey weighting creates ignorable non-response.
- What variables can we use for weighting.
- Cell weighting
- Rake weighting
- Trimming weights
- Design effect, effective sample size, standard error corrections.
Questions
Explain the difference between unit and item non-response.
Explain the difference between response, contact, and cooperation rates. Where are we most likely to see politically motivated non-response enter and bias our results?
Which of the following variables are inappropriate for weighting? Why? Gender, race, employment status, need for cognition, housing type (rent vs own), voting history.
Why does a large variance in weights increase survey variance?
How can you use the design effect to communicate a more accurate standard error?
I want to us cell weighting to adjust for gender (2 categories), race (5 categories), education (5 categories), and age (4 categories). How would I do this, and why is this approach unlikely to work?
14.2 Election Polling
The Big Picture
Election polling is funamentally more difficult to do because our population (voters) doesn’t exist yet. On top of the usual sources of error, there is a massive potential for coverage error to affect the survey estimates. Plus: the world gets to find out right away how wrong you are! Correctly specifying a likely voter model is key, though there is no one set of best practices. The 2020 election provides the best case study of why election polls are particularly biased now, and the main culprits are mis-measuring new/infrequent voters and partisan non-response.
Things to think about
- Why is this a coverage problem?
- Self-reports vs covariate approach to identifying LV
- Deterministic or probabilistic approach to applying LV to weights
- How does relying on self-reports generate ineligible units and under coverage
- What sort of variables are helpful (or not) in LV models
- How to measure accuracy of election polls (absolute vs relative)
- Where did 2020 polls miss?
- Why can we rule out survey mode as an explanation?
- Why are new voters and partisan non-response the leading contenders for the polling miss?
Questions
Of the following variables, which would you include in a likely voter model? Which would you leave out: same year primary participation, past vote, party ID, self-reported likelihood of voting.
What has to be true of voters and non-voters for a mis-specified likely voter model to create bias?
What is the difference between absolute and relative (or directional) error in election polls? Why are each helpful to know?
What can we learn from the fact that 2020 election polls missed more in highly Republican places?
Why can we rule out survey mode as an explanation?
Why are new voters and partisan non-response the leading contenders for the polling miss?
14.3 Advanced Topics in Weighting
The Big Picture
“Usually we assume the problem is that group X is too small, but the actual problem may be that group X is too weird.” The people in our surveys are weirdos. They are the kids in class straining their hands to answer the question. They are different from the population in ways that we cannot fully measure, which makes non-response non-ignorable. Bailey gives us a way to understand this mathematically and to visualize it. He also applies this to survey mode and shows why non-probability surveys have way more potential for bias. My research shows how bad of a problem partisan non-response is, and in particular, why simple weighting cannot fix it.
Things to think about
- When is non-response non-ignorable?
- How does NINR show up in terms of who is visible in our surveys?
- Understanding the “tilted fish” diagrams and why they represent NINR.
- The Meng identity equation for survey bias.
- Why does the ratio of sample size to population size affect survey bias?
- Why the problem of NINR bias is so much worse (potentially) in non-probability samples.
- If this problem is so bad, why are our surveys so good?
- How the POQ paper identified partisan non-response using and RBS poll.
- Why correcting for Partisan non-response doesn’t fix the problem
- Variation in polls over time or across polls is mostly just partisan non-response
- Traditional weighting cannot control for partisan non-response because it is happening within weighting cells.
Questions
Explain the difference between non-ignorable and ignorable non-response.
Bailey taught us that our survey respondents are “weird”. Making reference to the Meng equation, how does this “weirdness” translate into bias?
In the sampling chapter we learned that population size doesn’t matter, but the Meng equation explicitly states that the relative size of the sample and the population (data quantity) has a big impact on bias. Why is there this discrepancy?
How does the “random contact” nature of a probability sample reduce bias relative to a non-probability sample?
Why does within-weighting-cell partisan nonresponse make political polling difficult?
Contrast the relative merits and drawbacks of weighting to party ID, party registration, and past vote.
14.4 Survey Experiments
The Big picture
The modern online survey environment makes experiments much easier, which opens up a range of possibilities for understanding causality. We can never have true causality because of the fundamental problem of causal inference, but randomization gets us close. When we think about causality, we want to think about comparing two groups that are identical except for exposure to treatment.
Things to think about
- The fundamental problem of causal inference
- Flaws in cross-sectional designs
- Randomization as the solution to the problem
- The need to generate appropriate counterfactuals
- Difference between random sampling and random assignment to treatment
- Regression discontinuity as an example of non-experimental causality
- Experiments in practice via the APSR article
- “Proving” a null
- Manipulation checks
- Statistical power and false negatives
Questions
I want to know the effect of partisanship on support for welfare policies, so I use a survey to find the correlation between being a Democrat (or Republican) and supporting welfare policies (or not). From a causal inference perspective, what is the problem with this setup? How would you modify this to better get at the causal effect of party?
I want to know the effect of Biden dropping out of the 2024 election on the percent of people who supported Donald Trump. Luckily, I was in the field with a survey that was interviewing people while this event occurred. What kind of analysis can I do that will help me uncover the causal effect of the dropout on Trump support?
Imagine I completed a survey experiment and I found non-significant results. What two things do I have to rule out in order to conclude that there is no true effect of this treatment?
14.6 Panel Surveys
This section will not be on the final exam.
The Big picture
Unpacking causal impacts over time is incredibly difficult, but are made easier by panel studies, which involve re-interviewing the same people over time. By looking at changes within people we can rule out any confounding factors that only vary between people. We saw this applied to understanding the effect of smoking on health, the effect of partisanship on religious identity, and the effect of minimum wage hikes on employment. The more formal method for using panel data is a difference-in-difference approach.
Things to think about
- The limitations of cross-sectional research to understand temporal order of things.
- How a panel study helped rule out confounding factors for the link between smoking and cancer.
- The difference between sorting and socialization as explanations for the link between religion and politics.
- How panel data helps us understand how a base level of partisanship relates to a change in religion.
- Logic of a difference-in-difference approach and how it allows us to construct a counter factual like we would have in an experiment.
- The parallel trends assumption.
Questions
What are some political science questions that would require a panel survey? Why would a cross-sectional survey not work for your examples?
Explain how an increasing correlation across time between religion and partisanship can be caused by both sorting and socialization.
I want to know the effect of Tariffs on inflation. To do so I will use a Difference in Difference approach using the US as the treatment case and the UK as the control case. I gather information on inflation before and after Trump’s tariffs in both countries. Would this study satisfy the parallel trends assumption?
14.5 Social Desirability Strategies
The Big picture
While there are a few ways to deal with social desirability through careful question wording and survey ordering, there are several other advanced techniques that can do a better job. List experiments and Randomized response experiments leverage experimental logic and math to provide anonymity to respondents. IATs bypass explicit thoughts and directly access our implicit associations between concepts.
The question with the most costly social desirability bias for political science is the voter turnout question. Through careful comparisons, Jackman and Spahn show that social desirability is only part of the problem with this question, however.
Things to think about
Questions
Write a question that measures drug use that makes use of the standard advice for reducing social desirability bias.
Design a list experiment that measures support for political violence. Make sure that your experiments follows the best practices discussed in lectures.
Making references to the theories of Zaller and Feldman, discuss how IATs work to access our stored political considerations. Give an example of an implicit attitude that could be studied in this way.
For the Jackman and Spahn group make a table that shows what is being compared to what to show the effects of social desirability, non-response, and mobilization.