How to Read Someone Else's Poll

A poll lands in the middle of a campaign week. The top line says what it says, the staff argues about it for an hour, and somebody decides whether to change the plan. Almost none of the useful work in that hour is about the top line. It is about four decisions the write-up usually covers in a sentence each: who was sampled, who was screened out, how the answers were weighted, and how many people are actually behind the number being quoted.

What a margin of error covers

A margin of error describes sampling error and nothing else. It is the width of the band you would expect to contain the true value in 95 of 100 repetitions of the same procedure, and the phrase carrying the weight is the same procedure. Everything that could be wrong about the procedure itself sits outside the margin: who refused to answer, who was never reachable at all, how the question was worded, what was asked immediately before it, who conducted the interview, and which screen decided who counted as a voter. Those sources are usually larger than sampling error, and none of them shrink when the sample grows.

Two consequences are worth keeping at hand. A 2 point lead in a survey with a margin of plus or minus 3.5 points is a survey that cannot tell you who is ahead. And the difference between two surveys carries more uncertainty than either one alone, roughly 1.4 times as much when the samples are the same size, so two polls four points apart have not necessarily disagreed about anything.

There is also a quieter problem with the number itself. Weighting a sample so that it matches population targets makes the estimates less biased and less precise at the same time, because some respondents end up counting for more than others. Serious releases account for that and report a margin wider than the simple formula would give. Releases that quote the textbook figure for a heavily weighted sample are understating their own uncertainty, usually by a point or two and sometimes by much more.

Why the crosstabs break first

The number that gets quoted from a published survey is rarely the top line. It is a line from the crosstabs: support among independents, among voters under 35, among a particular group in a particular region. Those lines are computed from a fraction of the sample, and the margin printed at the top of the release does not apply to any of them.

How far a subgroup number can be from the truth

The same published survey, read four ways. Each band is the 95 percent sampling interval for the number of respondents behind that line of the crosstab.

The bottom band is the one to sit with. A result computed from 31 respondents has a sampling interval about 36 points wide. Printed as a single number it reads as a finding about young women in that county, and it is consistent with a tie and with a 30 point blowout at the same time.

The working rule is to find the unweighted count behind any subgroup number before repeating it. Several hundred respondents will support a subgroup read and several dozen will not, whatever the release says. If the release does not publish unweighted subgroup counts, that absence is itself worth noting.

Likely voter screens

Between the people who answered and the people reported as likely voters sits a screen. The screen is the point where a survey stops being a measurement and starts being a forecast.

Every option has a known problem. Self-reported intention inflates, because far more people say they are certain to vote than vote. Past-vote screens are more honest about history and worse about new registrants and young voters, who have no history to screen on. Modeled screens import whatever turnout assumption the modeler holds, which is a judgment about the future dressed as a property of the respondent.

Three likely voter screens on one set of interviews

The same respondents, the same weighting targets, three defensible definitions of who counts as a voter.

Five points of movement and a change in who leads, out of the same interviews. None of the three screens is indefensible. What you want from a published survey is a clear statement of which one it used, specific enough that you could reconstruct the other two.

House effects

A house effect is the persistent gap between one organization's results and the average of everybody else's, after timing and geography are accounted for. It is usually not dishonesty. It is the accumulated consequence of standing choices: the mode the organization interviews in, the frame it samples from, the screen it applies, the variables it weights on, the order it asks things in. Those choices are stable, so the gap is stable, and a stable gap is useful.

Nine surveys of one race

Each dot is one published margin in the same contest over the same three weeks. The two bands below compare the spread across the nine with what a single survey's stated margin of error covers.

Two things follow. The disagreement between organizations is wider than the uncertainty any one of them reports, which means the honest reading of a race is the spread rather than the average. And a single survey's distance from that spread tells you more about the organization than about the race.

The habit worth building is a short private record of each organization's direction over several races in your state. Which way they have missed, and by how much, is a better guide than whether they are considered good.

What a single survey can and cannot support

A single competent survey can support a large gap, the direction of a change within one organization's own series, and the rough standing of something nobody has asked about before. It can tell you an issue you had not considered is near the top of voters' minds. It can kill an assumption the campaign had been carrying for free.

It cannot support a 2 point lead, a subgroup estimate from a few dozen respondents, an estimate of what the electorate will look like, a claim about what caused anything, or the allocation of a media budget across markets. Using one survey for any of those is a decision to be precise about something nobody measured.

What to look for in the methods paragraph

The items worth checking sit close to the disclosure elements AAPOR's Transparency Initiative asks member organizations to publish at the same time they release findings, with further methodological detail available on request. Surveys that meet the list are easy to read. Surveys that do not are the ones to read hardest, because the absence of each item usually hints at the direction of the error.

• The field dates. A survey taken before a widely covered event is a historical document.
• The mode and the frame: how people were reached, and the list they were drawn from.
• Whether the sample is a probability sample or one in which respondents selected themselves, and whether the release says plainly which.
• Unweighted counts for the full sample and for every subgroup reported.
• The weighting variables and the population targets they were weighted to.
• Full question wording in order, including whatever was asked immediately before the vote question.
• Who paid for it.

A five minute procedure

Read the field dates and the sponsor. Find the unweighted count behind the exact number you care about. Read the full wording of that question and whatever preceded it. Find the screen and the weighting targets. Look up the same organization's last three results in your state and ask which direction they ran. Then, last, read the top line, and ask whether anything you just learned would change what you do on Monday.