Questions to Ask Before You Hire a Research Firm

The research decision usually gets made on one call, from a deck, in the week after a bad argument about strategy. Six areas of questioning turn that call into something closer to diligence. They work on any firm, including this one, and a candidate who asks all of them and then signs with a competitor has still done the right thing by the campaign.

How the sample is built

Start here, because everything downstream inherits it. Ask who the firm contacts, how it reaches them, what list or panel those people come from, who supplies it, and who is systematically absent from it. Then ask for the step-through rates.

What survives at each step of recruitment

Step-through rates for an illustrative fielding, with the number of people left at each step.

That is an ordinary fielding. The people in the final sample are roughly one in 85 of the people contacted, and nothing in the result tells you how the other 84 would have answered. That gap is the central problem in opinion research and no amount of weighting closes it. Weighting makes a sample match the population on the variables you weighted on. It does nothing about the variables you did not.

Self-selected samples are not disqualifying. Most political research runs on them, because probability samples at the price and speed a campaign needs largely do not exist. What matters is whether the firm says so in writing, reports uncertainty in a way that reflects it, and declines to describe its own work with the words the industry reserves for probability samples.

One more question belongs here: what does the firm do when a group comes in badly under-represented? The answers run from going back into the field for more interviews, which costs money, to weighting a handful of respondents up by a factor of nine, which costs nothing and quietly moves the result. You are entitled to know which one happens on your engagement, before it happens.

What the deliverable contains

Ask to see a redacted deliverable from a comparable race before you sign anything. Then ask what comes with it.

Where the hours go in one engagement

Share of total hours on an illustrative engagement. The document the client reads is a minority of the work.

Most of the hours sit upstream of the document, which is why the deliverable question and the sample question are the same question asked twice. What belongs in the scope: the full instrument as fielded, toplines weighted and unweighted, crosstabs with counts, the verbatim open ends, the data file, and a written statement of limits. A briefing is worth having and is not a substitute for any of that.

Ask what the turnaround is in business days from the close of fieldwork to the document in your hand. Ask who writes it. If the person who designed the instrument is not the person who writes the analysis, ask how those two talk to each other.

Who sees the instrument before it goes out

The questionnaire decides what the research can possibly find, and it is the part clients most often never look at. Ask whether you get to read the instrument before it fields, how long you have, and who has the final word on wording when the campaign and the firm disagree.

Then ask the three practical questions around it. Can you add questions, and at what point does adding one stop being free? Is the instrument piloted on a small group before the full fielding, and what happens if the pilot shows a question is being misread? And if a question has to change after the field opens, does the firm restart, or report two versions, or quietly pool them?

Pooling two versions of a question and reporting one number is common and it is the kind of thing that never appears in a summary. A firm that volunteers its own rule on this before you ask is telling you something useful about the rest of the engagement.

Who owns the data afterward

This gets settled in the contract and almost never discussed on the call. Four separate things can be owned: the response data, the weights, any scores or coefficients built out of it, and the right to analyze all of it after the engagement ends.

Ask whether you receive the response-level file. Ask whether the firm keeps a copy and what it may do with it, including whether your results can be pooled into anything sold to anybody else, aggregated or not. Ask who inside the firm can see your numbers, and whether the firm works for anyone else in your primary. Ask what is deleted at the end, and when.

A firm will have reasonable commercial positions on some of this. What you are looking for is a clear answer, in the contract, before the first interview.

What the firm will refuse to claim

Ask it plainly: what will you not say on the basis of this work? A firm with a real answer names things quickly, and a firm that cannot name anything it refuses to claim will claim whatever the client wants, which is a service available more cheaply elsewhere.

• Causation from a single cross-sectional read.
• Subgroup results below a stated respondent count.
• A forecast of who will turn out, presented as a measurement.
• Vote share to the decimal.
• That a message will work, as distinct from the finding that it moved a stated answer under test conditions.

How limits get disclosed

Every piece of research has limits. The question is whether they survive the trip from the people who did the work to the people who act on it.

How many stated limits survive the retelling

Count of limits written down at each stage, as one illustrative finding moves from the report to a social post.

Ask where the limits live inside the document. An appendix is a place to put things nobody will read. Ask who writes the version that goes to the general consultant, and the version that goes to the press, and whether the firm will put its name on a public release of its own numbers that it did not write.

The industry reference point here is the set of disclosure elements AAPOR's Transparency Initiative asks member organizations to publish when findings are released. Ask the firm which of those elements it publishes without being asked.

What happens when a finding is inconvenient

This is the question that separates firms, and it works best asked as a specific.

Ask for an occasion when the firm told a client something that cost the firm money. Ask what happens when the candidate's favorite message tests worst in the battery. Ask who has the authority to put an unwelcome number in the document, and whether that person reports to the person selling you the engagement.

The answers are hard to fake, because the honest ones are specific and slightly uncomfortable while the evasions are smooth. A firm that has never had this argument has either never had a client with a bad idea or has always lost it.

Two questions to ask yourself

Before the call, write down what decision this research is meant to change, and what result would make you do the opposite of your current plan. Without an answer to the first you are buying reassurance. Without an answer to the second you will find a way to read any result as support for what you had already decided, and the research will have been an expense rather than an input.