Message Testing Without Fooling Yourself

A campaign writes five messages, puts them in front of 400 people, and one of them wins by 14 points. The room now believes something, and the belief may be entirely a product of how the test was built. Order, wording, the strength of the counterargument and the number of respondents can each manufacture a 14 point winner out of five messages that are equally good.

What a message test measures

A message test measures movement in a stated answer after exposure to words, under conditions somebody constructed. What a campaign wants to know is whether those words will move a real voter in an environment where the opponent is also talking, where the voter is half paying attention, and where the message arrives eleven times over six weeks instead of once in a quiet room. The distance between those two things is the design problem, and most of the ways to close it involve taking the test's advantages away from the message.

Leading language

The fastest way to get the answer you want is to write the question that produces it, and it is usually done without intent.

One proposition, three wordings

Share agreeing with the same underlying proposition when it is asked three ways, on comparable samples. The neutral wording is marked.

Three wordings of one proposition, 19 points apart. The mechanisms are well understood. Naming a sponsor attaches the respondent's view of the sponsor to the item. A preamble that states a consequence makes agreement the cooperative answer. Agree or disagree formats collect acquiescence, which is why every item in an agree-disagree battery looks more popular than it is. And words with a settled valence in politics do the work of the argument before the argument arrives.

The discipline is to ask the proposition in the words a stranger would use, then test the loaded phrasing separately and label it, if what you want to know is what the loading is worth. That is a legitimate test and a useful one. It stops being legitimate when the loaded version is the only version and its score gets reported as the message's strength.

Read your own instrument as though the opposition wrote it. Any phrase you would object to in their questionnaire is producing a number you cannot use.

Order effects

Position in a battery changes a message's score, reliably and by a lot.

The same message read in five positions

Score for one identical message when it appears first through fifth in a battery of five.

Several things cause it. The first item is read against nothing while later items are read against whatever came before, which flatters the first and penalizes the rest. Respondents tire, and the later items get less attention. And a respondent who has already endorsed three things starts differentiating, because endorsing everything begins to feel like not answering.

The fix is randomization, or systematic rotation with the position recorded for every respondent so the effect can be reported on its own. If the instrument did not randomize, the advantage cannot be subtracted afterward and the first message in the battery is not comparable to the fifth. A vendor who presents a ranked battery without saying whether order was randomized has not been asked that question often enough.

Testing against the opponent's best argument

The most expensive corruption of a message test happens in the counterargument rather than the wording.

A message tested against a weak objection has not been tested. The opposition will not run the weak version. They will run the version their best consultant can write, with money behind it, repeatedly, in the places where it works best.

The same message against three counterarguments

Net margin for one message after the respondent hears nothing, a weak objection, and the strongest objection the opposition actually makes.

Only the third column describes the environment the message will actually run in. Building that column takes work that is unglamorous and cheap: read the opponent's own materials, their last three races, their best performing ad, and the attack lines their allies are already using. Write their case the way they would write it, then have somebody who disagrees with you check that you have not softened it.

Then test the exchange rather than the statement. Your message, their response, your rebuttal, with a measurement after each step. A message that wins on first exposure and collapses on rebuttal is worse than one that starts smaller and holds, because the first one spends money teaching voters to hear the other side's answer.

Sample size and what a small test can still tell you

Before deciding what a test should cost, look at what precision costs.

What another 1,000 respondents buys

Width of the 95 percent sampling interval as the number of respondents grows, computed from the standard formula.

The arithmetic is unkind in a specific way. Precision improves with the square root of the sample, so the second thousand respondents buys about a third of what the first thousand bought. A 400 person test is therefore not a cheap version of a 1,600 person test. It is a different instrument with a different set of answerable questions.

What a 400 person test can do: kill the one message clearly out of line with the rest, find wording that confuses people, catch a message that offends a group you need, separate messages that are 15 points apart, and collect open ends that tell you what people actually heard. The open ends are frequently the most valuable thing in a small test, and they are almost never in the summary.

What it cannot do: separate messages four points apart, read subgroups, allocate a budget, or support a ranking of five items. If the plan requires a ranking of five, the test has to be built for that, which means more respondents, randomized order, and a decision made in advance about how large a gap will count as a difference.

Reading a null result

A null result is information and it usually gets read as failure. Three different situations produce one: the messages really do have similar effects, the test lacked the precision to tell them apart, or the outcome measure was insensitive to anything at all.

Those are distinguishable. Look at the interval width first. If it is wider than the differences you cared about, the test was underpowered and the messages are untested rather than equivalent. Then check whether anything in the battery moved, including the items you expected to move. If nothing did, suspect the instrument or the outcome question. Then read the open ends, where a message that scored flat because it was consistently misunderstood looks very different from one that was understood and rejected.

A genuine null on a message the campaign was about to spend six figures behind is among the cheapest things it will buy all year.

Why retesting beats a single read

The winner of a single test is the message with the largest true effect plus the largest favorable error, and the second part does not repeat. That is why tested messages so often underperform once they are in the field. Part of what got measured was noise, and the noise does not come along.

Retesting is the correction and it is cheap next to the buy. Take the top two and the bottom one, put them in front of new respondents in a new order, in different words carrying the same proposition, against the same strong counterargument. A ranking that survives that is probably real. A ranking that reshuffles has told you the first result was a coin landing.

Then retest across time. A message that worked in August may be answering a question voters have stopped asking by October, and the environment does not send notice. Two modest reads six weeks apart are worth more to a campaign than one large read in the summer.