Polls and Headline Writing

. . . or how to lie with headlines.

I get very annoyed with the reporting of polls. One way to create news is to incorrectly headline or even incorrectly describe polling data.

For example, CNN uses the headline Poll: Romney & Gingrich Tied for Top Spot in reporting on the latest USA Today/Gallup poll regarding the Republican presidential race. In the text they explain that Romney and Gingrich are at 20% and 19% respectively, that this is well within the margin of error of the poll (+/- four percentage points), and is thus essentially a tie. The number part of this is essentially correct.

Then they say that Cain is following close behind, but they don’t point out at this point that Cain’s 16% is also within the sampling error of both of the leading candidates, or rather, that the probable range of Cain’s percentages largely overlaps those of the leading two candidates.

The margin of error provides a range within which the real percentage of the whole population is likely to fall. If you go to the Gallup site for this poll you’ll find that the confidence level is 95%, in other words, there is a 95% probability that each candidates percentage of the real population falls within +/- 4 percentage points of the poll’s result. Thus there is a 95% change that Cain’s percentage is between 12% & 20%, that Gingrich’s is between 15% & 23%, and that Romney’s is between 16% & 24%.

If you’re wondering why the polls seem to swing quite a lot among the leaders, this would be your explanation. If in a future poll, the number varies by less than four percentage points, that number would not necessarily reflect any change in that particular candidate’s support.

Essentially, the news writers can produce the story they want. It’s possible (though with multiple polls showing him dropping, it’s not likely) that Cain could still be leading this.

Now this particular headline may seem minor. But if you examine the headlines after just about any poll you’ll find that different news services spin the results differently, and that by reading the headlines and the first few paragraphs, you’ll get a somewhat different picture than you would if you read to the end of the story, or even better, go to the source of the poll.

In this story, while we are told that the difference between Romney and Gingrich falls “well within the poll’s sampling error” in the second paragraph, we don’t find the actual margin of error (+/- four percentage points) until the very last sentence. At that point, if we look back, we can see that Cain is also within that margin of error, or rather that the intervals of all three top candidates overlap considerably.

(For a write-up on this, see the Wikipedia article Margin of Error.)