Gallup was caught out badly, but other national pollsters were off, too. It’s time to look at different methods and new technologies
We all know that Gallup screwed the pooch in the 2012 presidential election. It had Mitt Romney leading through most of October and in its final poll by a point – a 5pt error. Gallup sought to prove to the polling world that it was seriously investigating its 2012 polling errors by issuing a report on Tuesday. In the write-up, Gallup noted that although there was no single cause, a likely faulty voter screen and too few Hispanics were among the problems. This comes as no surprise to others including Mark Blumenthal and myself.
It’s worth the time, though, to point out, as I have and Gallup did on Tuesday, that the Gallup effect was only half the problem.
The average of polls done in the final week, excluding Gallup and Rasmussen, had Obama’s lead over Romney more than 2pt too low. I might be willing to look the other way, except the polling average in 2000 had George W Bush winning and had a margin error of again more than 2pt. The error in margin in 1996 was off by 3pt. The 1980 average saw an error of more than 5pt. The years in between 1980 and 1996 were not much better. In other words, the “high” error in national polling even when taking an average isn’t new; in fact, it seems to be rather consistent over the years.
Worse than the error in the final polls was how the national polls took the consumer for a ride in October 2012 before finally settling in the final week. Anyone remember when Pew Research published a poll after the first debate in 2012 that had Mitt Romney up by 4pt among likely voters? I don’t mean to single out Pew, but because of Pew’s sterling reputation, this poll got an outsized amount of attention even as most of us suspected that it probably didn’t reflect the truth. Other pollsters, too, showed a bounce for Romney that propelled him into the lead after the first debate, though not all to the same extent.
The state polling, meanwhile, did not show an analogous large bounce. It consistently had Obama leading in the states he needed to be leading in. Moreover, it showed Obama holding very similar positions to those he did prior to the first debate in the non-battleground states.
Look at YouGov, for example, which polled before and after the first debate. In Florida, Obama was ahead by 2pt before the debate and 1pt after it. In blue New York, Obama was ahead by 22pt before the first debate and 24pt after. In red Georgia, Romney was up by 7pt before the debate and 8pt afterward. Pollsters like ABC/Washington Post, CNN/ORC,and Public Policy Polling (PPP) all did better on the state level throughout October than they did at the national level.
It’s not the first time the state polling beat the national surveys. Back in 2000, for instance, state poll followers knew that Al Gore had a really good shot at winning. National survey followers, though, were surprised when Gore won the national vote. That’s why smart poll aggregators like Drew Linzer, Nate Silver and Sam Wang barely looked at national polling in 2012 when trying to project the winner. It’s also why the Obama campaign didn’t conduct national surveys.
I asked Gallup about state polling on Tuesday, and why it didn’t try to do individual state polls and/or then sum up, as Silver did, to calculate the national vote. After all, besides polling accuracy, the ball game of presidential elections for pollsters is state elections. Gallup’s response was telling. First, it said that polling 50 individual states via live interviewer to come up with a national estimate would be too time-consuming and cost too much money. That’s fair. Second, Gallup said that it didn’t just poll the swing states because it was interested in knowing what all Americans thought, not just swing-state voters. (I agree and made the same point in an earlier column.)
But for those of us who are interested in knowing who is going to win, Gallup’s answer is not satisfying. Other live pollsters like CNN/ORC, Marist, Quinnipiac and the Washington Post did very good statewide polling in 2012. Gallup hasn’t conducted a statewide general election poll since 2006 and hasn’t done so in a general presidential election since 2004. (Those 2004 polls, by the way, weren’t very good.)
Moreover, the option now exists for pollsters to use other technologies to poll most states, if not all 50. We have interactive voice response (IVR) or robo-polls that are relatively cheap and can survey many people quickly. As long as you properly weight in younger voters, as does PPP and SurveyUSA, these polls work quite well in predicting who is going to win the national election. We also have the somewhat less expensive, randomly selected internet surveys such as Knowledge Networks, and the cheaper volunteer internet polling, which YouGov and Ipsos have implemented successfully. These volunteer surveys hold a lot of promise in the future as more and more people get rid of landlines and have computers.
The point is that there are proven ways to poll that produce more consistently accurate portrayals of the election than doing a single live telephone interviews of a randomly selected population in a national poll. In fact, it’s already being done. That’s not to say that good-quality probabilistic national surveys don’t have a place. No one has proven to me that IVR or non-random internet surveys are as good as probabilistic telephones surveys on issue questions beyond the ballot test. The problem, again, is that I’d look to other sources in preference to a survey that interviews some number of respondents in one survey, a different set of respondents in the next survey, and so on.
One of the biggest takeaways from the American Association for Public Opinion Research (AAPOR) conference is on the usefulness of panel research. That is, you have a set number of respondents, weighted to the correct population parameters, who get interviewed over and over again. This leads to less volatility, and you can actually see how different respondents are reacting to the campaign. Panels can be difficult to do by phone, yet are rather easily obtained in randomly-selected internet samples that pretty much everyone, including those against volunteer internet samples, agrees do just as good a job at finding the true public opinion on issue questions. You can actually see how well panels worked with the Rand American Life Panel. It was the only national tracking survey in 2012 that had both convention bounces and Obama leading throughout the month of October.
None of this is to say that live telephone surveying is bad or useless, by any stretch. Most of the national telephone polls in 2012 were better than Gallup’s. It just seems to me that we shouldn’t only be examining Gallup for 2012′s polling failings. It might be time for even the most ardent defenders of live telephone national interviews to look at other methods in greater depth. Whether it be for the presidential horserace or more in-depth issue questions, different and (in some cases) less expensive survey styles have shown a trend to do better or at least as well.