Margin of Σrror

Margin of Σrror -

Category Archives: US elections 2012

Gallup’s 2012 election polling errors were only part of the problem | Henry Enten

Gallup was caught out badly, but other national pollsters were off, too. It’s time to look at different methods and new technologies

We all know that Gallup screwed the pooch in the 2012 presidential election. It had Mitt Romney leading through most of October and in its final poll by a point – a 5pt error. Gallup sought to prove to the polling world that it was seriously investigating its 2012 polling errors by issuing a report on Tuesday. In the write-up, Gallup noted that although there was no single cause, a likely faulty voter screen and too few Hispanics were among the problems. This comes as no surprise to others including Mark Blumenthal and myself.

It’s worth the time, though, to point out, as I have and Gallup did on Tuesday, that the Gallup effect was only half the problem.

The average of polls done in the final week, excluding Gallup and Rasmussen, had Obama’s lead over Romney more than 2pt too low. I might be willing to look the other way, except the polling average in 2000 had George W Bush winning and had a margin error of again more than 2pt. The error in margin in 1996 was off by 3pt. The 1980 average saw an error of more than 5pt. The years in between 1980 and 1996 were not much better. In other words, the “high” error in national polling even when taking an average isn’t new; in fact, it seems to be rather consistent over the years.

Worse than the error in the final polls was how the national polls took the consumer for a ride in October 2012 before finally settling in the final week. Anyone remember when Pew Research published a poll after the first debate in 2012 that had Mitt Romney up by 4pt among likely voters? I don’t mean to single out Pew, but because of Pew’s sterling reputation, this poll got an outsized amount of attention even as most of us suspected that it probably didn’t reflect the truth. Other pollsters, too, showed a bounce for Romney that propelled him into the lead after the first debate, though not all to the same extent.

The state polling, meanwhile, did not show an analogous large bounce. It consistently had Obama leading in the states he needed to be leading in. Moreover, it showed Obama holding very similar positions to those he did prior to the first debate in the non-battleground states.

Look at YouGov, for example, which polled before and after the first debate. In Florida, Obama was ahead by 2pt before the debate and 1pt after it. In blue New York, Obama was ahead by 22pt before the first debate and 24pt after. In red Georgia, Romney was up by 7pt before the debate and 8pt afterward. Pollsters like ABC/Washington Post, CNN/ORC,and Public Policy Polling (PPP) all did better on the state level throughout October than they did at the national level.

It’s not the first time the state polling beat the national surveys. Back in 2000, for instance, state poll followers knew that Al Gore had a really good shot at winning. National survey followers, though, were surprised when Gore won the national vote. That’s why smart poll aggregators like Drew Linzer, Nate Silver and Sam Wang barely looked at national polling in 2012 when trying to project the winner. It’s also why the Obama campaign didn’t conduct national surveys.

I asked Gallup about state polling on Tuesday, and why it didn’t try to do individual state polls and/or then sum up, as Silver did, to calculate the national vote. After all, besides polling accuracy, the ball game of presidential elections for pollsters is state elections. Gallup’s response was telling. First, it said that polling 50 individual states via live interviewer to come up with a national estimate would be too time-consuming and cost too much money. That’s fair. Second, Gallup said that it didn’t just poll the swing states because it was interested in knowing what all Americans thought, not just swing-state voters. (I agree and made the same point in an earlier column.)

But for those of us who are interested in knowing who is going to win, Gallup’s answer is not satisfying. Other live pollsters like CNN/ORC, Marist, Quinnipiac and the Washington Post did very good statewide polling in 2012. Gallup hasn’t conducted a statewide general election poll since 2006 and hasn’t done so in a general presidential election since 2004. (Those 2004 polls, by the way, weren’t very good.)

Moreover, the option now exists for pollsters to use other technologies to poll most states, if not all 50. We have interactive voice response (IVR) or robo-polls that are relatively cheap and can survey many people quickly. As long as you properly weight in younger voters, as does PPP and SurveyUSA, these polls work quite well in predicting who is going to win the national election. We also have the somewhat less expensive, randomly selected internet surveys such as Knowledge Networks, and the cheaper volunteer internet polling, which YouGov and Ipsos have implemented successfully. These volunteer surveys hold a lot of promise in the future as more and more people get rid of landlines and have computers.

The point is that there are proven ways to poll that produce more consistently accurate portrayals of the election than doing a single live telephone interviews of a randomly selected population in a national poll. In fact, it’s already being done. That’s not to say that good-quality probabilistic national surveys don’t have a place. No one has proven to me that IVR or non-random internet surveys are as good as probabilistic telephones surveys on issue questions beyond the ballot test. The problem, again, is that I’d look to other sources in preference to a survey that interviews some number of respondents in one survey, a different set of respondents in the next survey, and so on.

One of the biggest takeaways from the American Association for Public Opinion Research (AAPOR) conference is on the usefulness of panel research. That is, you have a set number of respondents, weighted to the correct population parameters, who get interviewed over and over again. This leads to less volatility, and you can actually see how different respondents are reacting to the campaign. Panels can be difficult to do by phone, yet are rather easily obtained in randomly-selected internet samples that pretty much everyone, including those against volunteer internet samples, agrees do just as good a job at finding the true public opinion on issue questions. You can actually see how well panels worked with the Rand American Life Panel. It was the only national tracking survey in 2012 that had both convention bounces and Obama leading throughout the month of October.

None of this is to say that live telephone surveying is bad or useless, by any stretch. Most of the national telephone polls in 2012 were better than Gallup’s. It just seems to me that we shouldn’t only be examining Gallup for 2012′s polling failings. It might be time for even the most ardent defenders of live telephone national interviews to look at other methods in greater depth. Whether it be for the presidential horserace or more in-depth issue questions, different and (in some cases) less expensive survey styles have shown a trend to do better or at least as well.

guardian.co.uk © 2013 Guardian News and Media Limited or its affiliated companies. All rights reserved. | Use of this content is subject to our Terms & Conditions | More Feeds

Did Under Voting Cost Mount Vernon Schools the November Levy Election? (Part Two)

In my first post on under voting in Knox County, Ohio, I introduced the concept of under voting and discussed patterns of under voting in races in Knox County involving candidates. I found that the Gambier precincts exhibited levels of under voting that were below the Knox County norm in the presidential race, but that under voting rates in Gambier were much higher than the Knox County norm in other races down ballot.

This piece examines the effect of under voting on an issue race, focusing on the Knox County School Levy election that took place on November 6, 2012.

The Mount Vernon School Levy failed narrowly on November 6th, losing by a margin of 6813 votes in favor (49.3%) to 7014 votes against (50.7%). Had the levy gotten 202 more votes (a tie results in a loss), it would have passed. In the Gambier precincts, 241 votes or ~18.1% of votes cast were under votes. In the non-Gambier precincts, 390 votes or ~3.2% of all votes cast were under votes.

So, getting back to the central question, did the high rate of under voting in the Gambier precincts cost Mount Vernon Schools the November Levy Election? The answer to that question, of course, is complicated. Below, I will examine four alternative scenarios, each of which results in a slightly different answer.

Scenario One- Everyone votes, under voters all vote for the levy: This scenario, while perhaps unrealistic, is the most optimistic for the levy. Had the under voters in Gambier all voted for the levy, the levy would have passed by a margin of 7054 votes to 7014 votes (pending automatic recount). This scenario, however, is probably overly optimistic; unless the school levy could have generated the sort of enthusiasm as Barack Obama, it is at least somewhat unreasonable to expect that there would be no under votes at all in this race. It is also somewhat optimistic for the levy to assume that all under voters would vote for the levy if they had cast ballots.

Scenario Two- Everyone votes, under voters support levy at rate of voters: What if one assumes that everyone votes, but that the under voters support the levy at the same rate as those who already voted? This may be a more reasonable assumption than assuming that every under voter would naturally support the levy. In the Gambier precincts, 91.2% of voters supported the school levy. Had 91.2% of the under voters supported the school levy, the levy would have gotten approximately 220 more yes votes for a total of 7033 yes votes. However, under this assumption, approximately 21 of the under voters (~8.8%) would have voted no, giving the no side a total of 7035 no votes. Under this scenario, the levy would have failed by three (!) votes (a tie results in a loss). Obviously, the levy would have gone to recount under this scenario; the only thing that would be sure under this scenario is a lengthy legal battle.

Scenario Three- Under voting falls to norm outside Gambier, under voters support levy at rate of voters: The assumption that everyone votes is also somewhat optimistic; after all outside of the Gambier (and College Township) precincts there was some under voting in this race. If we reduce under voting in this race to the non-Gambier average of 3.2%, this means that ~43 under votes would still have been cast in this race, thus meaning that 198 fewer under votes would have been cast. By allocating these under votes in the same way as the formula in Scenario Two, 6994 total votes (increase of 181) would have been cast for the levy and 7031 votes would have been cast against the levy. As a result, the levy would have needed 38 more yes votes to pass under this scenario; however, as with the previous scenario, this result falls within the 0.5% margin to trigger an automatic recount in a local, county, or municipal election.

Scenario Four- Relaxing the Assumptions of Scenarios Two and Three: While the assumptions in Scenario One were likely too loose, the assumptions in Scenarios Two and Three may be too rigid. (Goldilocks had a similar problem with temperature and pudding!) In Scenario Two, I used the 91.2% support rate among all voters. However, it is likely that most of the under voters were Kenyon students as opposed to year-round Gambier townspeople (who make up a small portion of the Gambier vote). I also suspect that Kenyon-affiliated people may have supported the levy at a slightly higher rate than the year-round Gambier townspeople (although support must have been widespread in the village among all residents for the levy to get 91.2% of the vote). Therefore, I average Scenario 1 and Scenario 2 and say that 95.6% of under voters would support the levy.

Let me also relax the assumption of under voting- what if under voting in Gambier took place at a rate of 1.6% in the school levy election, half the 3.2% average for non-Gambier precincts? After all, the Gambier precincts showed in the presidential race that their voters are quite adept at filling out ballots when they want to make their voices heard. Is this assumption reasonable? Perhaps.

Under the relaxed assumption about under voting, ~220 under voters would be converted into voters. Using the assumption of 95.6% support for the levy, I find that supporters would gain ~210 votes and opponents would gain ~10 votes. As a result, the levy would have received 7023 votes in favor and 7024 against, failing by only two (!) votes (again, tie=loss). Once again, the election would have been decided by a recount.

So did under voting cost Mount Vernon Schools the November 2012 election? The answer to that question is a definitive “maybe.” That all depends on a.) which of the above scenarios one finds most convincing and b.) what one assumes would have happened in a recount.

The only other conclusion that can draw is that, had a lower rate of under voting taken place, the election administrator’s prayer most certainly would not have been answered. Most likely a lengthy recount process would have taken place that may have dragged on for weeks if not months.

 

 

Immigration reform won’t deliver a Latino voter ‘bonanza’ to Democrats | Harry J Enten

To assume that 11 million undocumented immigrants potentially eligible to vote will change US political arithmetic is erroneous

I have been quite skeptical about the ability of the Republican party to win over Latino voters. As others and I have noted, Latinos don’t vote Democratic just because of immigration policy. They vote Democratic because they are more ideologically “in sync” with the Democratic party. That’s why I’ve felt that going along with immigration reform was unlikely to net Republicans many Latino voters.

That said, I can’t agree either with the math in a Politico article titled “Immigration reform could be a bonanza for Democrats”. The article starts off promisingly enough with the premise that if immigration reform passed and undocumented immigrants became citizens, Latinos would start voting Democratic in even larger numbers. I can go along with this because the main reason anyone votes for or against a political party is for its economic platform, and 81% of first generation Latino immigrants say they want a “bigger government with more services”, compared to only 48% of Americans overall.

Politico then uses the commonly quoted figure of 11 million undocumented immigrants and claims that there would, therefore, be “up to” 11 million undocumented immigrants up for grabs if they all became citizens. If these 11 million then voted along the lines of the Latinos who cast a ballot in 2012, Obama would have won the national vote by 7pt instead of 4pt. He could have carried Arizona and even Texas, which were each won comfortably by Republican Mitt Romney.

The problem I have, though, is why would anyone use the 11 million figure for reference.

First off, 1 million undocumented immigrants are under the age of 18. I don’t care what your immigration status is, you can’t vote in United States presidential elections if you are under the age of 18.

Second, of the 10 million adults, 19% aren’t actually Latino; 11% are Asian. Asian voting patterns tend to be less stable than Latinos. In the past election, Asians went for President Obama by about 45pt. Twenty years ago, they went for Republican George HW Bush.

Without more research (and there is surprisingly little of it), it’s unclear to know how undocumented Asian voting patterns would change given immigration reform. Also, keep in mind that about 60% of these Asian immigrants are in California and Washington State alone – so they’re not exactly going to be a game-changer in the electoral college.

Third, I tend to doubt that all 8 million adult Latino undocumented immigrants would go for citizenship. A Latino Decisions poll says 87% of them would, but I’m skeptical. Only 60% of all legal immigrants actually apply for citizenship. Since the Immigration Reform and Control Act of 1986, only about 40% of the newly eligible immigrants by so-said act became citizens. It would be reasonable to expect a similar percentage this time.

Only 36% of Mexican immigrants who are eligible to become citizens have gone through the process. Mexicans make up 72% of all undocumented Latino immigrants. Six in ten of the other 28% of eligible Latino immigrants have become citizens.

All together, if undocumented Latino immigrants become citizens at the rate that eligible Latino immigrants do, then we’re looking at 3.5 million new voting-age citizens. That 3.5 million is a far cry from the 11 million we first talking about.

Finally, just how many of these 3.5 million undocumented Latino immigrants can be expected to vote in the presidential election? Per the standard census Current Population Survey (CPS), only 49.9% of all voting age Latino American citizens cast a ballot in 2008. (Note, there is no report available for 2012 as of this point.) Based on pre-election surveys and work by Michael McDonald, there is reason to believe that percentage may have dropped further in 2012.

All told, it would seem that only about 1.7 million new Latino voters would be added if undocumented immigrants were granted citizenship. Nationally, this would be a net of about 775,000 votes. This would increase Obama’s vote margin, but not to 7pt; it would only go up to about 4.4pt – in other words, half a point from where it actually was in November 2012. Even adding in new Asian voters, who vote at a lower rate than even Latinos, and other undocumented immigrants (and controlling for the percentage who apply for citizenship, percentage of citizens who vote, and the percentage who voted for Obama), the margin probably only goes up to, at most, 4.6pt.

The amount this would shift individual states in elections is debatable. Take Nevada, where, at last count, there were 190,000 undocumented immigrants – the highest percentage of any state population. Most of them are Latino. Apply the same math we did above, Obama would have gained about 17,000 votes. It would have increased his state margin of victory by 1.4pt. That’s not nothing, but we’re talking about the state with the largest percentage undocumented immigrants.

Most states aren’t close to Nevada’s undocumented immigrant population, while the ones that are simply aren’t competitive at the president level: Arizona, California, and New Jersey. No state in 2012 would have had a different outcome if undocumented immigrants were given the right to vote.

The truth of the matter is that passing immigration reform won’t be a votes “bonanza” for the Democratic party because of potentially or newly enfranchised undocumented immigrants. That doesn’t mean passing immigration reform will help the Republican party among Latinos; the GOP should probably still be worrying about its Latino voter appeal. But it’s not facing a landslide from a new citizen electorate.

guardian.co.uk © 2013 Guardian News and Media Limited or its affiliated companies. All rights reserved. | Use of this content is subject to our Terms & Conditions | More Feeds

It’s not Romney’s fault – the economy helped re-elect Obama | Harry J Enten

Conservatives won’t let go of the notion that Romney wasted an election, even though Obama benefited from economic growth

Sometimes there are beliefs that just won’t die, no matter how false they are. Witness a new article in Commentary by Michael Gerson and Peter Wehner. The authors argue that given the weak economy, Mitt Romney should have won the 2012 election. To be fair, these authors didn’t invent the claim, but they do take it to next the next level.

If you believe Romney lost when he should have won, then he must have made a fatal mistake which needs to be fixed so that Republicans can win again. For Gerson and Wehner, this logic led them down the path of demographics: they conclude that Romney’s biggest problem was a lack of minority outreach in a changing country. I don’t think you can say that Romney had a racial rainbow of support, exactly, but I do want to answer their basic question: did Romney flush away a victory that the economy should have delivered to him?

Actually, Romney always had an uphill battle: the economy narrowly favored President Obama’s re-election. Jamelle Bouie has stated this fact over and over again. John Sides, too, has stated this fact over and over again. Nate Silver has stated this fact over and over again. Have you noticed a pattern here? Smart experts reached the same conclusion. It’s part of the reason we all correctly called a close, but comfortable Obama victory in 2012. It’s why I said way back in September that it wouldn’t be Mitt Romney’s fault if he lost.

What are some people not seeing about the relationship between the economy and elections? For starters, most people don’t vote over absolute numbers, like an unemployment rate of near 8%. Voters choose whether to re-elect based on changes in numbers, like the unemployment rate dropping from 10% in 2009 to 8% in 2012. That’s why voters still overwhelmingly blamed Bush for bad economic conditions and not Obama, even if the unemployment rate remained high in a historical context.

To make the point mathematically (yes, that thing some fear), we can run a regression equation to predict Obama’s chances in 2012 – it’s not so bad: the equation compares growth in different sectors of the economy during prior elections. There were a few faulty models that were either based off of one economic variable or over-fit to match past results. The majority, however, favored Obama. Most of these either utilized just one economic variable or included a presidential approval variable. But in my opinion, one economic variable doesn’t tell the whole story and approval ratings can be influenced to some degree by the economy.

There are two models I know of that employed multiple economic indices and no approval ratings: Bob Erikson and Chris Wlezien’s leading economic indicator model and Nate Silver’s economic index. Erikson and Wlezien’s model simply takes into account continued quarter growth for elections since 1952, and uses the 10 leading economic variables (which are shown to be representative of overall strength in the economy), with quarters closer to the election weighted more heavily. Silver’s model takes into account rolling averages of a month to a year of seven economic variables, which range from personal income growth to forecasted GDP to the stock market for elections since 1968.

The results of these models were remarkably similar: small Obama victories. Erikson and Wlezien, who authored the great book The Timeline of Presidential Elections: How Campaigns Do (and Do Not) Matter, actually overshot Obama’s 3.9pt win by about a point, calling for a 5pt Obama victory. Silver’s model slightly undershot it by having Obama take the election by about 3pt. The average of their results equals a 4pt Obama victory, or for Romney to hit the infamous 47%, which is pretty much where the numbers fell.

The average of the two most thorough economic models nailed the election results. This clearly indicates that given the economy, Obama should have won in 2012. A one or seven point victory would still indicate that 2012 was mostly about the economy – precision is more luck than anything else.

It’s important to remember that these models have margins of errors and elections aren’t only about the economy. Candidate ideology, for example, is generally worth a few points. Romney was generally seen as one of the more mainstream Republican candidates of 2012. Perhaps the even more moderate Jon Huntsman might have lost by only 2-3pt nationally.

Field work may have put Obama over the top in Florida – the closest state in 2012. But even if a better field organization yielded Romney those 29 additional electoral votes, Obama still would have had a major win.

The general point, though, is 2012, like most national elections, was mostly about the economy. A Mitt Romney victory would have actually been a surprise.

So, what about the Gerson and Wehner argument that Republicans will continue lose elections in which they are favored, unless they adjust to a new reality? It comes from a flawed premise of what happened in 2012, and doesn’t have much bearing on how the economy affected the race. It’s a completely different question as to whether or not Republicans will lose elections that the economy hands them. I’ll have more to say about that coming days.

What I’ll say for now is President Obama won the 2012 election because of the state of the economy, not in spite of it.

guardian.co.uk © 2013 Guardian News and Media Limited or its affiliated companies. All rights reserved. | Use of this content is subject to our Terms & Conditions | More Feeds

Dick Morris exits Fox News: even he might have predicted this correctly | Harry J Enten

I’ve nothing personal against Morris, but I believe the public has a right to expect pundits to be accountable for their forecasting

I’m not a big one for schadenfreude but I confess to feeling a little frisson on hearing the news that Dick Morris was not getting his contract renewed by Fox News. It’s not that I wish Morris ill tidings; it’s that I want pundits to be held accountable for their incorrect forecasts.

Dick Morris has one of the worst electoral prediction records known to man.

I wrote last August on 10 Morris predictions that turned out to be not just wrong, but horribly off the mark. There were many more I didn’t even cover.

In the 2012 election, Morris topped himself. He kept insisting that Mitt Romney would win the presidential election. He had the “real polls” (whatever those were). His final prediction was Romney taking 325 electoral votes and the national vote by at least 5pt. Of course, it was Obama emerging victorious with 332 electoral votes and a 4pt national vote edge.

Morris’ performance earned him the title of “Worst Pundit of 2012” by Pundit Tracker. That’s an impressive achievement considering all the competition. In mathematical terms, Morris was wrong 80% in his prognostications: if you had placed a dollar bet on each Morris prediction in a prediction market, you would have lost 70% of your money.

I would be more forgiving of Morris if he had just said he was wrong, provided a real reason for his forecasts, and promised to be better in the future. After all, none of us is perfect or even close to it (see my mea culpa for slips in 2012). Instead, Morris blamed the usual “Hurricane Sandy allowed Obama to win” hypothesis, and then said he knew he had been wrong, yet he’d wanted to raise the spirit of Republicans. This sort of action from a person who is supposed to inform the public simply isn’t acceptable.

Analysts and pundits must be held accountable for their misses. They should not just be able to appear on television or in print the next day and say “oops!”, without consequences. Which is why I was glad that Fox News actually acted on Morris’ embarrassingly poor performance. It suggests that Fox executives do actually care, at least at some level, about the accuracy of the electoral analysis being broadcast.

Now, I just hope Fox doesn’t put Pat Caddell on again. They would be far better served by putting their own director of public research, Dana Blanton, on the air.

The only part of the Dick Morris departure that saddens me is that he has been booked to appear on Piers Morgan’s CNN show Wednesday evening. This can only be a misguided ploy to boost ratings, because if Morris’ appearance has anything to do with expertise, then I can only assume Morgan and his producers know very little about American politics.

In the past, Morgan has actually had a couple of great analysts on the show. Republican pollster Kristen Soltis was on more than a couple of times during the campaign. Soltis, part of the great Winston Group, really knows stuff.

Responding to the Morris news Tuesday night, I tweeted Morgan that he’d better off having Soltis back again. Soltis responded that she’d be “more than than happy to debate Dick Morris about the future of election forecasting!” I hope Morgan takes Soltis up on her offer: that would be worth watching.

But at least, Dick Morris being off the air at Fox is a win for anyone who cares about the quality of information being fed to the viewer.

guardian.co.uk © 2013 Guardian News and Media Limited or its affiliated companies. All rights reserved. | Use of this content is subject to our Terms & Conditions | More Feeds

The weighting game: why Rasmussen has Democrats winning 2014 midterms | Harry J Enten

To mend its 2012 reputation for leaning Republican, Rasmussen tweaked its methodology. Now the pollster has overcompensated

Throughout the 2012 election season, it became a commonplace that Rasmussen Reports had a consistent pro-GOP bias. Well, get ready to be surprised: Rasmussen now leans more Democratic than the average pollster.

This from the pollster that projected Mitt Romney to win the popular vote and did not call an Obama victory in six of the states he actually carried. Clearly, Rasmussen needed to change, and it did. But rather than solve its problems, it has ended up overcompensating by mistakenly weighting its surveys to the national exit poll demographics.

What do I mean? Let’s take a look at some of Rasmussen’s new demographic weighting.

Prior to the 2012 election, whites made up a little over 73% of the electorate in Rasmussen world. Now, non-Hispanic whites make up 72% of Rasmussen’s sample; blacks make up 13% and Latinos are at 10% – just like the exits.

So much for demographics. The biggest noticeable change, however, is Rasmussen’s party identification.

Rasmussen is one of the few pollsters to actively weight by party identification. It interviewed thousands upon thousands of voters, and then utilized a “dynamic weighting” scheme to get the correct percentage of Democrats, Republicans, and independents in the lead-up to the 2012 election. It ended up with a three-month rolling average of 4.2% more self-identified Republicans than Democrats.

The exit polls, though, had Democrats with a 6pt party identification advantage. Sure enough, Rasmussen now weights its polling to 38% Democratic and 32% Republican – the same exact spread as the exit polls gave.

Here are my qualms about Rasmussen’s methodology. First, I dislike weighting polls by party identification. Party ID is an attitude that isn’t consistent from election to election. Given Rasmussen’s major problems in correctly tracking party identification, I have zero faith it will be able to track any deviation from the 2012 exit polls.

Second, I’ve never heard of any reliable pollster weighting their polls to match the exits. The exit pollsters certainly do not. Exit polls are great surveys with error, just like any other poll.

Any pollster using random digit dialing, like Rasmussen, should start off with the census as a baseline for weighting. You should call cellphones or substitute with an internet sub-sample; Rasmussen tried the latter and seemingly failed. If you wish to see a smaller subsection of the population, like registered voters or likely voters, you ask a short battery of simple questions to determine who is a registered or a likely voter.

Third, exit polls often differ from most pre-election polls on demographics. As I discussed earlier this cycle, exit polls tend to have more minorities and young voters than other surveys. Democracy Corps, 2012′s most successful national telephone pollster, is a case in point.

Corps had non-Hispanic whites at 74% of the electorate – 2pt higher than the exits, which matches the average of national pre-election pollsters. Democracy Corps had 18-29 year-olds at 16% of the electorate (3pt lower than the exit polls), and people 65 and older at 18% (2pt higher than the exit polls).

And what about the infamous party identification? Democracy Corps had Democrats with only a 3pt edge over Republicans, 35% to 32%, respectively. If Democracy Corps had re-weighted its final poll using the 2012 exit polls, it would have found a result too favorable to President Obama.

Fourth, Rasmussen is weighting its likely voter model to 2012. In presidential years, most registered voters are likely voters. In midterm years, as 2014 will be, however, turnout is down and the electorate tends to skew towards voters who are, on average, older and whiter than voters in the registered electorate. That makes midterm electorates more favorable to the Republican party.

Democracy Corps recently tested the likely voter electorate for 2012 and 2014 (pdf), and discovered what you’d expect: the 2014 electorate is, at this point, projected to be 2pt whiter and have 2pt fewer 18-29 year-olds, proportionally, than turned out in 2012.

So, it’s no surprise that Rasmussen’s choice to weight its results by the 2012 exit polls creates a Democratic “House effect”.

Obama has a House effect, too. Among pollsters who surveyed registered or likely voters (besides Rasmussen), his controlled average net approval since his re-election is +6.8pt. (I only averaged registered and likely voters here because the general electorate has fewer minorities than the adult population at large, which makes it more conservative.)

Obama’s average net approval in Rasmussen’s polling since re-election is +10.6pt, which is nearly 4pt higher than the other pollsters’ results. ABC/Washington Post and Quinnipiac are the only pollsters who had it higher among registered or likely voters. That’s quite the turnaround from before the election, when Rasmussen rarely gave Obama a positive net approval, and was more Republican-leaning than actual results in 81% of the races it polled.

This Democratic House effect can likewise be seen on the national House ballot for 2014. Since the election, Rasmussen’s 11-week average has Democrats with a +7.4pt advantage. Even with biases against Democrats winning seats, a 7.4pt margin would almost certainly get them back the House – and with room to spare. For 96 weeks before the election, Rasmussen only found the Democrats ahead 5% of the time in the on the national House ballot, despite the fact that Democrats won it by a point.

There have only been two other pollsters since the election to survey likely voters on the House ballot: Democracy Corps and GWU Battleground. Democracy Corps pegged the Democratic lead at 4pt, while GWU Battleground put it at 3pt. Again, Rasmussen is nearly 4pt more Democratic than the average of non-Rasmussen results.

The Democracy Corps result demonstrates the problem with polling to the 2012 electorate. Democracy Corps has the Democratic lead on the House ballot at 7pt, which matches Rasmussen – under 2012 demographics. Under 2014′s likely demographics, the lead shrinks to 4pt.

All in all, I’m not really sure what Rasmussen is up to now. Its methodology has always been somewhat odd, but its response to a tremendous Republican bias in 2010 and 2012 is even stranger. It shouldn’t be weighting to the national exit poll, since this bad decision only shifts to a Democratic House effect. For its own sake, I can only hope that Rasmussen modifies its methodology again – and this time, improves it.

guardian.co.uk © 2013 Guardian News and Media Limited or its affiliated companies. All rights reserved. | Use of this content is subject to our Terms & Conditions | More Feeds

The top five polling lessons from the 2012 election | Harry J Enten

Cellphones, internet tactics and other vital things we should take with us as we close the books on last month’s election

The 2012 election is in the books, and as we head into the new term filled with new elections, there are polling lessons we should take with us. Yet, as this list of the top five polling rules from 2012 will demonstrate, there will always be exceptions to the rule.

1. When likely and registered voter polls disagree in high turnout elections, you should usually go with the registered voter surveys

Likely voter (LV) models at their heart are trying to separate the LV electorate from the larger registered voter (RV) unit. That just creates unnecessary noise in a presidential year because, as I’ve noted, almost all RVs cast a ballot.

This year LV samples were more favorable to Mitt Romney throughout the entire campaign. Among pollsters who conducted a poll among both LVs and RVs in the final week, Obama was ahead in the final LV aggregate by only 1.3 points. The RV average had him by 3.9 points and was thus far more accurate given Obama is leading by 3.7 currently. This follows a trend from past presidential years when RV polls were more accurate than their LV counterparts.

Exceptions to the rule:

Pew Research’s final LV margin of three points for Obama was more accurate than their RV margin of seven points. That’s noteworthy because Pew Research has one of the tighter LV screens in the polling business. Pew’s LV success marks three consecutive presidential elections where their LV model was more accurate their RV model.

The RAND poll LV model, which allowed voters to assign themselves a percentage likelihood of voting, had Obama leading by 3.3 points and was only 0.4 points off the mark. The RV RAND model had Obama ahead by 6.3 points – a 2.6-point error.

2. Cellphones are generally needed for an accurate telephone poll

If you’re trying to poll the overall population, and part of the population no longer has a landline, then you need to call cellphones. This “fact” has become especially true as the cellphone-only population explodes, and those with cellphones have different political ideologies even when controlling for demographics.

During the course of the 2012, polls without cellphones consistently painted a false pretty picture for the Romney campaign. Polls without cellphones in the final 21 days were more than a point more inaccurate than those that included cellphones. This includes some of the worst pollsters of 2012 such as Gravis and the infamous Rasmussen.

Exception to the rule:

Public Policy Polling was arguably the most accurate large-scale pollster of 2012 and didn’t compensate for not calling cells by utilizing an internet subsample. Their secret? They simply were able to correctly model the election day electorate, which is what polling is all about.

3. Internet polling is the wave of the future

Once upon a time, internet polling, and specifically non-probability internet polling, was shunned. Now with cellphones popping up everywhere and poll response rates plummeting, internet polling has gotten a second look because it’s cheap and accurate. Almost all age groups now use the internet and weighting can compensate for people who don’t.

Google, Ipsos, RAND, and YouGov performed at least as well as telephone polls, if not better. RAND was by far the most accurate national pollster of 2012. RAND and YouGov also tended to be stabler than telephone surveys in their results after the first debate.

Exception to the rule:

Zogby (JZ Analytics) is awful at internet polling. Their poor performance in 2004 delayed the acceptance of internet polling in the mainstream by at least five years, and they were again in 2012 one of the least accurate pollsters.

4. Internal polls published publicly generally should not be trusted

This one is fairly simple. Internal, or campaign, pollsters usually only release surveys to the public that are most favorable to their candidate. They may also only pass polls to the press that are based on turnout models more favorable than probable for their candidates.

In 2012, we saw that Mitt Romney’s internal polls were hilariously bad. We also witnessed that polls conducted for Republican groups such as Citizens United in the swing states that were way off the mark. During the Wisconsin gubernatorial recount, Democratic internal surveys were on average six to seven points more favorable to Democrat Tom Barrett than public polls.

Exception to the rule:

Any poll conducted by Democratic pollster Mark Mellman. He was most accurate state pollster of 2012. Mellman is the only pollster I’d trust in the state of Nevada where Latinos are generally undercounted by pollsters. He also did well in North Dakota when he argued early on that Democratic Senate candidate Heidi Heitkamp was competitive, even though most of us thought she had little chance.

5. When state and national polls disagree, you should generally go with the state data

The state polling always had President Obama winning a second term and state polling averages were correct in 49 or 50 states. The national polling had Romney much closer and even pulling into the lead after the first debate. It was only in the final week that it became clear judging off the national polls that Obama was going to win the nationwide vote. Even then, 90% of the polls underestimated Obama’s lead.

This was not the first time that national and state polls disagreed. In 1996 and 2000, national polling incorrectly had Bill Clinton winning by double digits and George W Bush winning respectively. This year also continued a streak wherein no presidential candidate won a state where they were not leading in at least one non-partisan statewide poll in the final weeks.

Exceptions to the rule:

The state polling was atrocious in a number of states. Twenty-eight of 29 polls , and 33 of the 38 polls in Virginia during the final month were biased against Obama. Fortunately for pollsters, Obama was leading by a comfortable enough margin in these states that this bias did not effect the projected winner in the polling aggregates.

guardian.co.uk © 2013 Guardian News and Media Limited or its affiliated companies. All rights reserved. | Use of this content is subject to our Terms & Conditions | More Feeds