by Tony Petrangelo

Mar 14, 2013, 11:00 AM

Mason-Dixon: Y U NO POLL GOOD

With The Minnesota Poll back in the game and with Mason-Dixon retained as the polling firm responsible for it, I thought it would be worth it to finish a post that I started in the aftermath of the election, but never actually got around to completing.

The post concerns the really bad, terrible, and downright awful election cycle that Mason-Dixon just had. When I compiled the performances of the half dozen pollsters who had polled the Presidential race in Minnesota, Mason-Dixon was by far the worst. And using the qualifier “by far” sells short just how bad they really were.

Nate Silver has done this for all of the pollsters as it concerns the Presidential results, not just those polling in Minnesota, so you should probably go read his post if you haven’t yet. The most salient point from that article, as it concerns this post, is the following:

Several polling firms got notably poor results, on the other hand. For the second consecutive election — the same was true in 2010 — Rasmussen Reports polls had a statistical bias toward Republicans, overestimating Mr. Romney’s performance by about four percentage points, on average. Polls by American Research Group and Mason-Dixon also largely missed the mark. Mason-Dixon might be given a pass since it has a decent track record over the longer term, while American Research Group has long been unreliable.

The StarTribune switched from their previous pollster, Princeton Survey Research, presumably because they missed the mark in the Dayton-Emmer race by a decent amount. PSRA’s final poll of the 2010 Governor’s race had Dayton 41, Emmer 34, Horner 13. The final results were Dayton 44, Emmer 43, Horner 12. (In a bit of irony, PSRA fared much better in Silver’s rankings this year then Mason-Dixon did).

Unfortunately for the StarTribune, the pollster they switched to had one of the worst election cycles of anyone who wasn’t a Republican internal pollster. Not only were their results bad, but they even managed to screwed up the weighting in one of their polls and had to re-weight it and re-release it. An almost unheard of thing to do.

Steve Singiser, Daily Kos Elections poll gatherer extraordinaire, did a top five worst polls of 2012 and guess who took the blue ribbon.

But their worst poll of the cycle was also way off: by a full eight points. Or thirteen points. It depends.

Confused? So was Mason Dixon, apparently.

The firm made a huge splash in the final week of the campaign when they released a media-sponsored poll in Utah showing veteran Democratic Rep. Jim Matheson losing by double digits to Republican upstart Mia Love. With that poll release (and a subsequent release by local pollsters Dan Jones and Associates showing Love up by a more modest 5 points), House pundits began to shovel dirt on the prospects of the sole Democratic member of the delegation.

But then, in the final hours of the campaign, Mason Dixon offered differing numbers on the race. It was an almost unprecedented step: a revision of a previously released poll. Apparently, M-D had weighted the poll using a statewide assumption of party ID, when the 4th district (centered largely in Salt Lake County) was quite a bit more Democratic than the state at-large. Thus, they recalibrated, and found Mia Love up by seven points (50-43) instead. She wound up losing by a single point, meaning that even in their final, last-ditch attempt at “unskewing” themselves, M-D still fell way off.

Mason-Dixon didn’t fare very well in Minnesota either.

Clearly the marquee race was the Presidential election between Barack Obama and Mitt Romney. And this is the race in which Mason-Dixon got it the most wrong. Back in December I put together a post that discussed the relative performance of pollsters in Minnesota.

Here is a reproduction of a table I created for that post:

The “Total” column is the total error of all polls done by the firm (represented in the “n” column). “Last 4” and “Last 2” refer to polls conducted in the last four and last two weeks of the election respectively.

As you can see Mason-Dixon’s results were terrible.

Now this chart only covers their performance in the Presidential race, but even when you look at all of the races in Minnesota combined Mason-Dixon was by far the worst. In the list that Nate Silver compiled of Pollsters nationwide, Mason-Dixon was the second worse, only fairing better then Gallup who had an historically bad cycle.

As Silver points out, Mason-Dixon has a good reputation and has been accurate in the past, so it’s hard to fault the StarTribune for hiring them in the first place. On the same note though, it’s hard to see why they’re sticking with them now, considering just how bad their results were compared to the other pollsters who polled not just Minnesota, but anywhere. Not to mention their laughably shoddy work in Utah’s forth congressional district.