The big story of the night–What’s are the pollsters doing wrong?
Published polls absolutely missed the dynamics of the governor’s race and failed miserably to predict the final 17% spread between Abercrombie and Aiona. In the end, it wasn’t even close, although published polls in the final weeks showed the race close or tied.
Here are the recent poll results as listed out by TMP PollTracker.
![[text]](http://ilind.net/images_2010/pollsoct.jpg)
Similar data were reported by the New York Times.
So what’s wrong with the methodology being used? Is it just that the Abercrombie campaign was different and somehow eluding a methodology that works in predicting more typical campaigns?
You could dismiss the polling failure as just a one-time quirk, except that its a direct repeat of the failure to predict the Abercrombie’s stunning primary victory margin.
Of course, since pre-election reporting is driven in large part by those poll results, which set the stage for reporting, editorializing, and general punditry, those poll errors led to errors in reporting the public mood and the tenor of the electorate.
There really needs to be some serious reassessment, although the sports metaphor that generally dominates election reporting is will make it easier for media to sidestep the issue.
I flagged a number of races that I thought might be close. I think all of them resulted in Republican challengers being rejected. Luckily, I finally got up to see the final results, and Marilyn Lee slipped past her GOP challenger by a handful of votes in a very tight House raise in Mililani. I thought she was losing when I last looked, which I found depressing, but that must have been the fog of political wars. Congrats to Marilyn.
Overall, though, it was a heavily Democratic night in Hawaii, which again stood apart from mainland election dynamics.
Discover more from i L i n d
Subscribe to get the latest posts sent to your email.
The danger in reading blogs like this one is when everybody starts discussing as though it is factual that an unknown person says an unknown person told them that their “internal polls” were right on the mark……AFTER the election. How convenient….yeah right, they knew they had 1 huge 17 point lead which is why they felt they needed to have The President of the United States call into a local radio show and pitch his candidates…..that looks like a move where they thought they needed the boost.
Duh,
Since I am the person you are most directly insulting, let me respond. I was told of the internal poll numbers prior to the first print out.
But if you prefer to keep things simple so you can understand them without the bothersome complexity, just stick with your insults.
Polling is complicated. I imagine the pollsters are scratching their heads and might make some changes. Or maybe not, since polls are widely used not only to gain information but to influence outcomes.
I started hanging up the phone if I heard silence at the beginning, and I refused to answer any polls at all this election. Well, maybe just one.
Could it be that disgusted Democrats did the same (that is, decline to be polled) and disgusted Republicans willingly told pollsters what they were angry about?
That’s just a guess. Who knows. Again, a poll may be run not to learn something but to influence the vote, and even a legitimate poll may suffer from others doing that and polarizing the electorate against the legit polls.
I think you hit the nail on the head Larry. R’s were angry and thus willing to speak their mind. The local Dems however were more defensive and likely to hang up and say they didn’t want to participate. Without doing an exit poll it’s impossible to know for sure, then again that might not have been accurate either.
Interesting that someone would prefer to have someone holding office who has no community service experience albeit a fairly decent education rather than a highly-experienced individual who supports changing the law to move our city into the second decade of the 21st century. But then NIMBYs rule pretty much in every location unless someone steps up to challenge them.
Demonizing people who are solid citizens with strong records of community service because they are barred from providing services by laws passed more than two decades ago – now that’s the crime. What do you want to do? Throw these property owners into prison – or seize their property – or force them to toss it into the foreclosure bucket?
And please don’t argue that these high-end properties can be rented to homeless people – or to employed stable people who clear $40,000 after required deductions.
Forgive me, CWD, but your comment is in response to what?
(Maybe my brain’s not functioning properly after last night’s elation).
Since when is it “demonizing” to tell the truth about someone’s illegal activities? Even a little illegal activity is still illegal which makes Richard Turbin a criminal. Turbin violates his own Community Association’s covenants against carrying on a business in a residential area. And when Turbin passed the Hawai‘i bar, he swore to support the laws of the State of Hawai‘i but considers his own wrongdoing like “having too much trash on your lawn.” Since you’re clueless, cwd, illegal vacation units increase homelessness by decreasing the available rental market for local people, not the homeless. Finally, if you don’t live next to an illegal vacation rental, you have no grounds to speak. I’ve lived near both a drug house and an illegal vacation rental and the problems they cause are the same: noise, traffic, parking, and too many people for your neighborhood watch to keep track of. I would even state that crime is a problem both produce because there are no more neighbors to daily help keep a community safe.
I got rid of my land line years ago. I received my first robo-call — EVER — this year for a state senate candidate in . . . . . . . . Alabama! Go figure. Oh, and as they closed that call, ” . . . . God bless!”
Or it could be that the internal polls continued right up to the eve of the election while the polls used by the media were conducted far enough ahead of the election for the campaign to have an effect…so there is nothing really wrong with the methodology, just the interpretation (they are snapshots in time).
Robo-polling–and the biases that come with it–is likely to account for many of the errors as well.
Polling is part science, part art. Good polling requires an understanding of the electorate, a healthy appreciation for the statistics, and a clear focus on the goals.
In conjunction with SMS Research as the fielding company, I was the pollster on both the Abercrombie and the Hanabusa campaigns. One of your bloggers invited my participation; here are my comments.
Yes, we were very accurate on the horseraces (in both the primary and the general we were within one percentage point on the Abercrombie races, and right on the mark in the general for Hanabusa – we did not poll the Hanabusa primary). HOWEVER, that’s NOT the value of campaign polling; at least not the way we do it. Our objectives are to track the primary voting blocs (e.g. ethnic, age, political leanings, union affiliation, gender) and to help direct grassroot efforts and media advertising. As we track those blocs, and factor in the context of how they respond historically, we can help re-direct discretionary funds and manpower. Without the polling, the campaign effort is scattered.
Some people use polls to raise money; others to stir the troops, others to impact public opinion; we only poll to help direct the campaign. When we are lucky enough to track the horserace up to election day, it’s a bonus to be right. I would add that the other major private pollsters who do campaign polling in Hawaii (i.e. Barbara Ankersmit of QMark; Pat Loui, until recently of Omnitrak; Jim Dannemiller of SMS) are all excellent. If given the resources and the assignment, they also would have gotten the horserace within sample error. But that’s not the typical assignment; the assignment is to help guide a campaign to victory.
You note in the original posting that the polls were all wrong. I disagree; but one has to appreciate some of their limitations:
1. Timing and Single Point Polling. Polls are taken at one point in time. An example is Becky Ward’s poll for the Star Advertiser. I believe Becky was right on the mark (Becky is an good pollster when given the resources for an adequate sample size). The problem is that they took only one point in time, then compounded the mistake by not reporting the results till two weeks later. By then, the political environment had changed. Becky reported Abercrombie with an 8 point lead. That was nice, because at the time she reported, we had no lead at all. Yes, two weeks earlier we had an 8 point lead, but not at the time of reporting. People thought we were going up, when in fact, we were on our way down. That turned around because of well targeted, incredible effort, but the public didn’t know it.
Merriman, who polled for Civil Beat, also had a problem of timing. They were terrific; they reported right after polling. But they reported a week before the election. They missed the significant momentum that both campaigns enjoyed. If they had taken one or two more points of measure, they would have seen the trend. But they only took one point in time; too early at that. By taking only one point in time, public pollsters also miss the ebb and flow of a campaign. Everyone tries to peak on election day, but that’s not always doable. This campaign season more than most, the competing campaigns moved in opposite directions in the last weeks and it was so clear with tracking polls.
2. Auto Call Biases. Merriman’s and Rasmussen’s polls used an auto call methodology. Given its miniscule labor costs, its capacity for large sample sizes, and its capacity for speed, this methodology is probably the preferred tool of the future. However, they will need to solve some serious issues with their current application before they attain the level of accuracy that we get in direct calling. First they have a significant bias to higher educated, higher income respondents and it is too hard to weight for that bias without incurring serious error. Second, and more important, they cannot account for non-sample error. That is they don’t know anything about the people who refuse to participate. With direct calling, I can call back people who refuse to participate, talk to them to understand their reasons for not participating, and factor that into my analysis. With auto calls, they have no idea who has hung on them and no way to find out. We think Merriman underestimated the Abercrombie lead, at that point in time, by two times.
3. Sample Size. The Star-Advertiser and other similar polls are simply too small to have any real value for overall political analysis, and certainly not for voter segment analysis. In the S-A poll Becky was forced to use anecdotal information and that does not make for good statistical analysis. The media needs to spend more on polling if they are going to do it at all. If not, they have a greater chance for error and misinforming the public. The political reporters at the Star-Advertiser are excellent; they need better polling information. Conversely, the media has to do a better job of educating the public as to what the public is seeing. The public needs to better understand the context and the limitations of polls so that they don’t overemphasize the importance of those particular results. It’s not enough to report these polling results and expect the public to put it in the proper context.
4. Hawaii idiosyncrasies. Mainland pros pooh pooh this, but it is crucial to understand poll results. For example, different ethnic groups vote at different levels of participation; newcomers to the islands react differently to messaging than do long term local families, and so forth. The pollster needs to understand the idiosyncrasies, he needs to observe the changes over time, and he needs to factor these into the analysis. If one is looking for accuracy, I would put up every one of the Hawaii pollsters mentioned earlier over any mainland pollster purely because of local knowledge acquired over time. This is not to say that mainland pollsters might not be better; in many instances they are better. But unless they factor in local idiosyncrasies, they will often miss the mark.
5. Cell phones. Cell phones are an increasing problem for pollsters. We try to neutralize the problem by using lots of RDD methodology or by cross checking voter registration against private cell phone registries or by simply weighting appropriately (but assuming higher error), none of which is totally satisfactory. Your bloggers are right; it’s a growing problem. I just don’t think it was as much of a problem this year.
6. Voter fatigue. I think the biggest problem with polling this year, was voter fatigue coupled with a proliferation of negative advertising. They lead to higher than normal refusal rates and significant levels of indecision. With the increased time between the primary and general elections, and with the ability of negative advertising to move opinion, both of these problems will increase and further complicate future polling.
Please don’t misconstrue my comments to imply that polling lead to victory. Polling is only one part of many, many moving parts that take good management and meshing.
I hope these comments constructively add to your conversation on polling. Polling in Hawaii is quite good. How we use it and how we publicize it and how we explain the phenomena they are measuring needs constant examination.
Wonderful, absolutely wonderful!
Glad to see the comment from Ray. This sort of discussion seldom comes out in the forefront.
If the sample size is too small, and so the poll is of questionable value, I would ask why the pollsters don’t refuse to do it. The public understands little of statistics or polling, I’m pretty sure that most people look only for who is ahead and maybe by how much. This may be what separates polling from science.
And I think Ray has disillusioned anyone from thinking that polls published in the newspapers or elsewhare are for public information. They are for strategic purposes: “Some people use polls to raise money; others to stir the troops, others to impact public opinion; we only poll to help direct the campaign. ”
So we, the pollees, are but pawns in the business of getting corporate-sponsored candidates into corporate-serving power.
I hope readers appreciate Ray’s comments. It’s rare when a pollster talks about his art. It’s even more rare when he praises his competition. Nice work, Ray.
I am not a campaign professional but my gut was telling me that it was so counter-productive for Djou to have national republican attack ads running with those mainland paid for voices at the end of them at the same time there was a Democrat “team” message being presented castigating mainland money and the negativity of the campaigns. Djou’s ads (or Republican ads) seemed to be playing into Hanabusa’s and the “team’s” hands.
I wonder if better polling or local expertise could have been used by Djou to take control of the campaign here. And if one has that, can you tell an outside committee you want them to keep a particular message or ad off the air?
Finally, I appreciate Ray sharing some detailed insight into campaign polling. One of the more interesting and informative reports I have read. And it came in the form of a blog post at Ilind.com. Nice!
I think many of us couldn’t get over the Djou ad in which he complained bitterly about Hanabusa’s attack ads, even as he showed a doctored ugly photo of her with sirens screaming in the background. It could be in the dictionary as a video definition of hypocrisy.
Right on Ray. On the mark. Thanks for your insight.