Questions about published polls, the election outcome, assessing the Democrats dilemma, etc

A couple of people have taken me to task, in light of the election results, for my earlier take on the Hanabusa-Case decision. Here’s an example:

So, Ian. Just want to know if you are embarrassed that you bought into that terrible Advertiser poll and switched your vote from Hanabusa to Case. I really didn’t care which of them won, but I hate that the media creates news by using polls, especially since they are so, so bad at doing them. And you bought it, hook line and sinker. You’re too good for that.

First, I should clarify that I didn’t “buy into” the Advertiser poll. I tried to survey the information available, which included that poll and others that were publicly cited, along with indirect evidence of polls unseen that led to outside groups to target Case from left and right.

When a conservative mainland group chose to target their negative media exclusively at Case and ignore Hanabusa, that certainly appeared to indicate that their best data found Case to be the one in a position to challenge their favorite. And when AFSCME targeted independent expenditures at Hanabusa’s opponents, they targeted both Case and Djou. Had they been confident, based on polling data available to them, that Hanabusa was going to be the clear Democratic leader, they wouldn’t have split their attention. They would have put everything into blasting Djou.

One problem at this point is that the Office of Elections hasn’t made their detailed data file with precinct-by-precinct results available yet, so I haven’t been able to take a good look at the numbers in order to test different explanations for the outcome.

So, no, I’m not embarrassed at all at my pragmatic suggestion that an Ed Case would be preferable to Charles Djou. And I am impressed by the amount of money that flowed into Hanabusa’s campaign in the last several weeks.

But the same reader had an earlier comment criticizing the Advertiser poll that I do strongly agree with. This has to do with the standards for responsible media polling.

It is absolutely WRONG for the Advertiser or other periodicals to be publishing polls so close to the election period. Why? Because people react exactly as you are reacting. This isn’t reporting news. This is creating news. People are altering their votes because of a survey taken by the Advertiser.

And the Advertiser is NOT giving you sufficient details to examine the accuracy of the poll. For example: What specific question was asked? What was the sample made up of? Where did the phone numbers come from (registered voters haven’t had to provide their phone numbers since the 90’s; how much systematic bias is there in the sample used — also did they use cell phone sampling — how did they then screen for registered voters, etc. etc., etc)? Was there any weighting done before reporting the data (must have been given the disproportionate sampling)?

Any of these questions would be answered in private polling. Yet the Advertiser presents a public poll without being forthright with any of it, making it almost impossible to judge the credibility of these results. And they get away with it because the average reader assumes the newspaper is an unbiased purveyor of the news.

The Honolulu Community-Media Council (now known as Media Council Hawaii) took up this same question in early 1974 by pulling together a group to study and make recommendations.

The meeting pulled in two academics, Earl Babbie and Dan Tuttle, to provide specific expertise on opinion polls. Babbie had created a polling manual for the media to use during the 1970 election. Tuttle presented a simpler 11-point guide to polling that the council eventually adopted at its February 26 (1974) meeting. The required information that Tuttle recommended newspapers disclose when publishing public opinion surveys included “sponsors, wording, definition of area, representativeness, size of sample, make-up of sample, how were contacts made, interview or secret ballot, dates of survey, margin of error and estimate of refusal rate.

[From MEDIATOR AND ADVOCATE: THE HISTORY OF THE HONOLULU COMMUNITY-MEDIA COUNCIL, a dissertation done at the University of Hawaii by Ralph Thomas Kam, May 2005.]

That was before my time as a member of the Media Council, but I recall the issue coming up again in later years, probably in the mid-to-late 1980s, and the Media Council again held public surveys up to the same disclosure standards. Apparently its something that needs to be updated and used again to hold the news media to a higher standard.

13 responses to “Questions about published polls, the election outcome, assessing the Democrats dilemma, etc

  1. Although not published in the article that provided the poll results, the HA did provide some of the information in question [e.g. questions that were asked and demographic info such as party leaning (quasi-affiliation), member of a union living in respondent’s household, age, etc].

    The problem I have with these polls (the HA poll in particular) is that the margin of error in some instances was ridiculously high. Although the margin of error was +/- 5.2 for the most important question relating to who the respondent would vote for, +/- 49.0 makes me question the methodology.

    The special election has been fascinating to me because of polls like these (I’m D2 because I’m country). I wonder how such polls actually influenced the election. Did polls actually help Hanabusa in the election by getting her opponents to focus their attacks on each other while mobilizing Inouye-crats to mail in their ballots? Interesting, very interesting.

  2. It wasn’t just “the Advertiser poll” that got the numbers wrong. So did “the White House pollster,” (4-26) and Ward Research (4-28). The polls which APPEAR to have benn more accurate were the DailyKos and Merriman River (Civil Beat) polls, which showed Djou ahead and the Dems in a virtual tie.

    But each of these polls was only a snap shot, however inaccurate, of the moment it was taken. So we have no way of judging how accurate a poll might have been the further it is from the actual vote.

    Dan Boylan was quoted in an oft-repeated AP story that Hawaii voters are very difficult to poll, particularly AJA voters who may say they are “undecided,” when they really mean, “It’s none of your business. So to Dan’s credit, he suggested the polls might be greatly underestimating Hanabusa’s support.

    I “bought into” the message of the polls that only Case had a chance to defeat Djou and was on the edge of calling together a media event in a local bar where unhappy Democrats could get together publicly, lament about the dilemma facing us, drink some liquid courage and mark their ballots. I had conceived it as an occasion for Hanabusa supporters to “be realistic,” hold their noses, impair their judgment and vote for Case. In discussing the idea with others, I found their were Hanabusa supporters who wanted the liquid courage to hold fast and stick with Colleen, while others wanted the excuse of being drunk so they could be UN-realistic and vote for Del Castillo.

    A high-level aide in Hanabusa’s campaign was assuring me their tracking polls were showing Colleen would beat Case, whose support was fading. They were angry the DCCC and White House had propped up Case beyond his “pull date” and were interfering with Colleen’s ascension.

    Some graduate student while write a thesis on the polling in the race. I look forward to reading it.

  3. Ed Case just sent out a whiney email about “the Dark Side of Politics,” explaining that he lost because “both of my opponents” spent $1M in “false attack ads” in the last weeks of the campaign.

    He cites a Politico article where GOP operatives are bragging how they “threw gasoline on the fire” of the tensions between the Democrats.

    I recommend folks read the article themselves. They really WANT to think they are diabolically clever, and Ed apparently wants to ascribe his defeat to their machinations. Certainly, no Democrat could be angry at Ed for his own behavior and statements. We could only dislike him if we are manipulated into it through “false attack ads.”

    Actually, I recognized early on that Ed might win the nomination and tried to hold back my fire in discussing him. Those times when I cut loose were in response to things he was saying, or his online supporters were saying, which reminded me what a self-righteous, pompous a$$ he can be.

    This latest email is just further confirmation of how defective and deluded he is. He thinks he will only be rejected if others conspire to cheat him. No, Ed. A lot of us disagree with you and dislike you. Not because we don’t know you well enough, but because we DO.

    I explicitly appealed on major blogs for Ed to make SOME overture to progressives to vote for him. I suggested he could try to explain, even apologize for his previous strong support for Bush’s war on Iraq. His people definitely read my remarks. I get feedback. I expect Ed himself read them. But he offered nothing. He appears to have thought we were trapped into supporting him if we wanted to defeat Djou, so why would he waste political capital by making it easier for liberals to vote for him?

    And, no, wrapping yourself with pictures of Obama was not enough.

  4. I’m the editorial page editor so I had no involvement in the polls conducted by the Advertiser newsroom, but I wanted to respond to some of the comments about the poll published on May 2. In previous editing positions here and at Mainland newspapers I’ve been directly involved in polls and poll stories and I can tell you that there is only one outfit more invested in doing polls correctly and without bias than the newspaper, and that’s the firm whose name appears on the poll. In this case, as it has been since the mid-1990s, it’s been Ward Research. I’ve worked closely with Becki Ward on probably two dozen polls on a variety of political and non-political topics and there is no one who takes her work more seriously. In this era of “faux” polling with automated callers and online surveys, Becki still does it the old-fashioned way with an actual human being doing the calling, using a precise method for ensuring geographic and demographic diversity and, important in this race, ensuring that the number was actually in CD1. The numbers come from Voter Contact Services, a company that matches updated voter registration information to listed phone numbers, meaning they are almost exclusively landlines. But these are also the people who tend to vote. People who answered the phone were asked if they were registered and how likely they were to participate in the special election. Only those who said they were registered and “very likely” to vote were asked this question, which appeared on the front page of the May 2 edition: If the special election were held today who would you vote for? The poll story clearly made the point this was a small sample of 349 voters and that it had a margin of error of +/- 5.2 points. That means Ed Case might have 28% of the vote, or as much as 33%, or as little as 23%. Becki always reminds us that this is a snapshot of a moving target and lots can change as election day draws near – I guess we assume readers look at it that way, too, but maybe not. One last point. One commenter made some crack about people wrongly assuming the newspaper is “an unbiased purveyor of the news.” Oh boy, the old left-leaning/right-leaning media canard. Yeah, we sat around and cooked the poll numbers so they’d do what exactly? If anything, Djou should have been pissed that his strong lead in the polls would fire up the opposition and lead right-leaning independents to simply shrug and not bother to vote since he had it in the bag. No human being is unbiased, but after doing this journalism stuff for 28 years I think I can safely say we’re pretty indifferent when it comes to politicians.

    • Jim,

      You decry “faux polls,” yet your paper was clearly engaged in running bogus “online polls” as a means to get hits on your pages and generate ad revenues. These online “faux polls” have helped cheapen public political discourse and undermine appreciation for real polls.

      Turning to “real polls”: I am sympathetic to the difficulties of polling in a political race. But let’s start with a frank admission. Despite the safeguards you list, your poll, and others, were way off the mark. I’m not interested in blaming you. I am interested in having more reliable information getting to voters, since voters will often rely upon polling data to help them choose how to cast their vote most effectively.

      You mention how a poll is a snapshot of a moving target. I find polls to be most useful when they use the same methodology and questions, over time, so voter attitudes can be tracked. Even if a poll is unintentionally biased, if the results are shifting, we can still learn something about trends.

      Unless or until you resolve these problems, why should we take your polls seriously?

  5. The precinct report is up

    Also, the only “poll” that I saw that predicted Hanabusa over Case was in a KHON article about the accuracy of “tweet” activity prior to recent elections.

    The numbers were roughly 49-28-24, Djou-Hanabusa-Case.

    • What’s needed is the “detail” file.
      Check the results of the 2008 General Election.
      On the left side of the page, there’s the “Summary File” and then the “Detail File”.
      The latter is database ready, so that you can download it and massage the data yourself looking for clues.
      Can’t do that with the pdf posted for the special election.

  6. We need instant runoff voting (IRV), especially in special elections like primaries. The whole Democratic Party split scenario could have been avoided, but it would also empower people to vote for candidates like Del as their first choice.

  7. ohiaforest3400

    At the risk of sounding like an obsequious schmuck, Jim Kelly’s comment is but one example of why the demise of the Advertiser, specifically, AND the loss of a two paper town, generally, is so lamentable. The Advertiser’s editorials have been SO much better since his return to the paper, even tho’ I haven’t always agreed with the positions taken. And can you imagine some of his predecessors making the time to comment as he did here, especially with the same attitude, edge, pizzazz to go with it? Not likely and, unfortunately, perhaps even more unlikely in a couple of weeks . . . .

    • Yeah, I love how they routinely bash the homeless now and come up with simplistic demands about issues they obviously don’t understand and don’t invest any effort to figure out.
      Oh, what pizzazz. Catch me as I swoon.

      • ohiaforest3400

        Fair enuf Heywood (or as “Click and Clack, the Tap-it Brothers” of NPR’s “Car Talk” fame would say, “Heywood Ja-Buzz-off”), but it’s a relative thing. It’s WAAAAY better than it was and as “someone” once said, “don’t let the perfect be the enemy of the good (or at least better)”.

  8. This has nothing to do with the polls, but with the special election in general… my feeling is that Case thought he would win this easily, because of his name recognition going into the race and his Democratic affiliation. But throughout the race, I thought he kind of “phoned it in” — particularly when thinking about the way he campaigned this time around compared to when he vigorously campaigned for the Senate seat against Akaka. I am glad that Hanabusa fared better and was pleasantly surprised when I saw the final results.

  9. For a RWDB read on the discrepancies between the polls and the election results, go to Andrew Walden’s blog:

    His “analysis” is helpful, as it provides a negative example of how NOT to interpret the results.

    He cites Boylan’s early warning about under-reported AJA opinion, but rejects it as “mystical.” Instead, he attributes it to a “fear of retaliation” felt by union members. In his view, “Hanabusa voters feared to speak up” when called by pollsters. Notso with Djou and Case supporters.

    Logically, this might lead one to believe that Hanabusa supporters feared “punishment” from Case or Djou supporters, while Case and Djou supporters had no fear of retaliation from the “Democratic Machine.”

    If union members REALLY feared punishment from their “union bosses” or the “Democratic Machine”, one would expect the polling discrepancy to go in the other direction. “Fearful union members” would over-report their support for Hanabusa , but then, in “the privacy of the voting booth”, would rebel against the bosses and vote for Case or Djou. So the polls would have shown MORE support for Hanabusa, while the election result would have fallen short.

    But Andy has an explanation for this. It is not his theory which is 180 degrees at odds with the data. It is that “such people” live in fear of retaliation but are so confused that they actually vote for one of the leading representatives of the oppressive machine!

    Gee, it is great to have a mind so flexible that data can be twisted 180 degrees without invalidating one’s thesis. To paraphrase Stephen Colbert, “social science methodology has a notorious liberal bias.”

    (Hey, if I am being unfair to Andy, someone else can read what he has written and set me straight.)

Leave a Reply

Your email address will not be published.