Would you publish?
We know outliers are inevitable, they’re something that cannot be avoided – but from a pollster’s perspective, running a poll where your results look a little iffy becomes a bit of a two edged sword. Regardless of whether they publish or not, ultimately they lose in the short term.
If they take the “publish and be damned” approach it sets the hares, like me, running around screaming “Outlier!”. Others might start to see manipulation where none exist, questioning the polling organisation itself rather than simply the poll it produced.
Alternatively, if they withhold the poll from publication, questions are inevitably asked about “Why?”. This too sets the hares running with accusations inevitably flowing about how X media organisation is boosting Y political party by not releasing polling data that is probably bad for them. What ever decision a pollster takes, there is an automatic downside to producing an outlier that extends beyond the actual numbers themselves.
It’s a really tricky question that every pollster has to deal with at some point in time – and most come down on the side of publish and be damned.
So what should a pollster do? What would you do if you ran a polling organisation that produced a result that was almost certainly an outlier?
Is there any way you could not lose in the short term simply by producing an outlier poll that probability alone suggests is inevitable at some time?
51 Comments
Pages: « 1 [2] Show All
But back to your question of whether to publish or not.
I suggest that this is only a dilemma because the polling companies themselves have unrealistically raised expectations by overclaiming accuracy for their polls.
For example, if you look at the websites of Morgan and Newspoll, they make claims about being the most accurate – giving examples of elections they predicted within less than 1%.
The Morgan claims are particularly amusing. On their website they claim that they were most accurate in the 2007 election because they had an average difference of .5% across all parties while Galaxy was .6% and Newspoll .9%. They also claim Galaxy and Newspoll overestimated the Coalition 2pp (actually by .7%) whereas they were most accurate (they only overestimated the Labor vote by .8%). I know that this doesn’t make sense but you have to realise it’s Morgan. Even funnier, Gary Morgan actually believes that his poll is more accurate than standard error estimates would indicate.
However, Newspoll also claim superior accuracy citing an example where they were accurate within less than 1%.
The pollsters themselves have encouraged an unrealistic perception of their accuracy. They have created their own problem and should wear it.
Pages: « 1 [2] Show All