A few things need to be pointed out today, because, frankly, the MSM is filled to brim with absolute horseshit about the reality of the Qld election polling results.
Firstly, the polls were not wrong. Journalists and the Commentariat were wrong – the polls were not. Let’s look at the last polls from both Galaxy and Newspoll and see where their results sat compared to the election result once we add in their respective margins of error.
Does that look like the polls were wrong to you?
No, of course not. The election result fell within the margin of error of the last poll from both Galaxy and Newspoll. Statistically, the polls were 100% as correct as they could be. Unless you’re willing to completely redesign the physical laws of the universe, polls can’t get any more accurate than these were.
This commentary about the polls being wrong is just blatant arse covering and a piss poor attempt to rewrite history by a bunch of journo’s that got.it.wrong. Their prognostications on the polling results throughout the campaign were fantasy plucked from a random orifice, that had absolutely no relationship to electoral reality.
When a Galaxy or Newspoll came out showing 50/50 or even a 51/49 lead to the LNP, the News Ltd press regaled us with tales of a strong LNP position, or the likelihood of a hung parliament or even the goodly chances of an LNP victory. The problem with this line is that it completely ignored the reality of the electoral pendulum – where ALP seats sat and on what margins. A 50/50 result would have been a 5 point swing, if uniform, leading to the LNP picking up around 12 seats. A 51/49 result would have been a 6 point swing leading to the LNP picking up around 15 seats if uniform.
Since the LNP needed 18 additional seats give them a chance with a hung parliament, and 22 additional seats to give them the Treasury benches in their own right – a 50/50 result doesn’t even come close as a piece of evidence that could possibly justify the media spin.
To really highlight the complete ignorance behind these MSM poll commentaries and to kill another furphy that’s popped up – that the betting markets were superior – we can turn the pendulum into a set of implied probabilities of ALP victory for any given ALP two party preferred result using a monte carlo simulation. For each tenth of a percentage point increase in the ALP TPP vote, a 200,000 iteration simulation was run using a uniform swing that the TPP result impied with a standard deviation of 4%. We can then see how many times from each of those 200,000 election simulations that the ALP would win at least the 45 seats needed to form government.
A 51/49 LNP result – the best polling result they ever achieved in the campaign – still had the ALP on an implied probability of winning government in their own right of 66%.
A 50/50 poll had the implied probability of an ALP victory at a whopping 94% simply because of where the seats that the LNP needed to win sat on the pendulum.
The final and largest sample poll of the campaign was Newspoll. Its two party preferred result was 50.1/49.9 to the LNP – an implied probability of 92.4% of victory to the ALP.
What the MSM seemed to forget or ignore was that elections aren’t decided by who wins the two party preferred, but by who wins enough seats to form government.
As for the betting markets being superior, we can also use that same simulation to knock this nonsense on the head. At the closing of bets, Centrebet had the ALP on $1.65 and the LNP on $2.27.
That equates to the betting markets giving the ALP a 57.9% implied probability of victory. Every single poll in the campaign however had Labor on a higher implied probability of victory with the lowest (Galaxy 51/49 to the LNP) being 66.4% and the highest (Galaxy 50/50) being 94%. The final poll – the Newspoll – had the implied probability at 92.4%.
Last time I looked, the ALP won government and 66%, 92.4% and 94% are all higher than Centrebet’s 57.9%.
So yes, the betting markets were correct, but the polls were more so – by a significant margin. Something Centrebet should absorb before they say anymore silly things.
UPDATE:
Via scorpio comes the perfect example of polling commentary that is absolute rubbish – factually wrong and completely ignorant of the topic matter: Conal Hanna at the Brisbane Times
22 Comments
Poss
I can imagine the various journo’s and know-it-alls, hunching Quasimodo like, over these results and crying
THE POLLS, THE POLLS
Someone stop those damned POLLS
(like quasi, I think their brain then begins to hurt)
Good analysis, Possum. This has been a bigger beat-up by the commentariat than that we experienced during 2007 with the 12 Month Federal election campaign.
Don’t these people feel even just a bit embarrassed when they spruke their nonsensical, partisan nonsense, day after day?
The probability of an ALP win implied by the polls just before the election you calculate as 92.4 per cent.
The probability of an ALP win implied by the betting markets just before the election you calculate as 57.9 per cent.
What was the actual probability of an ALP win just before the election? It seems to me that we don’t and can’t know, and that therefore we have no mathematical basis for concluding that either of the two implied probabilities, as calculated, is evidence of greater accuracy or statistical validity.
We can now say, after the fact, that the probability of an ALP win is 100 per cent, but the probability of an ALP win after the fact, at any election, is always going to be either 100 per cent or 0 per cent. If (at some other, or hypothetical, election) we had an implied probability based on polls, or a betting market, or some other formula, or just gut instinct, of 50 per cent, would we conclude that we were working with a bad predictor because it was so far off from either 100 per cent or 0 per cent? That doesn’t sound right to me.
I’m not a statistician, so there may be something I’m missing here. Please correct my work.
The MSM commentators must hate the fact that there is such a factor in polling as the MoE. It continually gets in the way of a good beat-up and tends to make them look ridiculous to an ever expanding level of awareness out there in the community. More and more people getting a grounding in statistical analysis and applied statistics.
Which brings me to my pet hate. The nonsensical question imported from the US which is basically useless as a measure. (Is the country,State, whatever, heading in the right direction?).
Just love some rational analysis of poll data. The opinion “leaders” only have one vote or for many in this case none.
Here’s a classic example of where the MSM never got it and where they still don’t get it. Either that, or they have, and are still, being deceitful and treating the punters as fools.
http://www.brisbanetimes.com.au/news/queensland/bwhy-mistress-was-no-match-for-wifeb/2009/03/22/1237656766766.html
They certainly fooled Federal and State Coalition commentators who were unanimous in condemning the inaccuracy of the polls during the telecast and since.
A number of them appeared to be quite upset about the whole thing which I thought was hilarious.
LOL Poss you might be interested in reading Bob Ellis’s take on polling
http://www.abc.net.au/unleashed/stories/s2519417.htm
He senses a conspiracy.
Possum, I agree that the MSM are full of shit, but to be fair to them, they’ve GOT to write something about the election, and the polls are the best information they have. The problem is that an MoE between 5% and 7% wide is completely useless for a close poll – as Barnaby said several times, a swing of that size will bring down most governments. If the polls are to be of any use to anyone, shouldn’t they have larger sample sizes?
What’s it look like if you combine those last two polls to get a “supersample”, Pollytrend style? (Is it possible to estimate the design effect for a weighted combination of those polls?)
Yet more evidence (as if we needed it) that the same news outlets that commission and pay (not insignificant sums) for political polling can’t even understand them. Maybe should they pay a little more to have them interpreted as well –though I guess in these economically challenging times it’s cheaper just to buy the box of Kleenex to remove the egg that is dripping from their faces.
Hi Poss -
Some good points there. You’re absolutely right that the journalists don’t understand the polls, & were producing commentary that the poll numbers didn’t justify. But I think you’re gilding the lily a bit. Three points: (a) altho the result was well within the error margin of any one poll, it’s significant that 5 polls (3 Galaxy, 2 Newspoll) all said basically the same thing, & if you aggregate those samples that brings down the error margin quite a bit (you’re the statistician, you can tell us how much); (b) you say “polls can’t get any more accurate than these were”, but of course they could: they could have picked the result exactly – being within the margin of error isn’t the same as being exactly right; (c) what J-D @3 said on the reality of implied probability before the event, with the further point that deriving that probability from the poll numbers requires an assumption that (at most) the only error in the polls is statistical error, ie ignoring movement of undecideds, house effects/bias, & the possibility that people will change their minds: gamblers are presumably allowing for all of those, so you would expect them to put the probability rather lower.
As Bob Ellis himself says, he gets it right sometimes.
His conspiracy theory might have legs if Newspoll relied on Rupert Murdoch for their income. I don’t think they do. I think they rely for their income mainly on businesses who want research to tell them what sort of advertising to run. One of the ways Newspoll touts for their business is by trying to impress them with the accuracy of their political polling (which I’m guessing is a small fraction of their work, but the only part that gets massive media coverage). Deliberately rigging their political polling would, for this reason, be throwing away their own bread and butter.
(The same would apply to all their pollsters.)
Some of the specific points Ellis makes actually cancel each other out, if you understand how the business works, which reinforces my doubts that he does.
Possum, like the dreaded “MSM” I also got it wrong saying the polls were proved incorrect and the betting market was superior. Thanks for the explanation.
Proving only that Bob Ellis is a tiresome, overrated bullshit artist.
All we can conclude is that the polls showed a higher implied probability of the eventual result being achieved than the betting markets did, by significant margins, for the entire the campaign. Beyond that it gets very very difficult. The point I was trying to make is that by the very yardstick the betting market PR folk use to claim their superiority over the polls – probabilities of winning vs. the eventual outcome – they weren’t more accurate at all.
Scorpio, thanks for that link, what a cracker of an example of what not to say!
Friendless,
Bigger sample sizes are always good, but the trade off is increased costs of the polling itself.Polling is actually pretty expensive business, so the larger the samples the higher the costs, which would mean less polls being commissioned over the course of a campaign for any media outfit with a set budget.
Caf,
I’m really hesitant about pooling those last two polls since they’re the only polls for the last week of the campaign, and the period both polls were in the field covers 3 days. To pool them we would be making an assumption that over those 3 days, public opinion was completely static. We could estimate a DEFF for them utilising previous polls to extrapolate the variance differences between Newspoll and Galaxy – but it would still be pretty much a sophisticated guess and the squeeze isnt really worth the juice since the election result would still fall within the 95% CI of the expected sampling error of the pooled poll.
Once the final election results are tallied up, the LNP primary will probably be a little higher than it currently is, the ALP primary a little lower than it currently is and the TPP a little lower for the ALP than it currently is. Both the primaries will be less than 1 point out from the final Newspoll, meaning most of the TPP error comes not from polling estimate issues, but from the way preferences were allocated by the polls to achieve a TPP estimate. In an OPV system, the TPP of the pollsters is pretty damn good!
Charles,
On aggregating the polls, we would be making the assumption that public opinion was constant over the period and there’s not enough evidence to back that assumption. People can and do change their mind – to pick out that change, like we do with Pollytrend in the sidebar, we need a great many more polls than were conducted in the lead up to and during the campaign. If there were 30 polls in the 6-12 months before the campaign and 15 polls during the campaign, we could do some spiffy things to track a likely trend – but with the small amount of polling we had to work with it’s really beyond our scope unfortunately.
When you say on the polls:
I’ll tell you something that will change the way you look at polls forever.
Let’s say we have a hypothetical poll, a poll so perfect in every way that it has absolutely no non-sampling error. The only difference between the results this hypothetically perfect poll produces and the true level of public opinion is in the sampling error which our MoE is derived from.
Let’s say that the true level of public opinion was 50% on some question, and that this perfect poll tried to estimate that true level of public opinion using a sample size of 2400 (which gives a margin of error of exactly 2%)
Whether the poll comes up with an estimate of 48,49,50,51,52 or something else is ENTIRELY up to chance.
1 in 20 times it will come up with a result that is outside of the 48-52 range – simply as a result of the mathematics of sampling and even though this hypothetical poll is perfect.
In the 95% of cases where this hypothetically perfect poll gets a result within the 48-52 range, it will be a random result drawn from a normal probability distribution with a mean of 50 and a standard deviation of 1.
That means the poll result will more often than not be closer to 50 than either 48 or 52, but the actual poll result itself is random.
If a pollster gets the result perfect, it’s luck. If three pollsters were measuring an election and the election result ends up within the MoE of all the pollsters final results – the pollster that got closest will claim they were the most accurate, but that’s just marketing spin as all the pollsters were equally right and it was nothing more than pure luck that one ended up being closer than the other.
Hi Poss -
Sure, given that there’s no non-sampling error, how close you get is pure chance. But (a) that’s a heroic assumption, & (b) even getting inside the margin of error is chance – if it’s one of the 5% that falls outside the confidence interval, that’s just bad luck too. And b/c it’s a normal distribution (ie not everything within the confidence interval is equally probable), the hypothetically perfect poll will be more likely to get the exact right result than any other particular result. There’s still such a thing as being right, even if the reason you’re right involves a large component of luck.
Possum: Thanks, that’s pretty much what I was wondering anyway – whether the election result would still fall within the CI of the pooled poll.
What all the stuff about bragging rights for polls really comes down to is that you can’t judge a pollster’s accuracy on one election – you need to look at how the pollster has gone over many elections. Just like you can’t draw a sensible conclusion by polling only one voter!
Oh and after Mumble’s comment in today’s Crikey I reckon a post might be in order explaining DEFF and why you can’t just add the sample sizes from different polls together
It’s a conclusion that is difficult not to be drawn to.