How two pollsters showing identical results can deliver two completely different accounts of voter behaviour – why timeliness matters if you want to avoid bad analysis
Today we see a Qld State-based Newspoll released by The Australian, accompanied by the usual bad analysis across the media spectrum that’s come to be a bedrock of the Australian political scene.
The Newspoll was taken across the three month period from July to September – where Newspoll adds Qld State-based political questions to their Qld component of their regular fortnightly national sample. After 3 months, Newspoll has a large enough Qld sample to publish the Qld based results.
In the ordinary course of events at the state level, quarterly results like this tell a pretty normal story – but Qld has been far from ordinary over the last few months, so the Newspoll component from the early part of its sample becomes a bit of an exercise in nostalgia and telling us next to nothing about what has happened *within* the long period of the sample.
To demonstrate how timeliness in polling is important when it comes to understanding what is actually happening among the population you’re measuring, we can compare monthly ReachTEL polls that were taken over the same three month period that the Qld Newspoll was taken.
First up – let’s look at the two party preferred results over the July to September period.
The ReachTEL monthly polling averaged out over the 3 months produces an identical result to the Newspoll that was in the field across that same period. Yet the polls tell two completely different stories because of the timeliness involved. Newspoll says the LNP has experienced a 3% swing away from it since the election and everything you’ve seen happening in Qld over the last few months has been little more than theatre.
The actual story is how the Newman government’s support has contracted from 68% in early July to 55% today – a massive 13 point drop – and one much more compatible with the events we have all witnessed in Qld.
We also see the same thing with the primary votes. Newspoll had Katter at a fairly ridiculous 1% in this poll (compared to their election result of 11.5%) – suggesting that they may not have added Katter to the list of party names they read out to respondents – yet Newspoll also had the broad “Others” at a massive 12 points.
UPDATE: Newspoll actually had Katter’s Australia Party on the readouts to respondents, making it an even more extraordinary result!
Comparing the ReachTEL and Newspoll results over the period tells the story – especially if we compare like with like and add the Others vote to the Katter Australia Party vote to accommodate Newpsoll’s weird treatment of KAP as 1%’ers. Again the results are nearly identical.
The Newspoll story is a 2 point primary swing away from the LNP compared to the election – a nothing to see here result – compared with the 12 point swing away from the Newman government since July. Same results for the two pollsters over the 3 months, but Newspoll washes out the underlying dynamics.
The approval ratings of Campbell Newman also exhibit the same behaviour. While Newspoll measures “satisfaction”, ReachTEL measures performance in terms very good, good, indifferent, poor and very poor. So we can use ReachTEL’s Total Good and Total Poor to compare with Newspoll’s Satisfied and Dissatisfied to give us a handle on generic approval.
Even with different questions containing different descriptions of generic approval, the results are again almost identical, and again tell two completely different stories. The Newspoll story is that Newman still popular while the ReachTEL story shows him crashing and burning like no other leader of a new government has in recent history. For instance, we can compare Newman’s approval with a bunch of other leaders of new governments to show the extent of the crash (click to expand).
So let this be a lesson in the dangers of timeliness when it comes to polling. Polling aggregated over long periods of time in dynamic political environments runs the risk of telling you a story that is at best meaningless and at worst completely wrong. Here the Newspoll and ReachTEL results averaged over the same period give identical results – but underneath, taking account of timeliness – the polls tell a completely different story. One is an accurate reflection of current Qld voter behaviour and the dynamics involved, the other is just bad analysis.