Remembering back to last month when the unemployment figures were released by the ABS showing a fall from 5.7 to 5.4%, the howls of incredulity from economic firms that guessed wrong on this was deafening. The ABS unemployment figures are actually derived from a poll, albeit an enormous survey of around 41100 people, and it was pretty much lambasted across the economic sector as a “rogue” result and a consequence of the ABS reducing their sample size. The reporting of the issue was just mind numbingly dodgy.
So the poor old ABS got slapped around for weeks on how their small sample size of 41100 was creating all this enormous volatility and uncertainty in their figures, and as a result of this hysterical (and let it be said, mostly ridiculous) gnashing of teeth and stomping of tootsies by private sector economic firms – the ABS has now restored their Labor Force Survey to run on a sample of around 54400.
To point out just how silly all this bleating has been over the last few weeks, we can use our new toy The Poll Cruncher to highlight how most of the commentary on this hasn’t been so much as clueless, but more a case of simply misappropriating the plot.
If we plug in 41100 as the sample size and 5.7 as the result for Poll 1, the margin of error on this survey comes out as 0.22%.
If we plug in the larger sample size that the ABS used to run and will now do so again, a sample size of 54400 with the same result of 5.7 as Poll 2 – the margin of error is 0.19%.
All this outrage has been over a structural improvement of 0.03% in the accuracy of the raw results!
Let’s get something straight here – this new sample size will make exactly three fifths of five eighths of sweet fark all difference to the unemployment figures. The trend is what matters, and this chicken feed increase in accuracy won’t make an ounce of difference to the trend estimates.
There is a law of diminishing returns when it comes to the sample size/accuracy nexus for surveys, where the marginal increase in accuracy (say, the increase in accuracy for every 1000 increase in the sample size) continually decreases as the sample size increases. When we get up into the tens of thousands as a sample size, the juice becomes barely worth the squeeze.
People have to realise that sometimes survey results hang out on the fringes of reality through no fault of their own – it is just how samples work in practice. The best we can do is increase the sample size so that the distance between where the definition of “the fringe” sits compared to the true results is small – but a 0.03 reduction in that distance is not going to change anything. If 5.4 was considered “rogue”, would 5.43 be considered any less of an outlier?
Of course not, but that’s the size of the distance we’re talking when we look realistically at the new change in sample size.
One of two things was at play here – either the poll was an outlier or the poll wasn’t. There was around a 5% chance that it was an outlier and if that was indeed the case, then we can’t really say anything about what the true figures were except that it’s highly likely to be somewhere around the fringes of either the 5.4 or 6% mark. If we had a larger sample size, it wouldn’t have made any material difference to the results.
If the poll wasn’t an outlier – there is still a chance that the true unemployment result was somewhere between zero movement and a small increase. If we now go back to the Poll Cruncher and change both sample sizes to 41100 (reflecting what actually occurred) and plug 5.4% in as the result for Poll 2, as well as 0 and 5 into our Min and Max values for Poll 2 , we get a probability of there actually being a zero to positive increase in unemployment over the period of 3%.
Worth mentioning is that we can use any large positive number instead of 5 here in The Poll Cruncher, as we are trying to find the probability of the increase being greater than or equal to zero rather than between any two given values.
No doubt, come next Thursday when the Labour Force Survey results for May are released, there will be a large gaggle of talking heads in the serious media waxing lyrical about how their criticisms of the ABS were justified (if unemployment goes up) or how there’s something wrong with the figures again (if unemployment stays the same or goes down).
The one thing we can probably all be sure of is that nearly every piece of commentary on the methodology of the unemployment figures will again be spurious nonsense – at least if last month was any yardstick to go by. I often wonder whether the talking heads have ceased selling their firms nouse as the product, and where the product they are trying to flog has simply become the sound of their own voice.
10 Comments
I always wondered what happened to the Talking Heads Poss; they were always better at making music than commenting on statistics
Good post. After the figures were released I came here to see what you had to say. Thanks for illuminating us on the real impact of the cost cutting measures, I now think I’d been a little too critical of the ABS cutbacks
Seems to me like the previous month’s figures were probably the dodgy ones rather than last month’s. Of course the meeja never looks too deeply at things like stats when they have a bad news story to run
Nice work.
Actually Pos, at least some of the criticism has been about the “methodology” or “definition of unemployment” with regard to this survey. The question (at least in my mind) is not so much the “accuracy” of the poll but the “accuracy” of the measurement method.
I think it is a bit suss to have all these hidden rules about what “unemployment” is and isn’t – when most thinking people would agree that they create an “artificial” measure.
The real question is “what is the value of an unconstrained measure of unemployment”? It certainly won’t be anywhere near 5.X%.
If the ABS (and others) want to run this poll and post their artificial numbers then that is fine – but it really isn’t a measure of “unemployment” and they should say so.
Perhaps the poll should be called “the politically convenient measure of something”.
Roxx
Unemployment is always going to be hard to measure because we’ll never get a single definition of unemployment that everybody will be happy with and which can be used for all possible purposes.
For instance, would you classify a person as unemployed who:
a) doesn’t have a job
b) is looking for work… but
C) will only accept the absolute perfect job if it comes along?
What about a person who:
a) works 3 hours a week
b) is looking for more work
c) will take any full time or additional part time job that’s offered to them
The first person is only dubiously unemployed but is counted officially, while the second person is as close to unemployed as you can practically get while not being classified as such.
For me, the official unemployment measure is perfectly good for what it is – a consistent measure of a particular definition of unemployment.
But that alone isn’t the whole story – if we want more than that, the ABS employment underutilisation rate is a good measure of underemployment.
http://tinyurl.com/m2nspg
What a lot of people don’t realise about our current definition of unemployment is that it hasn’t really been fiddled with by politicians – it’s actually an International Labour Organisation standard.
And while just about every politically interested person has an idea of what we think should be defined as unemployment, most of our definitions differ from each other. So we’ll always have this problem regardless of what we define unemployment as.
Exactly Pos,
And when the quarterly labour force underutilisation rate was 12.5% in February 2009 and this represents the largest quarter to quarter movement of the seasonally adjusted data since the series started – I would have thought that there would be less focus on an arbitrary number in the 5.X% range and somewhat more focus on another arbitrary number in the 12.X% range.
I reckon that “unemployment” is much closer to 12% than it is to 5% – and that is the issue we should all care about.
You don’t know that though.
Thanks Pos,
Of course this confidence interval reasoning is correct, and innumerate commentators have been beating up something out of nothing. Must have been short of a story that day.
BUT, is there a plausible explanation of the central argument – that unemployment DID go down? From where I am, I’ve seen corporates and SMEs lay off staff in Oct to Dec 08 in anticipation of death, only to need staff in Mar to May. I haven’t been talking with banks lately, but telcos, retail, health services and food have all felt that their prudence in Q2 was a bit (but not a lot) too strong.
Tomasso.
You are still wondering whether this is the case???
Not sure if you heard or cared, but your entry was discussed in estimates hearings of the Senate Economics Committee during questioning by the Chair of the committee (Senator Hurley) to witnesses from the ABS.
Didn’t know ltep.
I’m sure it made for riveting viewing!