Jan 17, 2010

# Nerdy Sunday - When Trends Go Bad

We often use the phrase “the trend is your friend” when analysing noisy data, primarily because it’s a pretty good rule of thumb for the type of polling, economic and dem

### Share

We often use the phrase “the trend is your friend” when analysing noisy data, primarily because it’s a pretty good rule of thumb for the type of polling, economic and demographic data we usually deal with round these parts. Yet sometimes, with certain types of data that exhibit autocorrelated random noise, the “trend”, particularly any local trend witnessed across a relatively short time period, can be extremely deceptive.

To demonstrate, first up we’ll create a really simple time series of data that will become the “reality we are trying to find” for the rest of the post. It will be the “real trend” we will try to find after we swamp it with random noise.

This trend is pretty simple – at observation zero it has a value of zero, at observation 1 it has a value of 0.05, at observation 2 it has a value of 0.1 – each observation the value of this series increases by 0.05. If we look at the first 20 observations of this series, it’s a standard straight line:

Next up, we need to create some random noise to overlay onto this trend – but we need to make that random noise similar to the sorts of wandering behaviour that regularly infects real world data series. This requires a two stage process, the first part of which is to simply generate some random numbers. For every observation, we will generate a random number between 1 and minus 1 inclusive, to 1 decimal place – so the numbers that will be randomly generated will be out of the set (-1, -0.9, -0.8, -0.7, -0.6, -0.5, -0.4, -0.3, -0.2, -0.1 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1)

Our first 20 observations of our random number series look like this:

Next, we’ll autocorrelate this series of random numbers for each observation (EDIT: autocorrelation is where the the value of a series at any given point is correlated with, in this instance, it’s most recent value). To start, our zero observation will have a value of zero and our first observation of our autocorrelated series will have the same value as the first observation of our random number series – which in this case is -1.

That’s just required to start the series off. Now, the value of observation 2 will be the 2nd observation of the random number series (0.8) PLUS the previous observation of our autocorrelated series (-1), to give us a value of -0.2. The 3rd observation of our autocorrelated series will be the 3rd observation of our random number series(0.9) plus the 2nd observation of our autocorrelated series (-0.2) …..etc etc. The value of the Nth observation for the autocorrelated series will be the Nth value of the random number series plus the (N-1)th value of our autocorrelated series.

This is what the first 20 values of this now autocorrelated random noise looks like.

It’s worth noting at this stage that the noise involved is much larger than the observation by observation increase in our “real trend” data. Our real trend data increases by 0.05 every period, but the noise can change by as much as 1 or minus 1, a 20 times larger increase than our real trend data, but in either direction.

What we do next to get a noisy version of our real trend data is simply to add the autocorrelated random noise series to our real trend series. If we do this for 300 observations, we can compare how our Actual Trend (the real data we are actually trying to find) compares to the Trend + Random Noise (the type of series that we would see in real life)… (click to expand)

What I’ve also added here is a simple linear regression line through that Trend+Random Noise series. What is interesting at this stage is how the noisy series is behaving fairly differently to the Actual Trend series that is its underlying basis.

If you were an economist, or a statistician or some other data scientist and you were given a series of data to analyse, it will often look like the red series above. The problem to be solved is what the real, underlying trend data might actually be. We know what the real trend is – it’s the blue line marked “Actual Trend” – but trying to figure that out without knowing it can be a complicated process, especially when the series has a lot of autocorrelated noise in it.

Let’s now look at the first 800 and 1500 observations:

As the number of observations increases, the noisy series slowly starts to converge on to the Actual Trend. If we now move on to the first 5000 observations, it really starts to become apparent:

Over an infinite number of observations, the regression line through the “Trend + Random Noise” series would become identical to the Actual Trend that is our underlying data we would be attempting to find.

What is worth pointing out at this stage, is that even over large numbers of observations, series with autocorrelated random noise (or series where more than one thing is influencing it) can produce local trends that are not representative of the the real Actual Trend underlying the data – trends can be deceptive even after hundreds or thousands of observations depending on the nature of the time series we are looking at.

Let’s now take a look at a small subsection of the series and run the numbers again, but where the regression line for Trend+Random Noise is just taken over the sample that is on the chart – it’s 260 observations long.

The local regression would have us believe that there has been a decline in the value of the series over this period, and a statistically significant one at that – even though we know that the Actual Trend has continued to increase at 0.05 units per time period. If this occurred in the real world with some real, important data series – we’d have 3rd rate columnists in the the press banging on about “Teh Decline!!!!!!”, accusing those professionally trained people that would be attempting to point out the “Actual Trend” as being dishonest conspirators.

This brings me back to the problem of people talking about things like “temperature decline since 2001” and the type of arguments used by some in comments in our post about it.

Folks, it’s an exercise in whistling out your arse – you are essentially arguing about random variation so large that it swamps any underlying trend.

Hopefully, from this, you can see why and how that happens, and why and how it is not only futile, but meaningless.

What is important is the larger trends over larger time spans and any structural change that might be measurable in a series, not twaddle about the behaviour of random noise over a short time frame.

## 120 thoughts on “Nerdy Sunday – When Trends Go Bad”

1. PeeBee

JamesK,

I’m glad you are back, because I have a very serious question to ask you (and could be asked of anyone).

I take it most of your information come from blogs of which there are many to choose from from both sides of the debate.

My question is: ‘how to you dismiss some blogs as useless and others that are good and worth quoting?’

I am intersted in your criteria.

2. deconst

If you want to use the most accurate temperature data available, satellite data commenced in 1978 so we now have two 30 year temperature trends for the period 1978 to 2010.

I am not sure what you mean by a 30 year period between 1975 to 2000. However I have some 30 year trends from around that time for you.

3. JamesK

What deconst….. the 30 year period 1975 – 2000?

4. deconst

It’s not statistically significant because there are a series of simpler reasons for why the 00s have plateaued: 1. Falling solar cycle 2. Pacific Decadal Oscillation low 3. Peaking El-Nino event in 1998.

Scientists are adamant that only a period of 30 years can encompass all these events. If there is a falling 30-year trend, Bolt, Tamas Candelabras & all would be justified in saying ‘There’s a decline!’ However a statistically significant decline hasn’t happened and won’t happen. The majority needs to move now on action on Climate Change.

Decreasing 30-year trends happened regularly before 1944. None have happened since then, and that is extremely significant.

5. Just Me

“So Bolt is actually correct to say it has cooled but not to say there has been a statistially significant cooling trend.”
JK @ 23

If the ‘cooling’ trend is not statistically significant, then there is no trend and it has not cooled.

6. Possum Comitatus

Thanks JamesK – sniff – it means a lot 😛

7. JamesK

Evan Can’t you recline back a minute or two and simply admire brilliance?

I notice this thread died in my absence.

I felt momentarily sorry for Possum and thought T’d give him a fillip

8. Evan Beaver

I thought you said you were leaving James?

What information would convince you of the truth in the AGW hypothesis?

9. JamesK

But Dave, what lefties promoted fallacy explains Martha Coakley the Democrat candidate for Ted Kennedy’s old Senate seat from the deep blue state of Massachusetts having a 30 point advantage over her Republican challenger Scott Brown at the beginning of the campaign whilst now the day before the election Brown has a 2-5 point advantage in most polls?

And that’s after The One came visited for her and did his waving his anointed hands about whist surreptitiously giving the world the bird trick. (I know they couldn’t fill the 3000 seat auditorium but still)

Can the clever psephologists not explain to the 7 million dumbass voters of the Bay state that the trend lines there are not statistically acceptable even though 2 months is more than 8 weeeks in politics?

Have the psephs failed to persuade the plebs to vote lefty come Tuesday?

Haven’t the psephs sweet-talked the plebs into accepting that nanny state health care is in their ‘best interests’?

Brown drives a pick-up for f–k sake.

He’s blow-torching the planet with his carbon emissions.

If Tony (aka Scotty) Abbott (and we know he’s brown already) hoons about in a V8 Falcon Ute, the only solid trend lines will be burnout tracks and the Krudd will be toast.

And seeing as how the planet has cooled the last decade you won’t be able to blame the charring of Krudd on climate change.

I mean it would be dramatic and disastrous if Krudd isn’t incinerated but that’s another story.

So what fallacy will you apply?

10. David Richards

Applying this to another fallacy promoted by the righties – that of the supposed swing back to the Liberals (based on one rogue Newspoll and 2 uncontested byelections), again you get a short term “trend” which is really nothing more than noise overlaid on the long-term trend. Given that the next election is less than 12 months away, you would want to see a more solid trend towards your party if you were to seriously entertain any thoughts of regaining government. Despite the “a week’s a long time in politics” mantra, there would have to be a rather dramatic and disastrous event to reverse the long-term trend.

11. Julian Watson

#109

Zing!!!!!!!!!!

12. PeeBee

Jamesk@107

At least there is one thing we agree on one thing:

Prejudiced opinion dressed as fact wins over reasoned or even interesting argument just because of the sheer onslaught of stupidity.

13. pancho

Great post Chinda. It is odd that a unanimous consensus of experts is constantly arrayed against muckrakers, populists and mineral lobbyists on the premise of some sort of ideological balance. The point about Gore being a rallying figure for both extremes is interesting.

Reflexive anti-elitism seems to be driving many of the responses as well. I thought the hatred of the elites was confined to those dastardly humanities faculties, but I guess it’s good to see science getting a run as well. There is a certain irony in the fact that much of the culture war argument against the humanities over the last decade was supposedly based on their embrace of post-modernism, there being no truth and everyone being entitled to their own facts. That one’s kind of gone out the window now that every conservative-cum-libertarian goon fighting big government and great big new taxes has become an environment expert with ever creative ways of reading/interpreting what doesn’t fit their political narrative.

14. JamesK

“Meanwhile, the climate scientists study, report and peer review like they always have”

Here’s at least three climate scientists that don’t agree:
http://rogerpielkejr.blogspot.com/2010/01/climategate-glantz-versus-chase.html

For the hyper-intelligent but reticent pancho:

http://www.co2science.org//articles/V10/N22/B1.php

http://www.co2science.org//articles/V10/N32/B2.php

http://www.co2science.org//articles/V10/N13/B2.php

That’s it. No more. It is pointless. I am now gone.

Prejudiced opinion dressed as fact wins over reasoned or even interesting argument just because of the sheer onslaught of stupidity.

15. CHRISTOPHER DUNNE

JK keeps banging on about the ‘last ten years’, yet the whole point is that taking just the last 10 years of data and trying to extrapolate a trend is statistically nonsense, especially when the much longer series shows that we have a clear upward trend.

But what we have before us is the proposition that once a troll, always a troll.

Proven beyond any reasonable doubt, huh?

16. zoomster

I knew JamesK was telling an untruth…he didn’t go away.

Chinda and Possum, I get extremely cross with my friend who is a climate change scientist and is regularly rung up by the media and ignores them. Thinks it’s beneath a scientist to get involved in public debate…so leaves it to amateurs who really don’t know what they’re talking about.

This is someone who, believe me, would be lapped up by the media, if they were prepared to risk their reputation as a ‘serious’ scientist. But no.

17. JamesK

I propose that if Evan proposes never to propose his presupposed version of the scientific method ever again then I will never propose that Evan is blog gadfly who has never proposed a proposal of consequence even once among myriad thousands of propsals?

18. pancho

Ignore us if you like Evan. I have nearly finished colouring in the wings to stick to the side of the JamesK Institute building. I am writing our poetic mantra (“you know little but believe much”) down the speed stripe, and we will ride triumphantly into town on the back of the Parrot smiting you pathetic “realists” as we go. Hurrah!

PS. After you have all read James’ 80, please move on to 84: “I rather suspect that the increase in atmospheric CO2 we have had has been also favourable.”

I rest my case.

19. Astrobleme

Damn, messed up the fomratting again.

[The point is that AGW cannot in the normal scientific sense be called any one ‘theory’ let alone one that makes sense.]

AGW is a scientific theory. I outlined it above for you. Here it is again:

[[The Greenhouse effect is an explanation of why the world is as warm as it is. At this distance from the sun and no other factors at play the Earth should be significantly cooler.

Certain gases have the effect of absorbing IR radiation. They absorb a photon and convert that into vibrational energy. This IR radiation must come from the surface of the Earth as theSun emits in the UV-Visible spectrum due to it’s high temp.

the gases that do this are called greenhouse gases, and include water, carbon dioxide, methane, cfcs, and some nitrous oxides.

Humans burn coal, and othe fuels that release Carbon dioxide.

The carbon dioxide is accumulating in the atmosphere.

More carbon dioxide leads to higher temperature in the atmosphere.

World gets hotter.]

20. Possum Comitatus

Chinda went:

[Why can’t people who aren’t climate scientists trust the opinions of those who are? I don’t get why everyone is suddenly such an effing expert they are able to dismiss the opinion (and years of experienced analysis) of people who actually do this for a living.

I mean, if a cancer specialist diagnoses that you have terminal cancer, would you believe the specialist or would you believe your local GP who tells you it’s all bullshit, accompanied by some outrageous explanation at to why the specialist is deliberately lying to you?

If your lawyer tells you the best thing is to plead guilty and wear the fine, would you trust him and his years of knowledge and experience or would you take the word of the law student in the next office who tells you to fight it all the way and hang the expense?

Why is it that the loudest people on both sides of the debate are those who have NO qualifications in any of the climate science disciplines?

And what’s with all the theologans on the denialist side? That’s the one that really does my head in. Why are they weighing in? What do they have to gain (or lose)?

I suspect this is not about the science and never has been. It’s about the long-standing framing of any environmental issues as “leftist” and Al Gore’s inconvenient documentary was the thing that galvanised the Right against this issue. Seriously. How many people the Left are deniers and how many people on the Right are card-carrying wamenistas?

Meanwhile, the climate scientists study, report and peer review like they always have]

The voice of reason!