tip off
120

Nerdy Sunday – When Trends Go Bad

We often use the phrase “the trend is your friend” when analysing noisy data, primarily because it’s a pretty good rule of thumb for the type of polling, economic and demographic data we usually deal with round these parts. Yet sometimes, with certain types of data that exhibit autocorrelated random noise, the “trend”, particularly any local trend witnessed across a relatively short time period, can be extremely deceptive.

To demonstrate, first up we’ll create a really simple time series of data that will become the “reality we are trying to find” for the rest of the post. It will be the “real trend” we will try to find after we swamp it with random noise.

This trend is pretty simple – at observation zero it has a value of zero, at observation 1 it has a value of 0.05, at observation 2 it has a value of 0.1 – each observation the value of this series increases by 0.05. If we look at the first 20 observations of this series, it’s a standard straight line:

trend20

Next up, we need to create some random noise to overlay onto this trend – but we need to make that random noise similar to the sorts of wandering behaviour that regularly infects real world data series. This requires a two stage process, the first part of which is to simply generate some random numbers. For every observation, we will generate a random number between 1 and minus 1 inclusive, to 1 decimal place – so the numbers that will be randomly generated will be out of the set (-1, -0.9, -0.8, -0.7, -0.6, -0.5, -0.4, -0.3, -0.2, -0.1 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1)

Our first 20 observations of our random number series look like this:

random20

Next, we’ll autocorrelate this series of random numbers for each observation (EDIT: autocorrelation is where the the value of a series at any given point is correlated with, in this instance, it’s most recent value). To start, our zero observation will have a value of zero and our first observation of our autocorrelated series will have the same value as the first observation of our random number series – which in this case is -1.

That’s just required to start the series off. Now, the value of observation 2 will be the 2nd observation of the random number series (0.8) PLUS the previous observation of our autocorrelated series (-1), to give us a value of -0.2. The 3rd observation of our autocorrelated series will be the 3rd observation of our random number series(0.9) plus the 2nd observation of our autocorrelated series (-0.2) …..etc etc. The value of the Nth observation for the autocorrelated series will be the Nth value of the random number series plus the (N-1)th value of our autocorrelated series.

This is what the first 20 values of this now autocorrelated random noise looks like.

autocorrelated20

It’s worth noting at this stage that the noise involved is much larger than the observation by observation increase in our “real trend” data. Our real trend data increases by 0.05 every period, but the noise can change by as much as 1 or minus 1, a 20 times larger increase than our real trend data, but in either direction.

What we do next to get a noisy version of our real trend data is simply to add the autocorrelated random noise series to our real trend series. If we do this for 300 observations, we can compare how our Actual Trend (the real data we are actually trying to find) compares to the Trend + Random Noise (the type of series that we would see in real life)… (click to expand)

trend300

What I’ve also added here is a simple linear regression line through that Trend+Random Noise series. What is interesting at this stage is how the noisy series is behaving fairly differently to the Actual Trend series that is its underlying basis.

If you were an economist, or a statistician or some other data scientist and you were given a series of data to analyse, it will often look like the red series above. The problem to be solved is what the real, underlying trend data might actually be. We know what the real trend is – it’s the blue line marked “Actual Trend” – but trying to figure that out without knowing it can be a complicated process, especially when the series has a lot of autocorrelated noise in it.

Let’s now look at the first 800 and 1500 observations:

trend800 trend1500

As the number of observations increases, the noisy series slowly starts to converge on to the Actual Trend. If we now move on to the first 5000 observations, it really starts to become apparent:

trend5000

Over an infinite number of observations, the regression line through the “Trend + Random Noise” series would become identical to the Actual Trend that is our underlying data we would be attempting to find.

What is worth pointing out at this stage, is that even over large numbers of observations, series with autocorrelated random noise (or series where more than one thing is influencing it) can produce local trends that are not representative of the the real Actual Trend underlying the data – trends can be deceptive even after hundreds or thousands of observations depending on the nature of the time series we are looking at.

Let’s now take a look at a small subsection of the series and run the numbers again, but where the regression line for Trend+Random Noise is just taken over the sample that is on the chart – it’s 260 observations long.

whentrendsgobad

The local regression would have us believe that there has been a decline in the value of the series over this period, and a statistically significant one at that – even though we know that the Actual Trend has continued to increase at 0.05 units per time period. If this occurred in the real world with some real, important data series – we’d have 3rd rate columnists in the the press banging on about “Teh Decline!!!!!!”, accusing those professionally trained people that would be attempting to point out the “Actual Trend” as being dishonest conspirators.

This brings me back to the problem of people talking about things like “temperature decline since 2001” and the type of arguments used by some in comments in our post about it.

Folks, it’s an exercise in whistling out your arse – you are essentially arguing about random variation so large that it swamps any underlying trend.

Hopefully, from this, you can see why and how that happens, and why and how it is not only futile, but meaningless.

What is important is the larger trends over larger time spans and any structural change that might be measurable in a series, not twaddle about the behaviour of random noise over a short time frame.

120

Please login below to comment, OR simply register here :



  • 1
    Posted January 17, 2010 at 10:45 am | Permalink

    As clean and clear an explanation as you can get. Like calculus, the non-intuitive nature of statistics trips up the armchair expert. Very useful, thanks!

  • 2
    Rocket Rocket
    Posted January 17, 2010 at 11:01 am | Permalink

    Yes, I love the way various people have to be increasingly selective in their time periods and “smoothing” periods to get the answers they want!

    As time goes by, they look increasingly silly. Eventually they tend to move on to Phase II – “Yes, the climate is warming, but it is not caused by human activity” (“And that’s what I’ve always said” – lots of a***-covering when their “good data gos bad”)

  • 3
    Steve
    Posted January 17, 2010 at 12:01 pm | Permalink

    Thanks Possum, I am a lurker on this site and find your posts extrmemly valuable.

    I do not know much about stats, though I am trying to learn. My understanding is that linear regression not only gives a trend but also a measure of the possible error in the trend estimate.

    Is this true and if so what is the error value for the temperature trend from 2001 that Bolt and others bang on about?

  • 4
    Just Me
    Posted January 17, 2010 at 2:15 pm | Permalink

    “Thanks Possum, I am a lurker on this site and find your posts extrmemly valuable.”

    Me too. Lovely clear explanations of things statistical that I do not pretend to understand well.

  • 5
    Posted January 17, 2010 at 2:32 pm | Permalink

    My projection is that the deniers will completely ignore this post. :)

  • 6
    Captain Col
    Posted January 17, 2010 at 2:52 pm | Permalink

    So Possum, what’s the ‘correct’ way to observe climate change data if a decade is too short? If the long term trend changes for some reason, how long will it take to notice that we should have a second trend line. Presumably, we are allowed to have several trend lines to depict interim variations to the single long term one.

  • 7
    Firstdog
    Posted January 17, 2010 at 3:02 pm | Permalink

    You are the wind beneath my wings

  • 8
    Posted January 17, 2010 at 3:25 pm | Permalink

    Thankyou Dog. I am glad, however, that I’m not the wind beneath your angry underpants :-P

  • 9
    thewetmale
    Posted January 17, 2010 at 3:39 pm | Permalink

    Very nice post Possum. Very nice indeed.

    The only problem is that you STILL haven’t proved global warming to be real! :-P

    Perhaps we need to call Sophie Black and get her to shut your blog down until you actually answer these important and pressing questions.

  • 10
    Posted January 17, 2010 at 3:51 pm | Permalink

    Yes, that’s all well and good but why haven’t you used a fourth order polynomial?

  • 11
    Posted January 17, 2010 at 3:54 pm | Permalink

    Because I was too busy planning to use a 6th degree polynomial!
    http://scienceblogs.com/deltoid/2009/01/the_australians_war_on_science_32.php

    :-D :-D

  • 12
    Posted January 17, 2010 at 4:38 pm | Permalink

    Sixth order? That’s just too powerful a statistical tool for warmists to disagree with. The IPCC may as well hang up its boots when papers start publishing sixth order polynomials. Just as well it wasn’t a 7th or Al Gore would’ve been banished back to the fiery bowels of Tennessee.

  • 13
    Posted January 17, 2010 at 4:42 pm | Permalink

    And not just any paper -The Australian. Comes with extra gravitas you know!

  • 14
    Steve
    Posted January 17, 2010 at 4:53 pm | Permalink

    Captain Col,
    Tamino has addressed the question of how much time is requred before a clear trend in climate atmospheric data can be seen.

    The answer is about 15 years as Tamino shows in the post at the link below:
    http://tamino.wordpress.com/2009/12/15/how-long/#more-2124

  • 15
    cud chewer
    Posted January 17, 2010 at 6:24 pm | Permalink

    Steve @14, thanks, that’s exactly the post I’ve been looking for for a long time.

  • 16
    cud chewer
    Posted January 17, 2010 at 6:25 pm | Permalink

    Possum, it would be fun to go further and show how many crimes can be committed with polynomial fits :)

  • 17
    Posted January 17, 2010 at 6:37 pm | Permalink

    That is a brilliant post. Of course, people will misread the plot of starting time versus estimated warming rate as saying that the temperature has been constant since 1975 and that since 1993 the world has been cooling and that the sharp peak at the end is measurement error.

  • 18
    Steve
    Posted January 17, 2010 at 6:54 pm | Permalink

    Here is a piece by James Hansen and others discussing a range of issues including the topic of this post.
    http://www.columbia.edu/~jeh1/mailings/2010/20100115_Temperature2009.pdf

    Note that according to GISS 2009 was the second warmest year on record (after 2005) but its temperature was so close to 1998, 2002, 2003, 2006, and 2007 that all of them (and 2009) are considered equal second hottest.

  • 19
    cud chewer
    Posted January 17, 2010 at 7:36 pm | Permalink

    For me it makes more sense, when dealing with someone who is honestly confused about the issue, to start with the proposition that the Earth is gaining heat energy, and then to ask the question “where does that energy go?”.

    Only a minority goes into heating the air. A lot goes into heating the oceans. Some goes into melting ice. Some goes into biological or chemical processes. Its pretty easy to explain then that the amount of energy that goes into the air can vary from year to year but the net gain of the whole planet is pretty stable and predictable. Over time though, the air has to get warmer.

    It would be nice if the scientific community were to popularise the concept “radiative balance”, or at least put it into more palatable phrases like “global heating” which emphasises the cause and not (one of) the effects.

    Regarding the saturation of the effects of CO2, this one explains things clearly (although its quite a read). To put it simply, the atmosphere is a lot more complex than a test tube.

    http://www.realclimate.org/index.php/archives/2007/06/a-saturated-gassy-argument/

  • 20
    Dan
    Posted January 17, 2010 at 9:33 pm | Permalink

    Thanks Poss

    That last thread made me so angry

    We love you man (you too firstdog)

    :)

  • 21
    JamesK
    Posted January 18, 2010 at 8:19 am | Permalink

    Awww … toats ma goats!

    Despite reflex diaphragmatic heaving, we love you too wee petulant dan dude.

    You too firstdog.

    Word to the wise little leprechaun…

    Possum may relish his toadies but Tim Lambert…

    Not so much.

  • 22
    CHRISTOPHER DUNNE
    Posted January 18, 2010 at 10:04 am | Permalink

    Hansen concludes his paper on the 2009 GISS series thus:

    The magnitude of monthly temperature anomalies is typically 1.5 to 2 times greater than the magnitude of seasonal anomalies. So it is not yet quite so easy to see global warming if one’s figure of merit is monthly mean temperature. And, of course, daily weather fluctuations are much larger than the impact of the global warming trend. The bottom line is this: there is no global cooling trend. For the time being, until humanity brings its greenhouse gas emissions under control, we can expect each decade to be warmer than the preceding one. Weather fluctuations certainly exceed local temperature changes over the past half century. But the perceptive person should be able to see that climate is warming on decadal time scales.

    This information needs to be combined with the conclusion that global warming of 1‐2°C has enormous implications for humanity. But that discussion is beyond the scope of this note.

    http://www.realclimate.org/index.php/archives/2010/01/2009-temperatures-by-jim-hansen/#more-2743

    (the whole article is an extrapolation of the issues raised in Possum’s piece)

    So in 130 years, 2009 was right up there, tied for second hottest global mean temperature, while in the Southern Hemisphere, it was actually the hottest.

    No doubt la Dolta will sit there on Insiders this year and repeat his claim that it’s been cooling for ten years, because, well, because he’s an ignorant doofus that thinks he knows something about statistics.

    But thanks to Possum, we know he doesn’t have a friggin’ clue.

  • 23
    JamesK
    Posted January 18, 2010 at 10:23 am | Permalink

    No Christopher Dunne.

    The bottom line is this: there is no global warming trend and hasn’t been since the mid-nineties according to Tamas Calderwood.

    There has been a cooling trend but it enters ‘statistical significance’ if you cherrypick a 2002 commencement date for the ‘statistical’ analysis.

    So Bolt is actually correct to say it has cooled but not to say there has been a statistially significant cooling trend.

    He didn’t.

    So he is actually correct.

  • 24
    calyptorhynchus
    Posted January 18, 2010 at 11:12 am | Permalink

    JamesK

    “The bottom line is this: there is no global warming trend and hasn’t been since the mid-nineties according to Tamas Calderwood.”

    Who he?

  • 25
    CHRISTOPHER DUNNE
    Posted January 18, 2010 at 11:16 am | Permalink

    JamesK, I await with excited anticipation, your paper on world climate, and the revelation that it hasn’t been warming.

    I’m sure the scientific community will be reeling from your brilliant analysis.

  • 26
    JamesK
    Posted January 18, 2010 at 11:19 am | Permalink

    Pity Possum closed his post on Bolt/Spencer.

    Feeling very agrieved Spencer had to go to Anthony Watts blog, a lesser option of course to answer Possum’s accusations of malfeasance and/or stupidity:

    http://wattsupwiththat.com/2010/01/17/spencer-hide-the-incline/

    @calyptorhynchus
    See the blog that Dan and Possum went bannanas about.

  • 27
    Astrobleme
    Posted January 18, 2010 at 11:19 am | Permalink

    JamesK

    You make no sense dude. The period from start 2000 to end 2009 was about 0.2 degrees warmer than from 1990-1999.
    All of the 2000s were warmer than every year in the previous decade except 1998, and warmer than any other years since records began.

  • 28
    Astrobleme
    Posted January 18, 2010 at 11:23 am | Permalink

    JamesK

    That graph on WUWT disproves your previous comment about no warming since the mid 90s.

    It also didn’t address what Possum was talking about. The incline he was talking about was the recent one, the red line stops earlier that the green one. His point is that Spencer chose 25 months because that’s when the uptick is less evident.

  • 29
    Astrobleme
    Posted January 18, 2010 at 11:24 am | Permalink

    James K

    Check the AMSU temps for the near surface layer (channel 4)

    http://discover.itsc.uah.edu/amsutemps/

    Oh look, they they’re the hottest ever recorded for January…

  • 30
    pancho
    Posted January 18, 2010 at 11:30 am | Permalink

    “And not just any paper -The Australian. Comes with extra gravitas you know!”

    It’s a conviction paper.

    “according to Tamas Calderwood.”

    Hahahahaha.

  • 31
    Evan Beaver
    Posted January 18, 2010 at 11:31 am | Permalink

    JamesK That WUWT post doesn’t express any reasons for changing from the 13 -25 month smoothing, other than ‘it provides more smoothing’. No reasons why the previous method suddenly became wrong.

  • 32
    Posted January 18, 2010 at 11:38 am | Permalink

    I did some analysis with WoodForTrees a while ago and there is no downward 30-year trend from 1944 onwards, however there are plenty of downward 30-year trends before that. Even given the plateau of the 00s, it’s pretty obvious there will be no decline given the strong upward trend of the 90s and the bumper El Nino indications at the start of this decade.

    Climatologists speak of climate in 30 year trends, right? Anything less than that is weather?

  • 33
    Posted January 18, 2010 at 11:43 am | Permalink

    Here are the bumper 30 year trend graphs, by the by (warning, major graph porn):

    1850 – 1940 1 2 3

    1920 – 2010 1 2 3

  • 34
    Barking
    Posted January 18, 2010 at 11:44 am | Permalink

    Chuck in a probability curve and it again reinforces just how scary the last 10-15 years have been.
    Slight tangent, one more of the many things that the deniers fail to see or address is that much of the scientific work is about the future, not now. Many of the projection relate to our grand children. This is the part of thier fraud that annoys me the most. Its our grandchildren who will piss on the graves of some of these people. I just hope that they have the ability to scroll back to post such as Possum’s to see that their were people who worked and stood up for them.

  • 35
    CHRISTOPHER DUNNE
    Posted January 18, 2010 at 11:51 am | Permalink

    Oh, and JamesK, when you’ve finished “proving” that there’s been no warming, how about knocking over that other scientific fraud: smoking causes cancer.

    Come on, the entire world needs you to show them the truth about these things!

  • 36
    JamesK
    Posted January 18, 2010 at 11:53 am | Permalink

    Can’t you read the graphs that Possum publishes Christopher?

    You need a scientific paper to tell you that there has been no global warming for a decade?

    What about an authority figure to tell you to pull your trousers down before sitting on the Thomas Crapper?

    Okay okay…I’ll help you out:

    http://news.bbc.co.uk/2/hi/science/nature/8299079.stm

    http://rankexploits.com/musings/2009/temperature-anomalies-v-absolute-projections/

    http://theresilientearth.com/?q=content/stat-model-predicts-flat-temperatures-through-2050

    Apparently even Kevin Trenberth IPCC lead scientist 2001 and 2007 can read Roy Spencer’s graphs:
    http://junkscience.com/FOIA/mail/1255523796.txt

    He “can’t account for the lack of warming” apparently. Also apparently it’s a “travesty” that he can’t.

    Note Christopher he is not a denialist like yourself.

    He knows there has been no warming. He doesn’t deny it.

    He’s just upset because his IPCC models “can’t account for it”

  • 37
    PeeBee
    Posted January 18, 2010 at 11:55 am | Permalink

    Barking@34.

    Do you think it is in the denialist make-up to change their mind. I often wonder what it would take but only they can answer that and I think I know their answer – ‘nothing’.

    But as the evidence stacks up against them, they shift ground (see with Spencer and the change in moving average – his explanation was to smooth out the 1998 blip – what rubbish, why did he wait until there was an up-tick on the 13 month moving average to change!).

    Still intellectual honesty in not his strong suit.

    I guess the best we can hope for is that they will acknowledge that the globe is warming, but it will be:

    1. Good for us
    2. It natural
    3. Whatever you can dream up so that you don’t have to do anything to reduce emissions.

    I’m also annoyed by their fraud, because I don’t want to leave a earth that is worse off to my kids/grandkids.

  • 38
    JamesK
    Posted January 18, 2010 at 11:59 am | Permalink

    Oh dear! My posts under permanent possumoderation

  • 39
    Posted January 18, 2010 at 12:00 pm | Permalink

    Too many links

  • 40
    JamesK
    Posted January 18, 2010 at 12:02 pm | Permalink

    Why not PeeBee?

    I want my kids and future generations to have a dreadful time.

    One way to do that is to spend ourselves into poverty I guess…

    Hey! Maybe you’re right……Good thinkin’

  • 41
    Astrobleme
    Posted January 18, 2010 at 12:10 pm | Permalink

    JamesK

    You really don’t understand the point of Possum’s post do you?

    Anyway, your links:
    BBC – load of garbage, yes according the HadCrut 1998 was the warmest, but every year of the 2000s has been warmer than any year in the 90s. The trend is up, the 90′s were about 0.2 C cooler than the 2000s.

    Look at this grpah of HadCrut temps:
    http://www.cru.uea.ac.uk/cru/data/temperature/nhshgl.gif

    Do you understand that global warming doesn’t mean each year will be warmer than the next. it’s about THE TREND… And the trend is up.

  • 42
    PeeBee
    Posted January 18, 2010 at 12:11 pm | Permalink

    JamesK

    you are a moron.

  • 43
    JamesK
    Posted January 18, 2010 at 12:13 pm | Permalink

    What PeeBee?

    You don’t want to discuss the “issues”?

  • 44
    Astrobleme
    Posted January 18, 2010 at 12:14 pm | Permalink

    JamesK

    Lucia’s analysis at the blackboard is meaningless.
    The multi-model mean is not what we should expect, the multi-model mean is not an estimate.

    What we should ‘expect’ is that the climate will lie somewhere within the range exhibite by the models. This is beautifully articluated here:

    http://julesandjames.blogspot.com/2010/01/reliability-of-ipcc-ar4-cmip3-ensemble.html#comments

    I doubt you will understand it, but the target analogies he gives are great. The multi-model mean is not the ‘target’ that the real world needs to exhibit for the models to be ‘correct’ but rather the real world should lie somewhere within the range exhibited by the models.

  • 45
    CHRISTOPHER DUNNE
    Posted January 18, 2010 at 12:14 pm | Permalink

    JamesK, you can believe as many journalists you like, (and la Dolta too if you must), but I’ll take Hansen’s analysis over the lot of them any day. His work, unlike theirs, IS peered reviewed:

    “The global record warm year, in the period of near-global instrumental measurements (since the late 1800s), was 2005. Sometimes it is asserted that 1998 was the warmest year. The origin of this confusion is discussed below. There is a high degree of interannual (year‐to‐year) and decadal variability in both global and hemispheric temperatures. Underlying this variability, however, is a long‐term warming trend that has become strong and persistent over the past three decades. The long‐term trends are more apparent when temperature is averaged over several years.”

    …and he concludes:

    “The year 2005 is 0.061°C warmer than 1998 in our analysis. So how certain are we that 2005 was warmer than 1998? Given the standard deviation of ~0.025°C for the estimated error, we can estimate the probability that 1998 was warmer than 2005 as follows. The chance that 1998 is 0.025°C warmer than our estimated value is about (1 – 0.68)/2 = 0.16. The chance that 2005 is 0.025°C cooler than our estimate is also 0.16. The probability of both of these is ~0.03 (3 percent). Integrating over the tail of the distribution and accounting for the 2005‐1998 temperature difference being 0.61°C alters the estimate in opposite directions. For the moment let us just say that the chance that 1998 is warmer than 2005, given our temperature analysis, is at most no more than about 10 percent. Therefore, we can say with a reasonable degree of confidence that 2005 is the warmest year in the period of instrumental data.”

    …so where’s your data JamesK?

  • 46
    Astrobleme
    Posted January 18, 2010 at 12:17 pm | Permalink

    James K

    David Stockwell is a denialist, heck he was (and probably still is) trumpeting Ferenc Miskovic’s “new cliamet theory” or whatever it was called. What aload of bollocks that was. This is fringe science.

    And Trenberth has already explained what he meant by that comment, but hey you’re interested in facts are you?

  • 47
    PeeBee
    Posted January 18, 2010 at 12:18 pm | Permalink

    CD@45,

    I can see it now Bolt’s headline:

    ITS OFFICAL, THE WORLD HASN’T WARMED SINCE 2005!

  • 48
    Posted January 18, 2010 at 12:19 pm | Permalink

    Here is a clean graph that illustrates The Death of the Downward 30-year Trend

  • 49
    PeeBee
    Posted January 18, 2010 at 12:23 pm | Permalink

    JamesK@43

    I don’t want to discuss issues with someone who declares:

    I want my kids and future generations to have a dreadful time

    But it does explain what motivates you.

  • 50
    Dewgong
    Posted January 18, 2010 at 12:27 pm | Permalink

    If you think taking action on reducing Co2 emissions means “spending ourselves into poverty” James K, then you’ve clearly been lapping up the fearmongering coming from the conspiracy theory lobby without a displaying shred of skeptical thinking about their claims and their own motives.

    Sure, global temperatures have not risen above 1998 levels, but that is irrelevant for many reasons, not least of all the one that is explained in this blog entry.

    If you think you can pick a change in the long term trend from observing data from such a limited time frame, then you can do what almost nobody else in the world can do, and might want to consider taking up share market or forex trading.

    Long term trends only ever become evident after the fact, if in 30 years global temperatures have plummeted, then that’s excellent, we will know we are wrong, but if they continue to rise, well, we’re in a bit of a sticky situation. But in addition to that, there is no reason why the Earth would change into a long term cooling trend. If you think it is going to do so you are going to have to explain why.

Please login below to comment, OR simply register here :



Womens Agenda

loading...

Smart Company

loading...

StartupSmart

loading...

Property Observer

loading...