Nerdy Sunday

Today we’ll have a bit of a squiz at how LOESS regressions (also known as local regressions and locally weighted polynomial regressions) are particularly spiffy for analysing polling behaviour over time. But before we get into that, we’ll have a look at our Intrade/Gallup Daily Tracker of the US Election showing the Obama surge continuing in both the polling and the Intrade markets:

Also worth noting is a Galaxy poll in the Sunday papers. Only having a sample of 400 and not running any voting intention questions – the poll belongs in the Sunday papers. View at your own risk.

Loess regressions work a little differently to ordinary run of the mill least squares regressions in that for each data point in the series we are analysing, we fit a locally weighted polynomial regression where we use only a small number of observations in the neighbourhood of the point to fit the regression model for that point.

We also use a weighting system in that the further away the observations are from the data point we are analysing, the smaller the weight they are given in the regression.

This let’s us get very non-linear looking patterns of best fit for dense data sets, where we don’t need to tell the regression that polling behaviour has changed or why, as the Loess regression simply “takes the weight” of the change in polling results and lets us know by approximately how much any given change was from it’s recent historical results, as well as when it occurred.

The other beauty of loess regression is that it can be as sensitive to individual polling results as we want it to be, simply by changing the specifications of the model. The more sensitive we make it, the more volatility we get – the less sensitive we make it, the less effect any given poll will have on the regression, including outliers and rogue polls. In one respect, a loess regression is like a very, very sophisticated moving average.

It’s probably best to explain visually, as the biggest drawback of a loess regression is the complexity in trying to describe any loess regression accurately in mathematical terms. They end up being very large and cumbersome equations and not much different from unintelligible sludge.

If we look at the Newspoll estimates of the ALP Primary vote since 1987 and mark every polling estimate as a simple data point we end up with:

It’s not hard to see some patterns there, but the patterns are of a type that can’t readily be explained by a simple linear, quadratic, cubic or some N degree polynomial trend.

So instead, we can run a few loess regressions through this to get some lines of best fit. First up we’ll set the Degree of the regression to 1, meaning that we effectively run large numbers of linear regressions through subsamples of the data set, producing a whole heap of straight line trends that change over time, and we then join them all up to produce a loess regression line of best fit.

The other setting we need to specify here is the Bandwidth Span which is the size of the subsample used at any given time to compute all those linear lines of best fit. The smaller the Bandwidth Span, the more sensitive the line will be to short term polling changes (because the smaller the subsample size). To highlight the difference the bandwidth size makes, we’ll run three loess regressions with spans of 0.15, 0.3 and 0.45. (just click on these charts to expand)

As we can see, the smaller the span, the more sensitive the regression model becomes. Also note that the regression lines at the very end of the chart didn’t, at these bandwidth spans, pick up The Narrowing in the last handful of Newspolls before the 2007 election.

So let’s run another three regressions using even smaller bandwidth spans of 0.025, 0.05 and 0.075:

At these smaller bandwidths, the three models are far more sensitive to individual polling results or clusters of polling results and did manage to pick up The Narrowing effect. The smallest span of 0.025 is probably too small here, confusing random error and poll to poll volatility for mini trends.

Yet Bandwidth Span is not the only tool we have available to ramp up or reduce the sensitivity of our regression, we can also use the Degree setting. The two charts above used a degree of 1, meaning that it was a linear trend calculated in each subsample. However if we increase that to a Degree of 3 where we will be calculating a cubic trend for each subsample and run those regressions again, we get:

That last chart is highly sensitive, probably too sensitive once the data starts becoming volatile as it was in the first 200 polling observations, whereas the Bandwidth pans of 0.3 and 0.45 in the first chart were not sensitive enough. However that span of 0.15 with a Degree of 3 looks pretty good. It picks up the big trends and polling changes, but still ignores the harshness of week to week and month to month polling volatility.

The big problems with Loess regressions in practical terms are that that they’re half art, half science. We need to specify the degree of the polynomial and the bandwidth we use based not only on what we are trying to determine, but also on what we know about the underlying data generation process – which in this case is the political behaviour and events of the time.

But it’s flexible in that it allows us to change the sensitivity of the regression to find turning points in the data series – we just have to be careful when we do it, and be transparent about it as well.

On the Pollytrack page, we keep a Loess regression on a series containing the results of all the major national pollster results.

The Degree and Span options are fairly dependent on data density where the larger the number of observations we have, the finer we can tune the regressions. With our All Polls loess regression, we don’t yet have the number of observations we need to run highly sensitive regressions (with higher polynomial Degrees and lower bandwidth spans), although we certainly will by the time the next election comes. So for now, we are using a Degree of 1 and a span of 0.3 for these charts, which gives us what we might call a ‘long view’ of the polling data, showing us longer terms trends as they manifest out. As we approach the next election, we’ll ramp up the sensitivity to measure all sorts of things with the polling once our polling observations increase in number.

It’s probably also worth mentioning that the Daily Tracker which Five Thirty Eight uses is a loess regression, so for any regular readers here of what to my mind is the best US polling analysis site – I hope this exploration of loess regressions gives you a better handle on what it is they do over there.

Finally – tomorrow will be a big polling day.