Under-estimating the cost of major infrastructure projects and over-estimating the demand is so chronic that forecasters deserve some harsh medicine, according to Professor Bent Flyvbjerg from Oxford University’s Said Business School. He says “some forecasts are so grossly misrepresented that we need to consider not only firing the forecasters but suing them too – perhaps even having a few serve time”.
Australians have plenty of experience with underperforming infrastructure projects. For starters, just in transport alone, there’s Brisbane’s Clem 7 road tunnel, Sydney’s Lane Cove and Cross City tunnels, the Brisbane and Sydney airport trains, Melbourne’s Myki ticketing fiasco, and the 2,250 km Freightlink rail line connecting Adelaide and Darwin. And they’re just the ones we know about!
Professor Flyvbjerg says cost overruns in the order of 50% in real terms are common for major infrastructure projects and overruns above 100% are not uncommon. Writing in the Oxford Review of Economic Policy, he argues that demand and benefit forecasts that are wrong by 20–70% compared with the actual outcome are also common.
Transport projects are among the worst performers (see exhibit). Professor Flyvbjerg examined 258 transport projects in 20 nations over a 70 year time frame. He found the average cost overrun for rail projects is 44.7% measured in constant prices from the build decision. For bridges and tunnels, the equivalent figure is 33.8%, and for roads 20.4%. The difference in cost overrun between the three project types is statistically significant and the size of the standard deviations shown in the first exhibit demonstrate the high degree of uncertainty and risk associated with these sorts of projects.
He also found that nine out of 10 projects have cost overruns; they happen in all nations; they’ve been a constant over the last 70 years; and cost estimates have not improved over time.
And it’s not just under-estimation of costs. Errors in forecasts of travel demand for rail and road infrastructure are also endemic. He found that actual passenger traffic for rail projects is on average 51.4% lower than forecast traffic. He says:
This is equivalent to an average overestimate in rail passenger forecasts of no less than 105.6 per cent. The result is large benefit shortfalls for rail. For roads, actual vehicle traffic is on average 9.5 per cent higher than forecasted traffic. We see that rail passenger forecasts are biased, whereas this is less the case for road traffic forecasts.
He also found that nine out of ten rail projects over-estimate traffic; 84% are wrong by over ±20%; it occurs in all countries studied; and has not improved over time.
Thus the risk associated with rail projects in particular is extraordinary. They face both an average cost overrun of 44.7% and an average traffic shortfall of 51.4%.
Professor Flyvbjerg identifies three possible causes for these errors. One is technical error – this relates to factors like inadequate data and the inherent difficulty of forecasting the future. It is the customary defence when projects fail. Another is psychological error, like optimism bias. Finally, there’re political and economic explanations, where promoters, investors and politicians deliberately under-estimate costs and over-estimate benefits.
Based on the various projects he examined, Professor Flyvbjerg rejects both technical and psychological theories as the primary cause of errors. He points the finger at political explanations – “strategic misrepresentation”, as he describes it. Promoters and forecasters have an incentive to intentionally under-estimate costs and over-estimate benefits to obtain approval and funding for their projects. This results in:
an inverted Darwinism i.e. survival of the unfittest. It is not the best projects that get implemented, but the projects that look best on paper…..Forecasting is mainly another kind of rent-seeking behaviour, resulting in a make-believe world of misrepresentation which makes it extremely difficult to decide which projects deserve undertaking and which do not.
Apart from “having a few forecasters serve time”, the key way to address the problem, he recommends, is via use of ‘reference class forecasting’. In essence, it involves comparing the project against a large sample of similar past projects to determine the range of cost and benefit miscalculations.
Professor Flyvbjerg illustrates this proposal by relating a story about a group of Israeli teachers and academics who were tasked with developing a curriculum for high schools. The team members were each asked to give their estimate of the time the project would take – their estimates ranged from 18 months to 30 months. They then asked a distinguished curriculum expert how long similar projects he’d been involved with had taken. After some consideration he replied that about 40% never finished; of the remainder, he could think of none that completed the task within seven years.
Virginia Postrel inteviewed Professor Flyvbjerg for Bloomberg. He told her when ‘reference class forecasting’ was introduced in the UK it stopped a number of projects dead in their tracks. “This has never happened before”, he told her. However the world’s biggest infrastructure projects, like HSR lines in China, are not subject to such testing. Data on these projects is simply not reliable – “if the party says there’s no cost overrun, there’s no cost overrun”. No wonder promoters look so longingly at China, says Postrel, where infrastructure glamour is the law.
Rail-ridership predictions are especially over-optimistic in the U.S. Postrel reports the average gap between expectations and reality is 60 percent, compared with 23 percent in Europe. Using Professor Flyvbjerg’s findings, her back-of-the-envelope calculation suggests California High-Speed Rail can expect to carry only 15.6 million passengers a year by 2035, rather than the 39 million projected.
Professor Flyvbjerg focuses on transport projects in this paper, but he says the same problems apply to other project types including ICT systems, buildings, aerospace projects, defence, mega-events such as the Olympics and the World Cup, water projects, dams, power plants, oil and gas extraction projects, mining, large-scale manufacturing, big science, and urban and regional development projects.
While things are bad enough with transport infrastructure, they’re even worse with ICT:
if a major project is not already messed up, injecting a good dose of ICT will do the job….As if it were not difficult enough to develop, say, a major new airport, we are now developing airports that depend on major new ICT for their operations, and we pay the price…… if you are doing ICT as part of a major project (or as a major project in itself) be sure to get ICT that has been developed and debugged elsewhere and that has a proven track record in daily use.
Hmmm….I can think of a few governments in Australia that wished they’d taken that advice.