When analysts are no longer enough.

There’s a cracker of an article titled Rise of the Data Scientist over on what is probably my favourite geek site on all of the internets – Flowing Data (which you really should check out, at least for the visualisations). It links into another article from Dataspora called The Three Sexy Skills of Data Geeks, both of which touch on the argument that conventional data analysis is no longer enough in these crazy data driven times. It tackles an issue that I find myself wrestling with here nearly everyday – how to bridge the gap between data analysis and public understanding of that analysis through data visualisation.

Until recently, an effective data analyst was one that simply knew their way around the theory and mechanics of statistics, had a good grasp of statistical software and where the standout analysts also had a really solid understanding of the practical knowledge of the topic that the data actually came from.

After completing the analysis and publishing it somewhere – usually obscure – the most common way the results of the analysis came to the attention of the wider public was through journalists in the mainstream media – or recently in new media.

But that sort of two stage process is becoming redundant as the world we live in becomes more and more saturated with information – data by any other name.

As public demand continues to grow for what was only recently very specialised fields, those attempting to meet that demand, attempting to bring more sophisticated analysis to the public sphere are facing a quandary – the way we used to do things is simply no longer good enough.

Being solid in statistics is now only part of the story – to be an effective analyst we also need to be designers, story tellers, programmers and knowledgeable in the fields we’re looking at. We have to design ways to transmit the often complicated results of our data analysis into something that is easily understood by folks that aren’t statisticians, in ways that can engage them and satisfy their curiosity. We effectively have to become the analyst, the publisher, the journalist and the web developer all rolled into one.

Unfortunately, there isn’t a university course around that accommodates what will become the bread and butter of the pointy end of new media over the next few years. Ideally, such a course would not only have a heavy quantitative empirical component – stats 100,200 and 300 units etc, but computer science components that cover some combination of Python, SQL, flash, javascript etc. Some design elements would also be useful, not to mention units from journalism as well as some major elective area of study on top of that – like economics for instance, or political science or some other area of specialist knowledge that provides the intellectual underpinnings for the topic that one wishes to analyse data from.

Yet, it’s not only in areas of new media where this is important. The general case of data analytics has grown way beyond delivering a 50 page report filled with graphical fluff, hosting a half dozen meetings to explain the results to the client and using some sleep inducing Power Point presentations as a prop.

Companies are demanding more for an X thousand dollar outlay on this stuff – but they are struggling to find people that can deliver it under the one roof. There’d be folks here that read this blog that have commissioned analytical work for their organisations (often from the usual suspects!) – but when was the last time that a piece of analytical work came in and contained a Dashboard, with drill downs, with what if scenarios, with results in a digital format that you could actually use in an ongoing basis along with the standard reports and metrics and analysis– and all of it explained in a way that is meaningful? When was the last time that you got a DVD full of useful spiffy toys designed specifically for your business and business data plus interactive walkthroughs of the results of the analysis rather than just a power point presentation?

There’s a few firms around that can provide that suite – for the cost of an arm and a leg and a first born child. But smaller consultants that can’t claim to have originated in Boston have the technical resources available now to produce such things, but the training is simply lacking. Technology advanced faster than our university education system could accommodate on this front – but tacking a few extras onto the end of a stats degree isn’t really an answer.

We need to approach this with a fresh perspective if Australia is to keep our reputation of producing world class analysts. That might mean we have to increase the range and types of statistics courses we offer – but what we are producing at the moment isnt tailored for the world we actually end up operating in.

The next generation of analysts need to be more than just data pigs – we need expand our understanding of what a data analyst is and what they are now used for, and give them the skills they need – skills which would already be in deployment throughout our economy if they existed.

The lowest hanging productivity fruit for any business and for any economy is understanding what is actually happening. You cannot manage what you do not know.

The one thing the internet does is provide an enormous capacity to know – what is needed to change that capacity into productivity is data management, analysis and interpretation. It’s why we need to develop a new class of data analyst.

On something completely different, via Dogma in comments, comes Hansard Possum – isn’t he a cute smelly thing!

If you want to waste some time, I found myself playing with this bit of spiffiness for about half an hour this morning. I still have no idea why – warning, may be addictive!