By John Fielding Jr, Analytics Project Manager
I’ve been in IT for 35+ years and fooling with this thing we call ‘Big Data’ knowingly for 14 years, and unknowingly for five years before that. For much of that time, my colleagues and I have been trying to uncover the proverbial “next big thing” in big data.
As it turns out, everybody (except maybe the guys wearing red shirts in a Star Trek landing party) has been looking for the same thing. A web search shows these predictions made just in the last year: Prescriptive Analytics, People Analytics, Small Data Analysis, Plug-N-Play Analytics, SAAS Analytics Marketplaces, Fast Clusters, Knowledge Clouds, Video Analytics, Quantified Self and Vehicle-to-Vehicle Connections. That’s a new one every few months. (I really like Prescriptive Analytics. One project was designed to help farmers look at weather data to decide when to buy crop insurance…I’ll be really impressed when they can tell me the perfect time to buy life insurance.)
Anyway, I figured it’s time to put in my two cents using my own direct experience to predict the future.
In the decade that I toiled at the Department of Education, the amount and variety of data we collected exploded. Despite all the aggregating and calculating we did, what was it that generated the most publicity? School letter grades. We had the most complicated algorithms for awarding scholarships, but everybody was more interested in hearing that Smallville Elementary got slammed with a D+.
In the years I spent in manufacturing, we collected a deep sludge pit of data, but what did the guys who didn’t come to work in shirts with their names on it want to see first? Give me my margins!
I spent eight years with a company that sold mortgage servicing software. We gathered enough data to make Fannie Mae’s fanny look fat in any dress. And what did the heads atop the totem pole want to see? Credit risk. Period.
In health care, we collect an industrial-size bedpan of information daily. But what is it that the guys who are paying for the analytics want to see? How they can keep people from going to the ER for a sinus headache.
Okay … so what’s the trend here? What does my data show? That despite the myriad of data being shoveled into our data warehouse, we have to simplify our results. The more simplified, the better.
Simplicity is the ultimate sophistication. –Leonardo Da Vinci
So, what are the barriers to simplification?
Facebook has 700,000+ updates per minute. Twitter generates 12 terabytes of data daily. Six hundred videos are posted to YouTube every minute. Ten billion things are connected to the Internet. In the face of all that data, a National Assessment of Adult Literacy study showed that only 13 percent of adult Americans are at or above the proficient level in prose, document and quantitative literacy, and 22 percent of Americans are below basic.
The most significant barrier to the simplification of big data is the basic inability of its consumers to fundamentally understand its power to effect change. And what kind of IT guy would I be if I didn’t propose that IT must take the lead in educating the C-suite suits?
How? Find out exactly what the guys at the top of the food chain want. Resist adding complicated reports to production environments. Push simplification at every turn, not just reporting. Locate your ad man and keep him close … locate your nemesis and keep him closer. Use plain English, symbols and icons. Testing results of what works and what doesn’t can differ between audiences … stay in your environment. Promote the good reports. Design comparative reports so that users can quickly and easily see top performers. Gather usage statistics on report suites and USE THEM. In the finest tradition of developers everywhere, explore other websites and reports and make the best aspects your own.
Retraining the pointy-haired bosses of the world will take time. We can only hope we succeed before whatever’s going to replace Big Data comes along. And rest assured – that “whatever” will come along. Maybe we should get cracking on what that might be!