Sergey Nivens - Fotolia
- Ron Karjian, Managing Editor
In psychology, the term gestalt describes the search for meaningful perceptions in a chaotic world -- finding the one reality that explains the whole. The same principle applies for data managers trawling to find meaning in overflowing lakes of undefined, unstructured data.
Businesses pour millions of dollars each year into purchasing and developing all variations of hardware and software to collect and analyze data from multiple sources. Depending on whom you ask and what survey you read, value gets mixed reviews from the industry's foot soldiers. Data scientists, business executives, analytics users, industry consultants and research analysts believe those x-bytes of collected data biding time and doing the backstroke in data lakes have plenty of value or very little value.
Most companies capture only a fraction of the potential value from data and analytics, a 2016 McKinsey Global Institute report concluded. The biggest barriers companies confront in extracting value are organizational, and many struggle to incorporate data-driven insights into their business processes, according to the report.
In determining the value of big data, business executives in a 2017 NewVantage Partners survey cited cultural impediments such as insufficient organizational alignment, lack of middle management adoption and understanding, business and technology resistance, lack of a coherent data strategy, the inability to create a shared vision, and lack of data governance policies and practices. While more than 85% of respondents said their firms have started programs to create data-driven cultures, only 37% acknowledged success thus far. Yet more than 80% of respondents reported that their big data investments have been "successful," and nearly 50% believe their companies can actually measure the benefits of their projects. Contradictions abound even in the same survey.
There's value in chaos
All of which provides little comfort for big data managers accountable to penny-wise, ROI-conscious CEOs and charged with the difficult task of measuring the value of their own data. So, what exactly is value, and how is it measured?
"That's the riddle everyone is trying to answer these days, especially with so much investment going into analytics," analyst and consultant Joe McKendrick said. "The value needs to be seen at two levels. At the organizational level, data analytics needs to be of material value to the business. At the next level, there needs to be value that helps enhance the productivity of teams and employees."
Some collected data may appear to have more value than other data, and the temptation may be to exile or even discard data that's deemed useless. That's where it's important to start with the premise that all data, regardless of where conceived, is created equal and therefore must be treated equally.
"[T]here's a lot of data that could flow into a data lake that no one would have any conceivable use for," McKendrick reasoned. "Then, five years from now, lo and behold, an innovation comes to the fore that scoops up that data for some unforeseen purpose."
Chew on this
October being World Series month, I think back to my younger days and collecting five-packs of statistics-laden Dubble Bubble Gum baseball cards. Who would have known then that a Mickey Mantle rookie card worth a penny in 1952 and lying dormant in someone's attic for decades is today valued at more than a million dollars? There was no way of knowing the true value back then, of course, but that hasn't stopped this old lamenter from kicking himself for discarding those shoeboxes full of baseball cards. It seems here that value is in the eye of the beholder and what the market demands.
On a different and much larger scale, there's a basic lesson to be learned in determining the value of big data. As data becomes increasingly commoditized, the McKinsey report noted, value will be determined by the quality, not the quantity, of data. And the greatest value may be found in scarce data, uniquely aggregated data and data that yields superior analytics.
But it's not an either-or situation. Quality data can only be found by combing a lake that's rich in quantity.
As global competition increases, data-driven businesses have to apply more reliable methods to measure real or potential value through data management tools, techniques and practices. There's too much at stake to leave that determination to guesswork, gut feeling, wishful thinking or old measuring sticks. It's ironic that companies immersed in collecting, prepping and analyzing big data to disrupt the marketplace and gain a competitive edge, in the end, struggle to accurately measure the true value of that data. Determining the value can indeed reap immense business rewards.
In baseball terms, today's card of an obscure rookie center fielder may someday be worth a million. Data managers need to keep that in mind as they stand at the edge of the lake and do their daily "gestalting."
Don't put off data governance procedures
How big data analytics delivers value
Big data architectures face big hurdles
Dig Deeper on Big data management
Analytics keys Golden State Warriors' wins, on court and off
Baseball's Twins deploy Databricks to improve analytics power
Discovering the value of IoT with data lakes
What data lake governance challenges do organizations face?