The basic premise of many science-fiction stories goes something like this: Humankind forges a new technological advance, and then things go haywire as said technology outpaces humankind's ability to manage it. It's hard to say whether big data will ever get the big-screen sci-fi treatment, but some smart people are asking if business decision making can keep up with big data advances.
Let's not paint big data too darkly. It brings hope -- not just of making money, but of improving social institutions, helping to cure disease and more. However, hope can be accompanied by fear. While privacy is the chief public concern in the new world of voluminous information, there are others, as well. The chance that bad decisions might be made based on misreading big data is one of them.
The potential for poor decision making was the topic at hand during a panel discussion at last week's MIT Sloan CIO Symposium in Cambridge, Mass. Panel leader and MIT professor Erik Brynjolfsson said many businesses are used to a decision process that is fueled by hunches and gut instincts. He and others said new ways of thinking about data and decision making are in order for organizations with big data environments. Executives need to rely on "facts and data," not their guts, Brynjolfsson advised.
For me, the talk at the MIT event brought to mind comments made by author and blogger Nate Silver earlier in the year at Gartner's MDM Summit. Data managers, business leaders and anyone in-between can gain a better view of what underlays data-driven decision making by listening to Silver. Something of a prodigy, he dropped a job as a KPMG consultant years ago to pursue an interest in baseball statistics, which he has parlayed into a burgeoning career as chief national data pundit.
Our brains are still cavemen brains in many respects.
Caveman as data scientist
At the Gartner conference, Silver raised a different red flag than the one waved by the MIT panel: He warned attendees to develop a sense of when they're being overaggressive in making decisions based on gathered data. The human mind remains attuned to the same impulses that influenced the thinking of our hunter ancestors, who had to pounce or flee quickly when prey or predators were near, Silver said. "Our brains are still cavemen brains in many respects," he said.
Understanding that we come to the decision-making process with some embedded bias -- "a point of view" -- is also crucial to making good choices, he said. That may just be commonsense, but commonsense is sometimes missing in the buildup to one catastrophe or another.
Sometimes probability theory can seem a long way from commonsense. Yet it is fundamental to forecasting in areas ranging from poker to finance, marketing and weather. Beginning to think in terms of the probabilities of different outcomes is another important element in avoiding big mistakes with big data, Silver said.
As in his book, The Signal and The Noise, Silver noted the National Weather Service as an example of a steadily improving forecaster. "They know they can be wrong, so they make progress at the margins," he said, pointing to a future in which decisions are made along the probabilistic lines of weather forecasts.
Infrastructure for better business decision making
To get another view on business decision making in the big data era, I checked in with James Taylor, CEO and principal consultant at Decision Management Solutions in Palo Alto, Calif. Long ago, his work in business process management and related areas led him to pursue decision management as a principal area of interest.
For more on
Learn how expanded Hadoop use cases are driving new enterprise features
What's behind the Actian-ParAccel deal, Pivotal plan?
Track Hadoop as it moves up from the sandbox
Listening to Taylor, you get the impression that too many companies spend money on big data analytics initiatives with ephemeral hopes that better decisions will automatically follow -- by osmosis, perhaps. "They say they spend money on big data and analytics to improve decision making, but if you ask which decisions they have improved you find they are remarkably imprecise," he said.
Leaving room for predictive analytics in the decision process is an important step, Taylor said. But, he added, companies will also want to look at other tactics. That might mean moving decision making closer to the front lines where the data is more familiar, or automating some big data decisions as part of operational systems.
Data management as a profession has always tried to stand for quality of data. Bringing together high-quality data isn't easy, and it won't get any easier as the data gets bigger. While "chief data officer" may be an overinflated title, there's a chance the data professional who knows all the strengths and weaknesses of a company's data will have a more important role going forward -- one that helps keep big data projects safely out of the sci-fi realm.
Jack Vaughan is SearchDataManagement.com's news and site editor. Email him at firstname.lastname@example.org.