Big data edged up in interest in 2011, but in 2012 it skyrocketed, potentially changing aspects of data management in a dramatic way. Big data systems spawned changes for managing and handling machine data, continuous extract, transform and load functions, operational business intelligence, big data in motion, cloud-based data warehousing and more.
Still, as big data enters 2013, no big data systems technologies are more active than NoSQL databases and the Hadoop framework, and it appears they have more room to grow. The Hadoop-MapReduce market alone is forecasted to grow at a compound annual growth rate of 58%, reaching $2.2 billion in 2018, according to an August 2012 MarketAnalysis.com report.
Judith Hurwitzindependent analyst
NoSQL and Hadoop appear to be major means for coping with unstructured data such as text and Web logs. Like Apache Hadoop, these technologies often have open source roots, and are still new as commercial products.
According to Judith Hurwitz, president and CEO of Needham, Mass.-based Hurwitz and Associates Inc., big data architecture and massively parallel processing are dramatically transforming the data landscape. "Previously, even though data was really important to companies, they didn't really have the capability of grabbing great amounts of data and analyzing it in real time," Hurwitz said.
"Now, things that were not practical are becoming practical. This has taken data out of its comfort zone," she said.
SQL takes a hit, punches back
As seen in the pages of SearchDataManagment.com, 2012 began with predictions of trouble for mainstay relational databases. The criticism proved partially prophetic. After battling many would-be replacements over the years, the SQL relational database seems to have met serious competition for handling massive amounts of data now -- or soon to be -- filtering through the enterprise.
The push behind the trend is the enterprise's desire to take on more unstructured data at a faster rate in order to become more data-driven in decision making. Customary approaches are being reworked to encompass the best of the new techniques.
Just a few of the many moves by established data management vendors in 2012 show the impact big data and Hadoop are having on the relational data status quo:
- IBM continued to add boutique data and analytics companies to its portfolio, though it happened with less frequency than in 2011. Big Blue's efforts ranged from small enhancements such as a NoSQL Graph Store for DB2 10 and InfoSphere Warehouse 10, to a very large PureData System appliance intended to "tame big data" in the enterprise.
- Oracle rolled out its big data appliance at the start of the year. The announcement was followed up later with Oracle NoSQL Database 2.0, which has automatic rebalancing, new application programming interfaces for handling large objects, and tighter integration with the Oracle database, allowing querying of Oracle NoSQL database records directly from SQL.
- Microsoft showed previews of Hadoop support for Windows Azure and Windows Server; Teradata Corp. released its Aster Big Analytics Appliance; and Informatica Corp. launched a Big Data Edition of its PowerCenter suite that was said to remove the need for Hadoop hand coding by bringing the programming task into the Informatica development environment.
SQL may have taken a punch or two in 2012, but it refused to go down for the count. Companies specializing in the alternative NoSQL and Hadoop side of things brushed up their SQL credentials this year. A prime example was Hadoop startup Cloudera Inc. It looked to enhance its SQL standing with Impala, a Hadoop software offering that supports interactive queries done in standard SQL.
Big data shift
Moves like this may indicate a bit of momentum -- one that sees SQL and NoSQL being mentioned together more often. In a way, SQL was downplayed in the early big data buzz.
"In the last couple of years, because of the big data movement, SQL has not been on everyone's lips," said Ronnie Beggs, vice president of marketing at San Francisco-based SQLstream, a maker of streaming databases. Meanwhile, he continued, "Big data and NoSQL [have] taken off as a topic, and hit the mainstream."
In 2013, we should see evidence of change there, he indicated. There has been of lot of effort in recent years to better enable NoSQL databases for SQL-style development, he said.
"It's simply evolving. What we see for next year is the return of SQL as an interface for all the big data platforms," Beggs said.
Big data defined
Big data is the voluminous amount of structured, unstructured and semi-structured data a company creates -- data that in many cases would take too much time and cost too much money to load into a conventional relational database for analysis.
Read more from the Whatis.com definition of big data
This evolution toward coexistence of the Hadoop framework, NoSQL and SQL approaches could mark a new step in big data's maturation. As 2013 approaches, there is the possibility that big data may move from hot topic to practical reality.
"I think people are trying to get through the hype of big data and really understand where the business value is," said Colin White, president and founder of Ashland, Ore.-based BI Research. "In 2013, I think we will see good examples of people getting business value from big data. It's not about big data, it's what you do with the data that matters."
While there is wide interest in the new technologies, not all companies will move to full-fledged big data systems at the same rate. This was borne out as an integration services manager at a major bank spoke recently with SearchDataManagment.com.
He marked banking as an area where some, but not all, of the basics of big data are in play. Banking and other fields see data that is big in volume, but not big in unstructured data. At least that is the case today.
"When you look at the tenets of big data, there are two parts. One, there is lots of it, and two, it is unstructured. The banks have the first part," he claimed. "But we are not collecting 'tweets,' at least not yet. We are in wait-and-see mode, looking to see how the financial data service market can handle it."