The media can smell buzz like sharks can smell blood. Big data is a case in point. As a buzzword, big data is lamentable -- it plays to a penchant for frantic headlines that shout "big, big, big." In a way, the term is a placeholder. It stands in place of something that is very real: the massive amounts of data -- structured, semi-structured and unstructured -- being generated in an increasingly digital world, especially by Web applications. However, all the buzz about big data shouldn't obscure the real shift in data management technology that is underway today.
In recent years, a wide assortment of technologies have arisen that handle data differently than in the past. People have had free rein to experiment with these new tools in ways that aren't economically taxing to their organizations. In fact, Hadoop file systems, graph databases and other types of NoSQL software, and in-memory caches and databases are all seeing greater use because they're often built to run on commodity hardware and to exploit ever-cheaper memory. It also hasn't hurt, of course, that many of those technologies are open source.
And in many companies, their time has come. The changes driven by the increasing interest in big data analytics have been highly disruptive to the data management status quo, as was seen during a panel discussion at the IDC Directions 2013 conference in Boston last month. The analysts on hand from market research firm IDC made it clear that the growing volume, variety and velocity of data are pushing data management leaders to rethink their data architectures.
Buzz should not obscure the real shift in data management technology that is underway.
It has all created a rich palette of technologies for managing data, and, at IDC Directions 2013, IDC Research Vice President Carl Olofson said that palette will only continue to expand. Increasingly, he sees a blend of the available options being put to use, pointing to the incorporation of SQL elements into NoSQL approaches, and the growing tendency of companies to mix and match a variety of data management technologies. It's a big change from the days when the relational database solely ruled the data management roost.
"Special instances seem to require new approaches. Not all problems can be solved by a single model of data [architecture]," Olofson said. "We may see some people saying, 'I am going to use this database engine for this problem; this database engine for that problem,' and so on." Even XML databases -- somewhat forgotten in the general drone of the big data buzz -- may come into further use, he said.
The late Edgar F. "Ted" Codd, the inventor of the relational model for database management, might be surprised by the rampant combining of different technologies. "Relational [databases] will expand to the point that Ted Codd wouldn't recognize them," Olofson said.
More on data management technology trends
Read another view on the big data market
Learn about how Microsoft's SQL Server database took the big data plunge
Find out about the role of the data scientist
Codd likely wouldn't recognize modern hardware infrastructures, either. Even though the systems needed to support advanced data analytics are much less expensive on a per-MIPS basis than previous hardware offerings were, the build-out of hardware architectures is expected to be massive.
IDC estimated that services and software comprised 38% and 24% of an $8.1 billion big data technology and services market in 2012, with infrastructure accounting for the remaining 38%. Looking out to 2016, when it expects the market to total $23.8 billion, services and software are forecast at 29% and 25%, with the infrastructure component comprising 46% of the spending.
Relational databases took care of most needs of the organization for many years. Sometimes they grew into fairly expensive propositions, but they came to be seen as a cost of doing business. Going forward, many business applications will still be "painted relational." But, perhaps, just as many will use the relational bits in combination with colors from the new data management technology palette.
Jack Vaughan is SearchDataManagement.com's news and site editor. Email him at firstname.lastname@example.org.
Follow SearchDataManagement.com on Twitter: @sDataManagement.