James Thew - Fotolia

SQL-Hadoop duo looks to ease programming in big data apps

An emerging crop of SQL-on-Hadoop query engines are enabling users to pair up the database programming language and big data framework, letting SQL developers query Hadoop data.

Initially, Hadoop was tied to MapReduce, a specialized programming environment for developing batch processing jobs as part of big data applications. But that's changing. In a session at the 2015 Pacific Northwest BI Summit in Grants Pass, Ore., Gartner Inc. analyst Merv Adrian said he expects SQL -- the standard programming language for mainstream relational databases -- to also become the primary analytics interface into Hadoop data stores.

The SQL-Hadoop pairing makes perfect sense. It enables the hordes of SQL developers, as well as other types of users who know their SQL, to write Hadoop queries in a familiar way. Being able to do so could help ease a Hadoop skills shortage that Adrian pointed to as one of the big barriers to broader adoption of the big data processing framework. In a Gartner survey conducted earlier this year, just 10% of the 284 respondents said their organizations were using Hadoop in production applications; more than half -- 54% -- said their companies had no plans to use the technology.

IT vendors are certainly betting on the SQL-Hadoop mix taking root in user organizations. Since 2013, they've introduced a still-growing collection of SQL-on-Hadoop query tools, some as open source technologies. Forrester Research Inc. analyst Mike Gualtieri joked about the proliferation of product choices at the 2015 Hadoop Summit in San Jose, Calif. "Fortunately," he said, "there are at least 13."

Adrian went one better, listing 14 SQL-on-Hadoop platforms on a slide in the presentation he delivered in Oregon -- and even that wasn't a full count of what's available. Most of the tools rolled out thus far haven't fully matured yet, and some may not ever bear much fruit because of the crowded market conditions.

SearchDataManagement has published a series of articles designed to give IT managers and other readers insight into what organizations can do with SQL-on-Hadoop technologies and advice on how to navigate through all the SQL-Hadoop combinations. In one story, we look at SQL-on-Hadoop deployments in two companies. In another, we drill down into the various SQL-on-Hadoop software options and look further into how SQL skills could help boost Hadoop adoption. Consultant Rick van der Lans weighs in with thoughts on choosing between SQL-on-Hadoop tools. And in a Q&A, Tripp Smith, another consultant, provides his thoughts on matching SQL-on-Hadoop engines to the right use case.

Good ol' SQL and ultra-hip Hadoop can make a powerful pair for managing and analyzing big data. But whether the duo turns out to be a true power couple, or more of an odd one, remains to be seen. And we'll be watching.

Craig Stedman is executive editor of SearchDataManagement. Email him at [email protected] and follow us on Twitter: @sDataManagement.

Next Steps

Read about one company's measured approach to building a Hadoop architecture

Don't get dragged under when implementing a Hadoop data lake system

Hadoop's core components are getting more company as vendors add new tools

Dig Deeper on Hadoop framework