Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Cloud and streaming technologies for Hadoop: Part boon, part muddle

Listen to this podcast

The Hadoop ecosystem is both a horn of plenty and a grab bag of data technology. This podcast sorts through some recent news of streaming technologies and Hadoop in the cloud.

The Hadoop ecosystem these days seems a never-ending supply of data technology. Sorting through the variety of Hadoop cloud and streaming technologies is an ever more arduous task for data managers, but there is a lot that can be worthwhile.

What drives interest in streaming is something called "actionable analytics." As today's IT shops get further from weekly and monthly reporting as well as batch processing and get closer to real-time processing that, for example, informs decisions or personalizes sales pitches, they enter a world of streaming technologies that include many moving parts.

Frameworks such as Apache Flink and Kafka Streams are joining an array of diverse streaming technologies that include Spark Streaming and Storm incumbents, according to Craig Stedman, SearchDataManagement's executive editor, who speaks with Jack Vaughan, senior news writer, in this edition of the Talking Data podcast.

That more varied technologies are becoming available is very indicative of big data today. However, plenty of assembly is required to integrate diverse parts, and to obtain business benefits, according to Stedman. Skills for such assembly can be in short supply.

"IT teams looking to build architecture have an abundance of different technologies to choose from," he said. The downside is that putting together the pieces you need tends to become a daunting task.

Meanwhile, trends at the Strata + Hadoop World conference in New York last month disclosed that Hadoop vendors are hearing footsteps in the cloud -- where, by some measures, Amazon's Elastic MapReduce software leads all other Hadoop vendors in terms of actual numbers of Hadoop users.

At Strata + Hadoop World, Cloudera described a move to usage-metering for its Cloudera Director cluster management software in the Amazon Web Services (AWS) cloud. For its part, Hadoop vendor Hortonworks discussed a technical preview that lets AWS users spin-up its Hadoop distribution, Spark and Hive on-demand. The broader use of "pay-as-you-go" cloud schemes for Hadoop could represent a sea change for the technology and its proponents. For more, tune into this edition of Talking Data.

Next Steps

Find out how Strata + Hadoop World showed ways to Hadoop cloud deployment

Learn how streaming technologies vied at Strata + Hadoop World 2016

Be there at Strata + Hadoop World in 2014 as Hadoop jumps the chasm