Hadoop may have started life as just another open source project, but over the course of 10 years, it has become bedrock for an important class of distributed data-oriented applications. At the Strata + Hadoop World 2016 conference in San Jose, Calif., SearchDataManagement's Jack Vaughan talked with Hadoop co-creator Doug Cutting to find out more about the big data technology's origins and where it's headed.
Cutting, coming from a background in search software development, helped originate the Hadoop framework while working at Yahoo Inc. circa 2006. In this video, Cutting, now chief architect at Hadoop vendor Cloudera Inc., suggests his search technology background predisposed him to look anew at data processing problems that many at the time would have addressed using relational database technology.
Hadoop grew up at the Apache Software Foundation as a card-carrying member of the open source movement. That open source connection is still important today, Cutting says.
Doug CuttingHadoop co-creator
"I've seen a lot more than I expected," Cutting says. "This project has sort of sparked a revolution in open source software, and people are moving off platforms they have been using for decades and moving to a new suite of open source tools."
Enterprises that can make this transformation enter a new style of development culture, with, in the early going, he admits, some Wild West aspects. But such traits are a hallmark of open source as done in Hadoop style. Cutting emphasizes that there are now a host of alternatives with which data managers can experiment.
So, agility is in order for people going forward with open source data processing frameworks.
"We encourage people not to spend too much time on designing the system up front," Cutting says. "Rather, [they should] take a guess and do incremental improvement. All the tools are designed to support that model."
Watch the seven-minute video from the Strata conference and learn more about Hadoop's roots, as well as its path forward.