Essential Guide
Managing Hadoop projects: What you need to know to succeed
Learn valuable information about the Hadoop ecosystem and framework in this guide -- its capabilities, its limitations and its place in a big data management and analytics architecture.
Introduction
Companies that need to process large and varied data sets frequently look to Apache Hadoop as a potential tool because it offers the ability to process, store and manage huge amounts of structured, unstructured and semi-structured data. The open source Hadoop framework is built on top of a distributed file system and a cluster architecture that enable it to rapidly ingest and process data for use in analytics applications; Hadoop clusters can also be easily scaled up by adding compute nodes based on commodity servers. But Hadoop isn't a cure-all system for big data application needs as a whole. And while big-name Internet companies like Yahoo, Facebook, Twitter and eBay are prominent users of the technology, and other leading-edge users are also taking advantage of it, Hadoop projects are new undertakings for many organizations.
Some industry analysts assert that Hadoop is still in its adolescent stages, a technology that requires more maturity and functionality before it's fully enterprise-ready. The release of the second-generation Hadoop 2 software in October 2013 added broader application support and features designed to improve cluster availability and scalability. Even so, Hadoop adoption remains relatively low. For example, only 10% of the 284 respondents to a 2015 Gartner survey said their organizations were using it in production applications; another 16% said they were running pilot projects or experimenting with Hadoop, but 54% had no plans to use the technology.
Although Hadoop is open source software, it's by no means free. Companies implementing a Hadoop cluster generally choose one of the commercial distributions of the big data framework, which poses maintenance and support costs. Typically, it also must be used in coordination with a range of complementary technologies from what is referred to as the Hadoop ecosystem. As a result, prospective users have to hire experienced programmers or train existing employees on working not only with Hadoop, but also with MapReduce and related technologies such as Hive, HBase, Spark and Pig.
For many people, big data deployments and Hadoop projects are one and the same. That isn't the case, but Hadoop does have a central role to play in many big data management and analytics initiatives. Learn more about the Hadoop framework in this guide, which offers different perspectives on Hadoop's capabilities and looks at the technology's ongoing development, how it can benefit users and where it doesn't fully measure up to IT needs.
1Understanding and using Hadoop-
Elucidating benefits, myths and facts about Hadoop
Before deciding to implement the Hadoop framework as a tool for managing and analyzing big data, IT decision makers should understand exactly what Hadoop is and how it works. In the articles in this section, experienced users and industry analysts discuss the potential benefits of Hadoop projects, dispel myths surrounding the technology and explore how using Hadoop clusters can generate a return on investment for organizations.
Walmart taps Hadoop to power new data-driven applications
New big data applications running in Hadoop systems are helping to fuel Walmart's quest for more revenue by mixing information from retail stores with online e-commerce data. Continue Reading
Hadoop cluster architecture requires focused approach for ROI
The manager of the Hadoop platform at RelayHealth advises others to stay focused on what will result in a return on investment when implementing and expanding Hadoop clusters. Continue Reading
Develop a plan before plunging into a Hadoop data lake
Depositing all your data in a Hadoop-based repository for analytics uses is an alluring option. But building a data lake isn't that easy, cautions consultant Wayne Eckerson. Continue Reading
How Western Union enabled Hadoop use in its operations
A Hadoop implementation isn't just about deploying a cluster and processing data. Find out what else Western Union had to do to absorb Hadoop into its operational processes. Continue Reading
Deploying Hadoop clusters is new territory for many IT teams
Hadoop projects can present managers with unfamiliar challenges -- particularly because many organizations still lack experience with the framework. Continue Reading
Ancestry.com groups team up on Hadoop system deployment
Data scientists and software engineers at genealogy information provider Ancestry.com worked together to employ Hadoop to support a DNA matching application. Continue Reading
Security services provider uses Hadoop, HBase to ease data flood
Solutionary Inc. is using a system based on Hadoop and its companion HBase database to help manage log data that it combs through in an effort to detect network security threats. Continue Reading
In enterprise apps, Hadoop needs to play nice with existing processes
In a panel discussion at the Hadoop Summit 2013, several enterprise users offered advice on putting Hadoop systems into action in real business applications. Continue Reading
Hadoop projects: Bringing big data into data warehouse environments
Hadoop helps IT teams looking to efficiently mix pools of big data with the information stored in enterprise data warehouses. Continue Reading
Twelve common Hadoop myths debunked
The Data Warehousing Institute's Philip Russom deconstructs a dozen misconceptions about Hadoop and gives his take on the realities of deploying and managing it. Continue Reading
What's the interest in Hadoop all about?
Consultant Tom Nolle examines the real reasons why so many people are interested in Hadoop -- and what's required to successfully implement the technology. Continue Reading
2Hadoop's ongoing evolution-
Keeping up with Hadoop news and trends
As with other technologies, Hadoop is continually evolving to meet shifting big data management needs and business goals. The articles in this section catalog Hadoop technology trends, offering a look at new functionality, expanding applications and supporting tools in the Hadoop ecosystem.
The Hadoop vs. Spark debate: Companion or competitor?
Apache Spark can be used both in big data applications and as a standalone service. We asked IT professionals how they think this will affect Hadoop. Continue Reading
Hadoop's original core no longer mandatory cluster components
Hadoop vendors are adding technologies that look beyond the Hadoop Distributed File System for some uses, enabling clusters to be built without either HDFS or MapReduce. Continue Reading
SQL query engines open up path to broader Hadoop adoption
Hadoop is still at the proof-of-concept stage in many organizations. But some are using new SQL-on-Hadoop tools designed to simplify use of the framework through SQL querying. Continue Reading
Users find that mixing Hadoop, other frameworks packs a punch
Hadoop may be in the spotlight, but some IT professionals say that mixing the distributed processing platform with other big data frameworks is what's really needed. Continue Reading
'A lot of maturity' needed to enable wider use of Hadoop
Hadoop usage will continue to grow, predicts consultant Joe Caserta. But first, he says, the technology and related tools need to address some maturity challenges. Continue Reading
Hadoop 2 expands potential applications -- and issues to weigh
The first version of Hadoop was primarily limited to running MapReduce batch-processing jobs. Hadoop 2 supports more applications, but users still face deployment challenges. Continue Reading
Role eyed for Hadoop in modernization, migration of mainframe apps
Hadoop systems are provoking changes in traditional data warehouse environments. And they might also alter the status quo for mainframe modernization and migration efforts. Continue Reading
Searching for the 'unknown unknowns' with Hadoop and Lucene
Hadoop and the open Lucene search engine are increasingly being paired up by software vendors in an effort to improve users' ability to search for information in pools of big data. Continue Reading
Storm-YARN pairing points to a Hadoop development schism
Yahoo's combination of Hadoop's YARN resource manager and the Storm event processor highlights the gap between enterprise needs and those of large Internet companies. Continue Reading
What's driving the rising importance of Hadoop management tools
The need for tools that can help manage Hadoop clusters is increasing as more users move beyond experiments and deploy the open source framework in real applications. Continue Reading
New enterprise features needed as Hadoop use expands
Hadoop vendors are trying to clear the way for increased adoption of the technology by offering add-ons that target issues keeping some corporate users from moving forward. Continue Reading
How many Hadoop distributions does the world need?
EMC and Intel entered the Hadoop ring by releasing distributions of the software in 2013, increasing the number of choices for prospective users to contend with. Continue Reading
Implementing Hadoop: Storage considerations
Get advice on the issues to take into account in deciding which types of storage systems to use as the primary storage layer for Hadoop data. Continue Reading
Hadoop tools taking on higher-level tasks
It's one thing to build a Hadoop cluster, but making it into a useful system for big data management and analytics requires a greater investment of time and resources. Continue Reading
3Hadoop issues and shortcomings-
Examining issues and weaknesses in the Hadoop ecosystem
While many users find Hadoop projects to be cost-effective and useful, they have some drawbacks to keep in mind in assessing whether it's the right technology for an organization. In this section, users and analysts discuss where Hadoop falls short, particularly in terms of real costs, ease of management, performance and overall capability, and offer advice on how to avoid problems on deployments.
Game on, thanks to new Hadoop cluster management software
Mobile gaming company Chartboost needed a way to better optimize its processing workloads. Software from startup Pepperdata proved to be the answer. Continue Reading
Don't get dragged under by a Hadoop data lake project
Read about the lessons that several IT managers learned while deploying Hadoop-based data lakes, including the need for new skills and IT and data management processes. Continue Reading
To-do list for data lakes: Get more efficient, not so 'messy'
Forrester analyst Mike Gualtieri says Hadoop data lakes could become a viable alternative to traditional data warehouses -- if I/O and data governance improvements materialize. Continue Reading
Diving into a data lake? Don't forget about Hadoop security
Hadoop isn't a single technology stack from a single vendor -- which makes it more difficult to develop security capabilities. But some help is on the horizon. Continue Reading
Analyze the true cost of Hadoop before buying into it
Sure, Hadoop is an open source framework -- but what will it really cost you? Consultant Richard Winter advises looking at specific application types when considering Hadoop. Continue Reading
Hadoop still not up to handling real-time analytics?
Software vendors are adding query engines that run on top of Hadoop in an effort to turn it into a real-time data analysis platform. But some roadblocks remain. Continue Reading
Big data without bottlenecks: Avoiding Hadoop throughput snafus
While various issues can bog down the performance of Hadoop systems, there are ways to steer clear of the pitfalls and ensure that your big data applications keep cruising along. Continue Reading
When Hadoop is the right technology to use -- and when it's not
There's no shortage of hoopla about Hadoop, but it isn't the answer to all big data application needs. Smart companies need to make sure it's a good match for their requirements. Continue Reading
Facing up to issues with MapReduce and Hadoop
Big data users can't wish away the challenges of deploying systems based on Hadoop and MapReduce. But taking some good first steps helps minimize the difficulties. Continue Reading
Hadoop data in motion adds challenges for operational BI uses
Developers and data architects building operational business intelligence applications may need to create fast messaging infrastructures to handle streams of Hadoop data. Continue Reading
Analytics users find pros and cons with Hadoop
There are potential advantages to using Hadoop in analytics applications, but it also can pose some hardships that prospective users should be aware of up front. Continue Reading
Big data projects might require more than Hadoop to succeed
Hadoop isn't a magic bullet for meeting big data needs, says Gartner analyst Doug Laney, who offers advice on how to reap the benefits of big data investments. Continue Reading
Panelists discuss pitfalls of Hadoop, other big data technologies
A panel of technology vendors and analysts weighs in on the upsides and downsides of technologies such as Hadoop and MapReduce. Continue Reading
4Videos-
Analysis of Hadoop and big data technologies
Watch the video interviews in this section for analyses and insights into the issues involved in evaluating, deploying and managing Hadoop tools and big data technologies. Well-known consultants and industry analysts share tips on adoption of Hadoop and other big data tools and on how to implement successful big data management and analytics programs.
Examining Hadoop adoption rates -- and why they aren't higher
What's slowing the adoption of Hadoop? SearchDataManagement editors take a look at the obstacles facing the big data framework in mainstream organizations.
Hadoop industry initiative focuses on better interoperability
Now that organizations are getting a handle on using big data, attention is shifting to Hadoop interoperability. But a vendor initiative on that could end up causing a schism.
Expert advice: Start by finding a business need for Hadoop
"Don't do it to do it -- do it because there's a reason," advises consultant Rick Sherman, saying that Hadoop projects should be tied to a compelling business case.
Using Hadoop: A user's viewpoint on when it makes sense
Ryan Fenner, an enterprise data architect at MUFG Union Bank, discusses Hadoop's potential value and uses, as well as its relationship with traditional data warehouses.
White: Don't get misled by hype in selecting a Hadoop platform
Consultant Colin White discusses the maturity of Hadoop tools and details some of the key issues to consider when evaluating Hadoop distributions.
Eckerson: Hadoop can bring big data ROI for users
TechTarget analyst Wayne Eckerson discusses the potential benefits and challenges of using Hadoop systems to run big data applications.
Rogers: Skills shortage impedes adoption of big data tools
Shawn Rogers of Enterprise Management Associates explains a common roadblock to adoption of big data systems and technologies -- a lack of big data skills in organizations.
McKnight: Getting started with big data analytics
Consultant William McKnight discusses big data basics, offering practical advice on key issues related to big data management and analysis.
Eckerson: Tips on using big data analytics software and tools
Wayne Eckerson offers advice on using big data analytics technology and shares his view of the big data big picture.
5Glossary-
Glossary of Hadoop-related terminology
This glossary offers definitions of key terms pertinent to Hadoop projects and big data initiatives.
6Hadoop quiz-
Test your understanding of the Hadoop ecosystem
Take this brief quiz to see what you have learned about Hadoop.
Take This Quiz
Start the conversation
0 comments