BACKGROUND IMAGE: iSTOCK/GETTY IMAGES
Using big data and Hadoop 2: New version enables new applications
-
Article
Big data users find more to do with Hadoop 2's YARN resource manager
With Hadoop 2's YARN resource manager, the distributed processing framework is no longer bound to MapReduce, which enables Hadoop systems to do more. Read Now
-
Article
Will Hadoop 2 have wider appeal than the original version?
The release of Hadoop 2 -- and the wider range of options that come with it -- may give prospective users more reasons to move forward with adoption. Read Now
-
Article
Examining Hadoop 2's new features: Key questions answered
Learn about the most important features of Hadoop 2, including YARN and new federation and high-availability components for the Hadoop Distributed File System. Read Now
-
Article
Unwrapping the layers of big data: YARN, MapReduce, Hadoop clusters
MapReduce and Hadoop have matured as technologies, and together with YARN they are capable of performing faster, more flexible big data functions. Read Now
Editor's note
Early adopters of Apache Hadoop, including high-profile users such as Yahoo, Facebook and Google, had to rely on the partnership of the Hadoop Distributed File System (HDFS) and the MapReduce programming and resource management environment. Together, those technologies enabled users to process, manage and store large amounts of structured, unstructured and semi-structured data in Hadoop clusters.
But there were limitations inherent in the Hadoop-MapReduce pairing. For example, Yahoo and other users have cited issues with the first generation of Hadoop technology not being able to keep pace with the deluge of information they're collecting online because of MapReduce's batch processing format.
Hadoop 2, an upgrade released by the Apache Software Foundation in October 2013, offers performance improvements that can benefit related technologies in the Hadoop ecosystem, including the HBase database and Hive data warehouse. But the most notable addition in Hadoop 2 -- which originally was referred to as Hadoop 2.0 -- is YARN, a new component that takes over MapReduce's resource management and job scheduling duties. YARN (short for Yet Another Resource Negotiator) enables users to deploy Hadoop systems without MapReduce. Running MapReduce applications is still an option, but other kinds of programs can now be run natively as well -- for example, real-time querying and streaming data applications. The enhanced flexibility opens the door to broader uses for big data and Hadoop 2 implementations; in addition, YARN allows users to consolidate multiple Hadoop clusters into one system to lower costs and streamline management tasks. The upgrades in Hadoop 2 also boost cluster availability and scalability, two other issues that held back the first version of Hadoop.
Even with the added capabilities, Hadoop 2 still has a long way to go in moving beyond the early adopter stage, particularly in mainstream IT shops. But the new version heralds a maturing technology and a revamped concept for developing and implementing big data applications. This guide explores the features of Hadoop 2 and potential new uses for Hadoop tools and systems with insight and advice from experienced Hadoop users as well as industry analysts and consultants.
1Maximizing the potential of Hadoop 2: Opportunities and challenges
Hadoop 2 can support applications in a wider range of programming modes and data-crunching capacities. In addition, the Hadoop framework is being tapped for involvement in areas such as mainframe modernization and mobile app development. In this section, learn about new trends in the use of Hadoop and hurdles that could get in the way of the technology -- and Hadoop users.
-
Article
New applications for big data and Hadoop 2 -- new challenges, too
Hadoop 2 extends the framework's capabilities beyond batch processing and serving as a big data landing pad, but there are new issues to consider as well. Read Now
-
Article
Evolving ecosystem around Hadoop enables new programming approaches
Hadoop 2's features include new development options that expand the potential for innovation in building big data applications. Read Now
-
Article
Hadoop 2 gives U.K. companies new options, but usage lags U.S. levels
Many organizations in the U.K. and Europe may not be ready to take full advantage of the broader data management capabilities that Hadoop 2 has to offer. Read Now
-
Podcast
A broadening concept of Hadoop: News from Hadoop Summit 2013
Learn what industry insiders and users had to say at Hadoop Summit 2013 about the increased flexibility in Hadoop 2 and potential new enterprise uses for the technology. Listen Now
-
Article
Does Hadoop have a role to play in mainframe modernization?
Hadoop clusters are threatening the data warehousing status quo. Could they also make incursions into the world of mainframe modernization and migration? Read Now
-
Article
Integrating Hadoop for mobile application development
Get tips on how to use existing Hadoop applications to help meet the data needs of mobile application users. Read Now
2Weighing Hadoop 2's place in business analytics and operations
In this section, discover how Hadoop 2 supports business analytics and enterprise operations -- and get advice on what's needed to make the potential uses a reality in companies wanting to take advantage of its added functions. Consultants and experienced users discuss what Hadoop 2 has to offer and what challenges stand in the way of getting valuable business benefits from the upgraded Hadoop framework.
-
Article
Analytics applications on firmer ground with Hadoop 2
Consultant David Loshin explains how the separation of resource and application management in Hadoop 2 boosts its ability to support data analytics. Read Now
-
Article
Hadoop 2 demands highly sought-after big data skills
While organizations can now deploy Hadoop 2, many don't have employees in-house with the skills needed to support the platform. Read Now
-
Article
Can Hadoop keep up with real-time business analytics needs?
While Hadoop vendors are touting its ability to function as a real-time analytics tool, there are still questions about whether it's really suited for real-time uses. Read Now
-
Article
Exploring best practices for using Hadoop in enterprise operations
Strata Conference panelists discuss Hadoop best practices for enterprise deployments and the ongoing evolution of Hadoop from proof of concept to operations workhorse. Read Now
-
Article
Charting new frontiers in geospatial data with Hadoop
Learn how satellite operator Skybox Imaging is now able to find new meaning in satellite data with analytics help from Hadoop. Read Now
Machine Learning Meets Data Quality
Read an exclusive interview with Andrew Burt, chief privacy offer and legal engineer at Immuta Inc., on data governance and machine learning integration. Plus, uncover steps IT managers are taking to improve data quality in their big data environments in order to ensure analytics accuracy.
3Brush up on your Hadoop 2 vocabulary
Terminology abounds in the world of Hadoop, particularly because there are so many add-on tools in the Hadoop ecosystem. In this section, you'll find definitions of various terms associated with big data and Hadoop 2.
-
Definition
Apache Hadoop YARN
Apache Hadoop YARN is the resource management and job scheduling technology in the open source Hadoop distributed processing framework. Read Now
-
Definition
Apache HBase
Apache HBase is a column-oriented key/value data store built to run on top of the Hadoop Distributed File System (HDFS). Read Now
-
Definition
Apache Hive
Apache Hive is an open source data warehouse system for querying and analyzing large data sets that are principally stored in Hadoop files. Read Now
-
Definition
Apache Pig
Apache Pig is an open-source technology that offers a high-level mechanism for parallel programming of MapReduce jobs to be executed on Hadoop clusters. Read Now
-
Definition
Apache ZooKeeper
Apache ZooKeeper is an open source file application program interface (API) that allows distributed processes in large systems to synchronize with each other so that all clients making requests receive consistent data. Read Now
-
Definition
Apache Software Foundation
The Apache Software Foundation (ASF) is a non-profit, decentralized corporation that oversees development of Apache software. The organization was founded in 1999 is based in Forest Hill, Maryland. Read Now
-
Definition
Big data
Big data is an evolving term that describes a large volume of structured, semi-structured and unstructured data that has the potential to be mined for information and used in machine learning projects and other advanced analytics applications. Read Now
-
Definition
Big data analytics
Big data analytics is the often complex process of examining large and varied data sets -- or big data -- to uncover information including hidden patterns, unknown correlations, market trends and customer preferences that can help organizations make informed business decisions. Read Now
-
Definition
Big data management
Big data management is the organization, administration and governance of large volumes of both structured and unstructured data. Read Now
-
Definition
Hadoop
Hadoop is an open source distributed processing framework that manages data processing and storage for big data applications running in clustered systems. Read Now
-
Definition
Hadoop 2
Apache Hadoop 2 is the second iteration of the Hadoop framework for distributed data processing. Hadoop 2 adds support for running non-batch applications as well as new features to improve system availability. Read Now
-
Definition
Hadoop cluster
A Hadoop cluster is a special type of computational cluster designed specifically for storing and analyzing huge amounts of unstructured data in a distributed computing environment. Read Now
-
Definition
Hadoop Distributed File System (HDFS)
The Hadoop Distributed File System (HDFS) is the primary data storage system used by Hadoop applications. Read Now
-
Definition
MapReduce
MapReduce is a core component of the Apache Hadoop software framework. Read Now
-
Definition
Unstructured data
Unstructured data is information, in many different forms, that doesn't hew to conventional data models and thus typically isn't a good fit for a mainstream relational database. Read Now
4How well do you know the Hadoop ecosystem?
Take this brief quiz to test what you know about the Hadoop framework.