This content is part of the Essential Guide: Cloud data warehouse guide: Using Redshift, rival platforms

Enterprise data warehouse architecture endures -- and evolves

The traditional data warehouse doesn't get the hoopla that Hadoop and NoSQL systems do, but it's still a core component of most IT infrastructures. And it's even learning some new tricks.

Data warehouses can seem a bit humdrum, especially in comparison to newer technologies such as Hadoop and NoSQL databases. In the big data era, building an enterprise data warehouse architecture doesn't have the same dinner-party cachet as setting up a Hadoop-based data lake does. But the EDW and its smaller relatives -- data marts, analytical databases, data warehouse appliances -- still have a prominent place at the IT table, feeding trustworthy data to business intelligence and analytics users.

In many cases, that includes users involved in big data analytics programs. In a survey conducted last summer by The Data Warehousing Institute, 78% of the 328 respondents said their organizations were using data warehouses to help support advanced analytics applications. By comparison, only 14% cited NoSQL systems, while 13% said they were using clusters based on commercial Hadoop distributions and 12% had deployed platforms running open source Apache Hadoop. A 2013 survey conducted jointly by Enterprise Management Associates Inc. and 9sight Consulting found similar results: Thirty-four percent of the 259 respondents said their big data environments included enterprise or federated data warehouses, while 30% listed data marts and 42% said analytical database platforms or appliances were in their big data mix. Hadoop and NoSQL software were chosen by just 16% and 22%, respectively.

And in a Magic Quadrant report on data warehouse databases published in March 2014, consulting and market research company Gartner Inc. said most of its clients had decided that optimizing their existing EDWs was the best short-term strategy for meeting new analytics requirements. Gartner also noted that data warehouse vendors were accelerating their adoption of new tools and techniques to better handle modern workloads, adding cloud support and features such as parallelization.

You'll find a variety of content on SearchDataManagement and SearchBusinessAnalytics that offers insight on some of those new capabilities and the data warehouse's evolving, but continuing, role in BI and analytics architectures. That includes analysis from several IT consultants. In one article, David Loshin explains in more detail why data warehouse systems aren't ready for retirement yet. In another, Wayne Eckerson weighs in on why you shouldn't listen to data warehouse bashing by big data vendors. Rick van der Lans examines the logical nature of modern data warehouses, while Claudia Imhoff and Colin White discuss ways to extend BI and data warehouse architectures to meet new analytics requirement. And in a podcast Q&A, Rick Sherman assesses the maturity of cloud-based data warehousing technologies and big data management services.

In addition, we look at the marriage of data warehousing and Agile software development at several user organizations, and we detail the deployment of a marketing analytics system that combines MongoDB Inc.'s NoSQL database with a cloud data warehouse service. Another story focuses on a project at MUFG Union Bank that is designed to bring data from various shadow IT units into a centrally managed enterprise data warehouse architecture. And in a video interview, Sam Strum, director of data services at ocean shipping services provider Inttra, offers advice on managing data warehouse projects. If you're planning, embarking on or involved in one, hopefully it will leave an enduring legacy -- in the positive sense of that word -- in your organization.

Craig Stedman is executive editor of SearchDataManagement. Email him at [email protected] and follow us on Twitter: @sDataManagement.

Next Steps

Get information comparing the total cost of using Hadoop clusters and data warehouses

Find out why temporal data capabilities have become a key data warehousing feature

Read about an enterprise data warehouse project at consumer goods maker Unilever

Dig Deeper on Data warehouse project management