This article originally appeared on the BeyeNETWORK.
Abstract: As a technology, Data Warehousing (DW) has been successful over the past decade in establishing itself as an essential part of enterprise systems. The market for this technology has, likewise, grown large and diverse. However, major changes are likely over the coming year. What will be these changes? And, what will be the forces that cause these changes? This article explores these questions using the concepts of Clayton Christensen, a professor at Harvard Business School, who wrote the books Innovator’s Dilemma and Innovator’s Solution.
Every successful technology sows the seeds of its own demise. Transistors and jet engines are successful technologies and have faded into their respective infrastructures. Who today uses the phrases “transistor radio” or “jet airplanes?” Likewise, data warehousing has been a successful technology and is starting to fade into the infrastructure of enterprise systems. Instead of focusing on database design or data extraction, companies are now focusing on delivering better decision support across their entire organization. But they are realizing that the data warehouse is still essential to this new goal, just like that jet engine is critical for your next business trip.
Clayton Christensen, a professor at Harvard Business School, has described the maturing of technologies and their impacts on the marketplace from an insightful perspective. He predicts that a company leveraging a successful technology will sustain a continuous stream of innovations as dictated by their most demanding customers. Along with its competitors, this company will establish the standard and be the mainstream of the market. This is the traditional sustaining strategy that relies on competitive advantage through continual innovation.
Let’s apply Christensen’s concepts to the Data Warehousing (DW) market. Figure 1, below, shows the maturing of this market from the early 1990s. The vertical axis is functionality (i.e., performance or capability) of the DW products. In the early years, customers had to create most of their systems by internal development since DW products were often immature and inadequate. This situation is shown at Point A around 1992 where available products had functionality less than the requirements of the market.
Figure 1--Sustaining Strategy
Then for a period of over a decade, customer requirements and product functionality were in synch, and the DW market blossomed into a mainstream market of $8 billion dollars. Data warehousing was transformed from a novelty by a few pioneers to an essential element of every enterprise system.
As technology matures, product innovation, however, often outpaces the requirements of customers. This situation is shown at Point B as a surplus of DW functionality. I believe this describes the current market. This illustrates the ‘innovator’s dilemma’ since continuous innovation sows the seeds for the market’s demise. Most customers do not complain because pricing stays constant over this period and capabilities increase significantly. The DW market will remain stable unless other forces disrupt it.
According to Christensen, there are two possible types of disruptive forces: Low-End and New-Market.
Low-end disruption is direct competition from products with reduced functionality and reduced prices, using a different technology. As shown below in Figure 2, these challengers will initially be discounted and have little impact on the main market because their capabilities are not sufficient for most customers. This continues until the functionality is ‘good enough’ for the customers who are ‘over-served’ within the main market. As shown at Point C, the market may suddenly flip, with market share moving rapidly toward the low-end disruption, driven by superior price-performance levels.
Figure 2--Low-End Disruptions
During 2005, the DW market will be experiencing a low-end disruption from Microsoft’s SQL Server 2005. Without additional cost, a host of BI services will be packaged with the database. Because of the integration with .NET Visual Studio, many third-party software vendors can now offer their products with data mining and multidimensional analysis for just an extra click. For many companies in the main business intelligence market, these pervasive BI add-ons may be ‘good enough’ to pull them away from the usual BI vendors, such as Business Objects, Cognos, and the like.
New-market disruption is indirect competition from products that are more convenient and easier-to-use. This draws ‘non-customers’ (relative to the main market) into a new market. Christensen states, “New-market disruptive products are so much more affordable to own and simpler to use that they enable a whole new population of people to begin owning and using the product.”
As shown below in Figure 3, a new market is created and does not initially disrupt the main market. In fact, the new market will strengthen the main market by raising profit margins because the less profitable customers have fled to the new market. Over time, the functionality of the new-market disruption will improve to where the two markets will collide, with the larger new market absorbing the older main market.
The DW market is following this same scenario. With the emergence of data warehouse appliances, like that of DATAllegro and Netezza, I predict that a new DW market will be created over the next few years. Companies who have not invested in DW technology will be drawn into this new market through self-contained data warehouse appliances that are convenient and easy-to-use.
Figure 3--New-Market Disruptions
These companies could be small or medium businesses; however, the larger potential may be corporations with departments requiring new DW capabilities or having data volumes that are growing exponentially. “The market opportunity for non-consumption and, especially under-consumption of DW resources, is extremely large. In every client conversation, we see companies limiting DW usage or leaving whole departments unsupported,” said Stuart Frost, CEO of DATAllegro.
The critical factor will be the ability of data warehouse appliances to be truly an appliance. data warehouse appliances must be more than just a low-end alternative. They must be simple and reliable, delivering value at the press of a button. Plug in a microwave, press a button and zap your pizza for dinner. Plug in a data warehouse appliance, press a button and improve customer service. This level of convenience and usability will be the challenge for data warehouse appliances.
In conclusion, there are some bumps in the road for the DW market this year. Along with continuing acquisitions, low-end products and new market segments will challenge us all. So, will data warehousing fade from our IT landscape? Yes, just like jet engines have faded from air transportation. You do not think about it, but the technology is all around you.
 C.M. Christensen & M.E. Raynor, The Innovator's Solution: Creating and Sustaining Successful Growth, Harvard Business School Press, September, 2003, http://www.theinnovatorssolution.com/. Also see http://www.claytonchristensen.com/ for his other publications.
 IDC Market Analysis: Worldwide Business Analytics Software 2004-2008 Forecast, Report #31837, 2004.
Ibid, p. 45