This article originally appeared on the BeyeNETWORK.
The other day, I was chatting with some consultant friends about their most recent set of engagements. It was interesting to hear that new data warehouse development efforts are rare. Occasionally, someone will start up a new data warehouse, but not as many as in years past.
Instead, the new thing for consultants is to rebuild existing data warehouses. With the overhauling of the data warehouse, the consultant goes back to the organization and builds the data warehouse the way it should have been built in the first place.
So the interesting question is “how did we build so many unsuccessful data warehouses in the first place?”
There are several reasons for this:
Building a series of data marts and calling it a data warehouse
This is the predominant mistake corporations have made. With data marts:
- there is no single source of the truth;
- there is no reusable foundation on which to build; and
- there is no reliable and immediately available source of data for other processing, such as data mining, etc.
In a word, the organizations that chose the data mart path discovered that they have been sold a bill of goods. A series of data marts simply does not deliver the foundation needed for long-term sustainable integrity of data, decision-making and growth.
There is a real irony here. The data warehouse professionals tried as hard as possible to point out that building numerous data marts was not building a data warehouse. And when this multiple data mart approach failed, what did the user say, “our data warehouse wasn’t a success”. The truth is that the data warehouse approach did not fail. The corporation had never tried the data warehouse approach. The multiple data mart approach failed.
Building the data warehouse under the “big bang” philosophy
Some consulting organizations built a large and well-honed methodology for the building of operational systems. When data warehousing developed, those consulting organizations ran the methodology through the word processor and converted the operational methodology to a data warehouse methodology. This is what they thought they were doing. The truth is that, the classical operational development methodology is so foreign from the data warehouse development methodology that, this approach ALWAYS spelled disaster.
Within the appropriate DSS development approach, called the spiral development approach, it is assumed that you will rebuild major parts of the data warehouse as soon as the end-users get their hands on it. This is because the end-users do not know what the requirements for the data warehouse will be. This implies a fundamentally different development approach than the operational “big bang” approach to the building of the data warehouse.
Not involving the end-user
Once in a blue moon you run into a data warehouse that appears to be a technological masterpiece. Except for the fact that the end-user never knew what was going to be in the data warehouse. So one day the data warehouse is complete and the end-user says – “what’s this? I don’t need this information.” Now the data warehouse becomes useless.
There are probably other reasons why a data warehouse has to be rebuilt. But this short list covers most of them.
The sad thing is that once a data warehouse has been built incorrectly, there really isn’t much choice but to go back in and rebuild it. In other words, there usually isn’t a lot that is salvageable from the wreckage.
But at least the consultant that is hired doesn’t have to try to convince the corporation of the right way to build the data warehouse. Through a lot of pain, the corporation now knows how to succeed and profit from a data warehouse.
Bill is universally recognized as the father of the data warehouse. He has more than 36 years of database technology management experience and data warehouse design expertise. He has published more than 40 books and 1,000 articles on data warehousing and data management, and his books have been translated into nine languages. He is known globally for his data warehouse development seminars and has been a keynote speaker for many major computing associations. Bill can be reached at 303-681-6772.
Editor's Note: More articles, resources and events are available in Bill's BeyeNETWORK Expert Channel. Be sure to visit today!