News Stay informed about the latest enterprise technology news and product updates.

Building the imperfect data warehouse

Sometimes, a perfect data warehouse is not an option.

This article originally appeared on the BeyeNETWORK.

The movement toward data warehousing has taken different paths in different industries. One industry in particular is of interest – the insurance industry. In the insurance industry, the shakers and movers for data warehousing were not the large leaders of the industry, but the mid-sized insurance companies. 

The really large insurance companies faced several daunting obstacles when it came to building a data warehouse. Some of the obstacles were/are: 

  • The size of legacy systems. The legacy systems are so large and need so much constant repair that trying to do anything with them is a real challenge. The older legacy systems have their own momentum, and each legacy system takes on its own individual life. The owners of the old stovepipe legacy systems actually like their systems the way they are and are not about to change those legacy systems (until they die or retire, whichever comes first).
  • Leadership. To build a data warehouse requires leadership and vision. Because of the size of the legacy systems and the length of time those systems have been in place, the vision and leadership for a data warehouse actually has to come from the president of the insurance company. The problem is that insurance company presidents are not technicians and have other important things to do. There just isn’t anyone with the scope and the reach to actually mandate a data warehouse.
  • The organization. The organization becomes invested in their own systems. They become enamored of the job they are doing and view anything that is disruptive to the job as undesirable. And there is no question that data warehousing is disruptive to older legacy applications. 

In a word, there is just too much that needs to be done and no one in a position to mandate that things get done when it comes time to build a data warehouse. As a consequence, large insurance companies languish. In doing so, they make the environment worse because they build the stovepipes even higher. 

But smaller insurance companies do not have quite the problem of scale that the large insurance companies have. Certainly smaller insurance companies have legacy systems. However, the scale of things is such that building a data warehouse is not an insurmountable obstacle in a smaller insurance company. (Notwithstanding, it is still a challenge to build a data warehouse in a small to mid size insurance company.) 

So – if you were standing in a large insurance company today, what would you do if you recognized that you needed a data warehouse but could not overcome the corporate hurdles? One option is to do nothing. This is a very easy option to take but doesn’t really accomplish anything. The stovepipes grow taller and the lack of integration of data grows worse. 

Another option is to build an “imperfect” data warehouse. An imperfect data warehouse is not ideal, but on occasion, it is the only option. Stated differently, an imperfect data warehouse is better than nothing at all (although, admittedly, it is not as effective as a perfectly designed and built data warehouse). An imperfect data warehouse is a purely practical exercise. 

What exactly is an imperfect data warehouse? 

An imperfect data warehouse is one whose scope is something less than the enterprise. Instead, in a fit of practicality, the data warehouse is built for a subset of the data of the corporation. You know that there will be overlap and redundancy, but – in the interest of moving forward – you build a data warehouse anyway. 

There are lots of ways to build subsets of corporate data. Some of them are:

  • build a data warehouse for North America, not for Europe, Asia and Africa,
  • build a data warehouse for product manufacturing, not for services and distribution,
  • build a data warehouse for financial aspects of the corporation and for nothing else. 

There are as many ways to subdivide the data found in the corporation as there are imaginations. 

The good news is that by subdividing the corporation, you can work within a smaller, more manageable framework. The bad news is that there undoubtedly will be corporate overlap and redundancy. At a later point in time that overlap and redundancy will have to be resolved. But at least there will be some benefit and some value in building something – the less than the perfect data warehouse.

Bill is universally recognized as the father of the data warehouse. He has more than 36 years of database technology management experience and data warehouse design expertise. He has published more than 40 books and 1,000 articles on data warehousing and data management, and his books have been translated into nine languages. He is known globally for his data warehouse development seminars and has been a keynote speaker for many major computing associations. Bill can be reached at 303-681-6772.

Editor's Note: More articles, resources and events are available in Bill's BeyeNETWORK Expert Channel. Be sure to visit today!

Dig Deeper on Data warehouse software

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.