Speakers at this week's TDWI World Conference in Chicago offered up loads of advice about how to deal with problematic...
data warehouse architecture and implementation challenges like the proliferation of data sources and information management silos.
And technology professionals attending the event, hosted by The Data Warehousing Institute, had plenty of data warehousing challenges on their minds.
Some -- like Kevin Najimi, a senior integration developer at a financial services firm -- were at the show looking for fresh ideas about out how to evaluate and perhaps re-architect established data warehouse implementations.
"We are in the process of rethinking our architecture in general," Najimi said. "I think one of the biggest challenges in our industry and in our firm, specifically, is the amount of data we have and the proliferation of third-party applications and third-party data sources, and bringing those together."
Conference attendee Kenneth Jarvis, an application developer and data warehouse team leader with HCA, a Nashville, Tenn.-based health care services company, said he is also concerned with such "classic" obstacles to data warehouse success.
"We've got data coming in from disparate systems and we're trying to make it look homogenous to the users," he said. "That's always been difficult and always will be."
Jarvis said another challenge is keeping up with the various information management practices used by HCA's business units.
"We're a very large company," he said. "Everyone has got their way of doing things, and it [can be] inconsistent from one part of the company to another."
Still other attendees, like Suvendu Datta, an information management professional with an insurance company, were looking for advice on how to launch a data warehouse initiative. Like many insurance industry organizations, Datta's company is simultaneously dealing with an influx of electronic information and strict data retention regulations.
Datta is hoping that his firm's new data warehouse program will make things easier.
"The main challenge is the architecture, because we have so much complexity," he said. "We work with actuaries and actuaries come up with all different metrics. We have huge volumes of data and [about] 10 years of history -- and they now want 15 years [of history]."
Datta's company runs more than 90 homegrown and commercial business applications, and many of them house duplicate information. He said cleaning up any data quality issues prior to the data warehouse implementation poses yet another challenge.
"We have silos right now. It's all siloed," he said. "You can find the same information in multiple places."
Datta said speakers at the TDWI conference affirmed his plan to roll out the data warehouse project to one system or business unit at a time, while remembering to keep enterprisewide goals in mind. He said the data warehouse project will initially be focused on financial information.
"Basically, our plan will be to take one project, create it in the new architecture with data warehousing, and then start building slowly on top of that," he said.
Tipping ‘the sacred cows’ of data warehouse architecture
Some of the more unorthodox advice given to TDWI conference attendees came from Evan Levy, the co-founder of Baseline Consulting Group Inc., a data management consultancy recently acquired by DataFlux.
Levy said the attendees should take another look at some of the accepted "sacred cows" of data warehousing and -- if it makes sense -- consider implementing alternative approaches.
For example, Levy told the classroom to at least reconsider the idea that a person should not use third normal form for operational systems.
"Our goal is to call that into question -- the presumption that you can or can't do certain things with dimensional or third normal form," he said. "We're not saving lives. Everyone takes our job way too seriously and assumes there is only one way to do stuff."
Levy said decisions about the data warehouse architecture design should be based on the specific requirements of individual organizations, and not necessarily the most popular approaches.
When making architecture decisions, Levy said IT professionals should think hard about what makes sense for their environments, how many users they are supporting, the overall level of sophistication and complexity, and budgetary constraints.
"Keep one thing in mind," he said. "We don't build an architecture because we like to [look at] pretty pictures. We build the architecture to support a set of business premises and requirements."