Data integration is a critical component of business intelligence (BI) processes, which are predicated on pulling together data from multiple source systems and consolidating it in data warehouses for analysis. But data management analysts warned that a BI data integration strategy can quickly go awry if it isn’t well designed and properly executed.
For example, Claudia Imhoff, president of consultancy Intelligent Solutions Inc. in Boulder, Colo., said bad timing can spoil everything: Data needs to be loaded into a data warehouse in time to be used for the planned BI purposes. Fully understanding an organization’s BI requirements, particularly when users need real- or near-real-time access to data, is a must for BI and data integration teams, Imhoff said.
Good data quality is just as important, according to Imhoff, who said that correcting and cleaning up bad data shouldn’t be a function solely of the BI data integration process. “Errors are happening everywhere else along the way, so you need to figure out where they are coming from,” she said -- and then work to prevent data mistakes from finding their way into source systems in the first place. In effect, Imhoff added, data integration and BI professionals are given the job of consolidating faulty data and then get the blame when it isn’t perfect. “We need to get people to understand that they shouldn’t just shoot the messenger,” she said.
Ted Friedman, an analyst at Gartner Inc. in Stamford, Conn., thinks that not paying enough attention to data quality is the biggest BI data integration danger companies face. “I’ve been following data integration for more than 10 years,” he said. “And I still spend days talking to organizations that are not getting the usage and trust and acceptance and value out of their BI efforts because the quality of the data is not good enough, and they haven’t done the right things to fix that.”
Data quality problems clearly affect more than BI data in wayward organizations, Friedman said, but he sees poor data quality as one of the primary barriers to successful BI programs. The shortcomings, he added, typically result from “not focusing [on data quality] early and often enough, and simply not doing enough to mitigate quality issues” as information is moved into data warehouses.
James Kobielus, who was an analyst at Forrester Research Inc. in Cambridge, Mass., before taking a job with a technology vendor earlier this year, also pointed to missteps on data quality as a common source of trouble for BI data integration efforts.
“Organizations think they can simply load data from their various back-end applications into a data warehouse and it will be usable without cleansing it or doing match-and-merge or transform [processes],” Kobielus said while he was still at Forrester. But doing so sets up companies for some nasty surprises, he added. For example, “they end up with six records on the same person and don’t know which one is the right one,” Kobielus said.
BI data integration’s dramatic effect
Another big source of inconsistent data, and drama, stems from internal debates over what constitutes a system of record, said Jill Dyche, co-founder of Baseline Consulting in Sherman Oaks, Calif. For example, she noted, there can be arguments about which transaction system should be used as the source of customer addresses. Such conversations often then turn to the definition of “address”: Is it a customer’s billing address or shipping address, or its headquarters location if that differs from the other two?
“That’s when the arguments ensue and business people become disaffected with the BI team’s ability to understand and deliver the right data,” Dyche said. “So then someone just decides to forklift everything into a single database, which the business people then refuse to use.”
Barry Devlin, founder of 9sight Consulting in Cape Town, South Africa, thinks the most problematic mistake is not including the right people in the process of crafting a BI data integration strategy and plan. “The people who really understand data and what it means are a particular subset of the business community who have been playing with data over the years -- they are the gurus and the power users,” Devlin said. As a result, he added, they’re best equipped to define what data needs to be integrated in order to create effective BI applications.
But in many cases, it’s left to the IT department to develop the data integration plan in addition to doing the implementation work, Devlin said. While IT pros may have a reasonable understanding of an organization’s data, to Devlin they aren’t the real experts. Bringing the two groups together to work on BI data integration can be a challenge, but it’s a must, he said.
Imhoff said there also is a strong tendency, especially in organizations that are new to BI, to inadequately scope the data integration requirements of a BI project and then put together an unrealistic schedule for delivering what’s needed. Integrating data and loading it into a data warehouse can take up as much as 60% to 80% of the overall BI development effort, Imhoff said. And, she cautioned, a project team that tries to do too much of that at one time can end up falling flat on its face. “You can’t eat the whole elephant at once,” she said.
Alan R. Earls is a Boston-area freelance writer focused on business and technology.