With business intelligence (BI) software programs expanding in size and scope -- and more rank-and-file business users creating reports than ever before -- organizations are growing increasingly concerned about the data integration challenges of BI projects, according to the results of a new SearchBusinessAnalytics.com survey.
The March poll of SearchBusinessAnalytics.com members found that data quality and data integration challenges are two of the biggest problems plaguing BI initiatives. The results also indicated that software buyers want BI applications that can be easily integrated with enterprise applications from vendors like Oracle Corp. and SAP AG.
The best practice is to really sit back for a couple minutes and really get a handle on what you're trying to do with the data and how the data needs to be transformed.
Rick Sherman, founder, Athena IT Solutions
A total of 44% of respondents said data integration would be one of their biggest BI-related challenges this year. That's a sharp increase from last year, when just 23% of respondents cited integration woes.
To learn more, SearchDataManagement.com got on the phone with Rick Sherman, the founder of Athena IT Solutions, a Stow, Mass.-based firm that provides data warehouse and business intelligence consulting, training and vendor services. Sherman explained why data integration concerns are at the forefront of the BI practitioner's mind and offered tips on how to face down some of the most pressing data integration challenges. Here are some excerpts from that conversation:
Our latest survey results suggest a sharp increase in the number of organizations that are concerned about the data integration issues associated with BI applications. Is something new happening in the marketplace that is driving these concerns?
Rick Sherman: I think what's new is the fact that the [data integration] issues are more visible and more people have access to BI or are trying to do reporting, analytics and BI than ever before. It isn't that the problems are new. It's that they're more visible because more people are encountering them. The other point [from] a business user context [is that they are] initially trying to do reporting and analysis from an existing operational application [and it's just one] source. [There] might be data quality issues, but they do not have to integrate data, because they're getting it from one source. As soon as you start doing that, you start needing to get data from other applications, and that's when you start encountering more data quality issues and then more data integration issues. It's like [going] from walking to jogging to running. [And] then we're back to the issues that a lot of folks had early on in data warehousing and BI, which [center on the idea that] integrating data across different sources is tough.
What's so tough about it?
Sherman: It's tough not because of access issues or getting to the data. It's tough because most companies big or small [have] issues with inconsistent product lists, inconsistent customer lists [and] people using different metrics or key performance indicators between departments. All of that falls under the data integration umbrella and a lot of that has to do with people agreeing on what the definitions of things are and figuring out how to make the product list, customer list [and] employee lists [be] consistent. [As] soon as you start pulling the data from different places [there is a need for data integration]. Making it consistent and making it so it's valid is the tough part. I think we've got a convergence of things that really haven't created a new problem but have exposed a lot more people to the data integration, data quality and data consistency problem.
What are some of the initial challenges that users face when taking on a BI-related data integration project?
Sherman: We find when we do consulting and when I'm doing training and education that the people and even the vendors that are new to it don't quite get how tough this stuff is. You've got these vendors who keep thinking that you don't have to build a warehouse and you don't have to do master data management and customer data integration. [They say they've got tools that] you can plug into any source you want and [pull data] into a report. But, you know, that is just access. That is not integration. Similarly, if you have, for example, a warehouse and you have an MDM hub [then] it is an access issue, so those tools work great. But then at some point the business users that fell into listening to that realize that they have to do integration. [At] that point they realize that it's not just an access thing, it's an integration thing.
Respondents to the survey indicated that they want BI applications that can easily integrate with major enterprise applications from the likes of Oracle and SAP. Do smaller or niche BI software vendors have a track record of problems when it comes to integrating with such applications?
Sherman: You know, it's funny if you think about 10 years ago versus now. Ten years ago, the smaller vendors didn't have access to -- and there wasn't as much knowledge as to what was in -- SAP or Oracle apps. But, especially with services and SOA and everything else coming out, the ability to access the enterprise applications has gotten easier and easier. So I really don't think that's as big an issue today. I think that what happens [is that at] the larger firms you have SAP and then you have all the enterprise apps that Oracle has acquired over the years. You've got two major spheres of application knowledge that you have to have. But when you get down to the SMBs [small and medium-sized businesses], there are hundreds of enterprise apps, such as financial apps geared toward smaller firms and, more importantly, you start getting into industry-focused applications. [The challenge for SMBs is that they] have a lot more BI apps that might not have as much knowledge as to how to access their [enterprise apps] or those enterprise apps might not be as open as SAP and Oracle are. SMBs also have the issue of figuring out how to integrate with a lot more sources than if you're talking about larger firms.
What advice do you have for organizations that are interested in purchasing new BI software, but who are also concerned with integration issues?
Sherman: There isn't as much differentiation when it comes to the BI and the [extract, transform and load (ETL)] tools as there used to be. [For example, all of] these tools can go out to relational sources. Almost all of them can go after some sort of unstructured data like Excel. All of them have support services now [and] all of them can access [sources] in real time. So I don't think there [are many differences in terms of] actually getting to the sources themselves. But there is prebuilt access that you can get for some of the applications like SAP and Oracle. So based on the apps you have, it is probably good to see if the BI or ETL vendors have any special hooks into those particular applications. [You] might be able to leverage that.
When it comes to BI and data integration, what is your favorite best practice?
Sherman: I think my favorite one for integration is that you need to understand and document the different business rules related to how you're going to integrate the data. [Oftentimes, too many] people start coding too quickly or too many people start moving data from point A to point B. The best practice is to really sit back for a couple minutes and really get a handle on what you're trying to do with the data and how the data needs to be transformed. That means talking and getting the definitions down -- and IT folks don't like to talk. You get into these meetings and a lot of times they can get political and IT folks don't like to handle politics. So you really need to be able to manage that. But the key is to really get the definitions and understand how you're trying to integrate the data. It's not a tools issue. It's really a definition issue.