Thanks to the rise of "big data" and more companies needing real-time integration for business intelligence (BI), about 40% of data management professionals feel data integration projects will be one of their top challenges of the year, according to SearchDataManagement.com's 2011 readership survey.
In an effort to help out with this problem, SearchDataManagement.com talked to two veteran industry insiders -- Steve Yaskin, the chief technology officer of Sunnyvale, Calif.-based data virtualization and cloud integration firm Queplix Corp., and Bill Gassman, a research director with Stamford, Conn.-based analyst firm Gartner Inc. -- to get their thoughts on the best ways to ensure integration success. Here are the top five data integration best practices they had to offer:
1. Make sure you have clear visibility into data. A recent study from the Institute of Electrical and Electronics Engineers (IEEE) found that over 40% of time in the typical data integration project is spent on discovery -- the process of inventorying an organization's data stores.
That's why Yaskin thinks it's a good idea to consider using automated data discovery and data dictionary tools, such as those offered by Queplix and others. But the bigger lesson, Yaskin says, is that it's important to gain visibility into data sources before beginning any data integration initiative.
"A lot of times we see customers, especially large organizations, and they have something as big as SAP or Oracle and the system had been customized over the years. What happens [over time] is that the business owner may not have full visibility into what's going on behind those sources," Yaskin said. "Getting instant visibility into these objects is a tremendous first step."
2. Weave data quality into the process. Software vendors are increasingly bundling data integration with data quality tools. Gartner analysts predict that the markets for data integration and data quality tools will eventually collapse into one -- a trend that reflects the importance of making sure that data quality processes are woven into the fabric of data integration plans.
Despite growing customer demand for bundled integration and quality tools, many companies continue to be in denial about the dismal condition of their information stores, according to Yaskin.
"Before they can integrate the data between one or more systems in a meaningful way, they have to cleanse the data," he said. "Look at what state the data is in. What are the misalignments between the data? What are the nulls? What are the date ranges? What are the heuristics of that data?"
3. Form a data integration competency center. Companies launching a data integration initiative could benefit greatly by organizing a small group of employees to work together to keep the project on track, according to Gartner. The analyst said this group would ideally include somebody from the technical side, who understands how integration tools work; business users who run analytics reports based on newly integrated data; and company decision makers -- the consumers of the information.
"Have an organizational strategy or some sort of competency center to really keep the strategy going, make sure the skills are there and make sure the continuity is there as people change jobs," he said.
4. Retain control of information when dealing with cloud integration firms. Cloud data integration and data virtualization vendors like Queplix are everywhere. They include Dell Boomi, Composite Software, Informatica and the list goes on. Organizations mulling the prospect of doing business with such companies should closely examine their data management policies.
"The number one thing is making sure you know who owns your data," Gassman said, "because if you relegate that to a third party, and then that third party goes out of business or you want to change vendors, you need to make sure that the change is easy to do."
5. Look for tools suitable to business users. Data integration issues aren't just IT's problem anymore. Increasingly, rank-and-file business users are getting into the act through so-called self-service business intelligence (BI) and analytics applications. Whether organizations are seeking to combine information through data virtualization or a more traditional form of data integration, experts say it's always good to seek tools that help business users by employing visual user interfaces and drag-and-drop templates.
"This way, every time [business users] need to bring a new object into the mix and run a BI tool on that object or run a report, they don't need to run down the hall to the IT department," Yaskin said.