Interest in open source data integration technologies growing

By Jeff Kelly, News Editor 30 Sep 2009 | SearchDataManagement.com

News on data management trends and technology Digg This!     StumbleUpon     Del.icio.us

While implementations are still few and far between, interest in open source data integration technology is slowly but surely growing as a weak economy continues to put pressure on IT budgets.

According to a recent survey by Gartner, 11% of organizations in the market for data integration technology have evaluated open source tools along with commercial products. That's nearly double the percentage that said they considered open source data integration in a 2008 survey.

Still, implementations of open source data integration tools are rare compared with commercial deployments, according to Ted Friedman, an analyst with Stamford, Conn.-based Gartner, who analyzed the survey's findings. The two leading open source data integration vendors – Talend and Pentaho -- count only several hundred paying customers between them, Friedman estimates.

Of those, most -- 83%, according to the survey -- use open source data integration tools for ETL functions supporting business intelligence (BI) and data warehousing projects. That's because most open source data integration tools have "less mature capabilities for supporting" other styles of data integration, including data federation and real-time data integration techniques like change data capture, Friedman said.

More on data integration Read about Informatica's acquisition of a CEP vendor to complement data integration   Learn why one data integration vendor is offering its technology at an hourly rate on Amazon's EC2   Find out what Oracle's acquisition of Golden Gate Software says about change data capture

Children's Hospital of Omaha has been using open source data integration from Talend since earlier this year. Reflecting the survey's findings, however, the hospital uses the open source ETL tools mainly to integrate data from disparate databases and an electronic medical records system to a Sybase IQ data warehouse for billing and other analysis.

For a number of non-ETL jobs, like integrating data to and from clinical lab systems, the hospital uses an eLink Systems interface engine that transmits data in the HL7 message standard.

"There's some limitations Talend has, but for the money and our use cases, it really fit the bill," said Wendy Worthing, manager of data services and technical training at Children's Hospital.

Friedman expects those limitations to be addressed by vendors like Talend over the next two to four years, making open source data integration an attractive, lower-cost option for some companies.

"Open source tools in this market are starting to really see some traction," Friedman said. "And I do think [that] as these tools grow in maturity, you're really going to see more balance in their uses."

One new use case for open source data integration tools could be master data management (MDM). Talend, based in Los Altos, Calif., announced on Monday that it had acquired the MDM-related assets, including an MDM repository technology, of Amalto Technologies. Amalto specializes in business-to-business exchanges.

Talend had in place both the data quality and data integration capabilities needed to support MDM implementations, but it lacked the actual MDM repository, said Yves de Montcheuil, vice president of market strategy for Talend. With the acquisition, he said, companies, especially small and midmarket firms, will have the ability to restart stalled MDM projects at a fraction of the cost.

Philip Russom, senior manager of research and services at the Renton, Wash.-based Data Warehousing Institute, said Talend may have hit on a winning combination, one that could help propel the open source data integration market.

"Users go through a maturation process where they mature or move from data integration to data quality," Russom said. "And after they've done data quality for a while, they mature and move on to something else, and quite often that something else is master data management."

Friedman said: "Sometimes synchronization of master data across systems can very suitably be done in a bulk and batch fashion. So absolutely I think the [open source data integration] tools are relevant to the MDM domain."

For now, companies in the market for data integration tools should consider open source alternatives when their needs are largely for bulk and batch loading, Friedman said. But he cautioned that open source data integration, while less expensive than commercial products, comes with its own costs.

For example, companies using open source data integration software will have to perform much of the customization with other data management technologies themselves, as most open source vendors lack robust partner networks. There is also a dearth of skills around open source data integration in the marketplace.

And though ETL represents a significant percentage of most organizations' data integration needs, there are others that open source technology just can't meet at this point, according to Friedman.

"ETL is becoming just one style of many that organizations need to support," he said.

Tags: Enterprise data integration (EDI) software,  Extract transform load tools,  Enterprise information integration and data federation,  VIEW ALL TAGS

Digg This!     StumbleUpon     Del.icio.us

RELATED CONTENT Enterprise data integration (EDI) software Informatica hopes to bridge the business/IT divide with latest release Data integration for Software as a Service business intelligence software evolves Yearbook publisher finds BI reporting, data integration tools from same vendor Efficiencies can be gained at the intersection of middleware and data integration Data warehouse project leads to data mashups in D.C. Understanding the latest data integration strategies and technologies Understanding key data integration trends and business drivers Informatica acquires complex event processing vendor Agent Logic Informatica offers data integration software by the hour on Amazon EC2 Oracle's GoldenGate acquisition underlines demand for real-time data integration capabilities Extract transform load tools Data integration for Software as a Service business intelligence software evolves Efficiencies can be gained at the intersection of middleware and data integration Informatica offers data integration software by the hour on Amazon EC2 Open source data integration tools good choice for 'standard' ETL jobs Operational BI gets a boost from real-time data loading Data integration costs hurting companies with poor negotiation skills Wholesaler turns to data integration adapters for mainframe to Oracle migration Pervasive the most persuasive, cost-effective data integration platform, study says Integration competency centers centralize data integration projects Data warehouse case study: Midsized insurer goes enterprise with data warehouse Enterprise information integration and data federation Handling project management issues for a proprietary ownership conflict Understanding the latest data integration strategies and technologies Understanding key data integration trends and business drivers Data federation technology complements fragmented data warehouses Data integration costs hurting companies with poor negotiation skills Wholesaler turns to data integration adapters for mainframe to Oracle migration Enterprise-wide customer data quality still elusive at most organizations Exec explains IBM's Information On Demand (IOD) initiative SaaS-based data quality and integration tools gaining momentum IBM unveils new data management software, services

RELATED RESOURCES 2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems Search Bitpipe.com for the latest white papers and business webcasts Whatis.com, the online computer dictionary