Sergey Nivens - Fotolia

Manage Learn to apply best practices and optimize your operations.

Data integration framework needs more horsepower to handle big data

Big data architectures are complicating the data integration process for more and more IT teams, driving them to hit the gas on efforts to expand their integration capabilities.

In many organizations, the rubber is meeting the road on the need for an upgraded data integration framework that incorporates big data platforms. And you could easily hit some potholes on the journey -- or worse, end up in a ditch.

For starters, big data architectures typically include a combination of internal systems and external data sources. They also add various types of unstructured and semi-structured data, in addition to structured transaction data. Hadoop data lakes and NoSQL databases pose different integration challenges compared to traditional data warehouses. The growing adoption of stream processing tools puts pressure on IT teams to rev up the data integration process to real-time speeds.

That nets out to a lot of added demands -- and new investments. In a 2016 Magic Quadrant report, Gartner said the need to blend existing IT infrastructure with big data systems, cloud platforms and other emerging technologies is ratcheting up the number of data integration initiatives getting the green light from corporate executives.

TDWI analyst Philip Russom made a similar point in a December 2015 report on modernizing a data integration framework. Without broader integration capabilities, "organizations cannot satisfy new and future requirements for big data, analytics and real-time operations," Russom wrote.

But there's still work to be done. Gartner analyst Merv Adrian said in an October 2016 blog post that ingesting data into data lakes had been a big discussion topic with user clients at the company's annual Symposium/ITxpo conference that same month. Much of the focus, he added, was on finding data integration tools to help in "managing and documenting the process better."

This handbook offers advice on navigating the new demands to help your organization polish up its data integration framework -- and stay out of the big data integration breakdown lane.

Next Steps

Other articles in this handbook:

Metadata management tools help data lake users stay on course
Effective metadata management processes can prevent analytics teams working in data lakes from creating inconsistencies that skew the results of big data analytics applications. 

Streaming, connectivity new keys to data integration architecture
The growing use of the cloud, big data and data from outside sources has complicated data integration, making the addition of data streaming and broader connectivity a must.

What are key features for choosing the best ETL tools for your needs?
Choosing the right ETL tool for your data integration requirements can be a challenge. Here's a rundown on what to look for in ETL software and potential vendors to consider.

This was last published in February 2017

Dig Deeper on Enterprise data integration (EDI) software

PRO+

Content

Find more PRO+ content and other member only offers, here.

Start the conversation

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

-ADS BY GOOGLE

SearchBusinessAnalytics

SearchAWS

SearchContentManagement

SearchOracle

SearchSAP

SearchSQLServer

Close