freshidea - Fotolia

Solid data integration techniques pave clear path for info

Implementing an effective data integration strategy is becoming increasingly important as organizations look to collect and analyze information from a diverse set of data sources.

In 2010, officials at Montreal-based financial services firm CDPQ realized that its data management architecture needed a transformation -- badly. Point-to-point connections between systems made moving data around the company an arduous process that wasted IT resources. About 1,600 data-extraction jobs ran each night; many of them were similar in nature but had to be run separately because of a lack of a well-designed architecture and effective data integration techniques, said Alexandre Synnett, CDPQ's vice president of data management. Further illustrating the IT complexity, he told attendees at the 2015 TDWI Executive Summit in Las Vegas that there were as many servers in the organization as there were business users.

There were business ramifications, too: Users had a hard time accessing data to analyze it. CDPQ, which manages public-sector pension funds in Quebec, decided to bring its IT environment out of the 1980s in order to help it become more data-driven. Synnett said the company is now close to completing the staged deployment of a new architecture in which data flows more fluidly from operational to analytics systems and the data integration process is much simpler. Now, he added, end users can more easily get to the data they need to make more-informed -- and hopefully better -- investment decisions.

CDPQ's example points to the importance of an effective data integration strategy -- and to the predicament organizations can find themselves in when they lack smooth data pathways between systems. SearchDataManagement has published a variety of content that offers insight on data integration best practices and advice on building a data integration framework to support increasingly diverse pools of data and modern application needs. In one article, consultant Saurabh Jain looks at what goes into creating a distributed data management architecture to take the place of the enterprise data warehouse. In a similar vein, reporter Jack Vaughan details a data integration project at Western Union, which tied a new Hadoop cluster to other internal systems to support analytics applications.

In other stories, Vaughan examines a healthcare company's integration-fueled analytics efforts and writes about the emergence of self-service data preparation tools that use machine learning technology to simplify the integration process for data analysts looking to pull information from multiple data sets. Also, in a Q&A, consultant Rick Sherman offers tips on integrating on-premises and cloud applications. Has your organization implemented a robust set of data integration techniques and project management approaches? If not, its data architecture might be living in the past -- and that isn't a good way to meet current business needs for useful information.

Next Steps

More on data management at CDPQ: making data governance a collaborative effort

Big data applications can require new data integration approaches

Dig Deeper on Enterprise data integration (EDI) software