Home > Ask the Data management / BI Experts > Integration and SOA data services Questions & Answers > Optimal data integration performance during the data explosion
Ask The Data Management Expert: Questions & Answers
EMAIL THIS

Optimal data integration performance during the data explosion

Jill Dyché EXPERT RESPONSE FROM: Jill Dyché

Pose a Question
Other Data Management Categories
Meet all Data Management Experts
Become an Expert for this site


Digg This!    StumbleUpon Toolbar StumbleUpon    Bookmark with Delicious Del.icio.us   


>
QUESTION POSED ON: 13 June 2006
A TDWI survey found that 55 percent of respondents expect their company's data volumes to rise more than 25 percent in the next year and a half. With this pending data explosion, it seems that merely boosting processing power is an insufficient strategy for maintaining proper data integration (of course, management disagrees). If you had to outline basic components for maintaining optimal data integration performance in the face of the data explosion, something that could easily understood by management, what would they be?

>
I'm abridging what could be a protracted answer to a complex question. But you're right; it's not all about the addition of linear processing capabilities.

Indeed, there a few different dimensions to consider when planning to scale your BI environment. For instance, there is the "mixed workload" factor -- query processing versus ETL. Adding additional processing power and memory is the most effective means of addressing incremental growth of data. It's important to realize however, that there's a point at which the physical table structures -- not to mention the physics of disk I/O -- will prevent the linearity of queries (e.g., in order to support x percent new data, add x percent new storage and CPU). Because ETL is much more I/O intensive than standard query processing of most BI environments, simply adding more storage and CPU can be impractical. Bottom line, you can't just throw more horsepower at more data.


Digg This!    StumbleUpon Toolbar StumbleUpon    Bookmark with Delicious Del.icio.us   



RELATED CONTENT
Extract transform load tools
Data integration for Software as a Service business intelligence software evolves
Efficiencies can be gained at the intersection of middleware and data integration
Interest in open source data integration technologies growing
Informatica offers data integration software by the hour on Amazon EC2
Open source data integration tools good choice for 'standard' ETL jobs
Operational BI gets a boost from real-time data loading
Data integration costs hurting companies with poor negotiation skills
Wholesaler turns to data integration adapters for mainframe to Oracle migration
Pervasive the most persuasive, cost-effective data integration platform, study says
Integration competency centers centralize data integration projects

Enterprise data integration (EDI) software
Informatica hopes to bridge the business/IT divide with latest release
Data integration for Software as a Service business intelligence software evolves
Yearbook publisher finds BI reporting, data integration tools from same vendor
Efficiencies can be gained at the intersection of middleware and data integration
Data warehouse project leads to data mashups in D.C.
Understanding the latest data integration strategies and technologies
Interest in open source data integration technologies growing
Understanding key data integration trends and business drivers
Informatica acquires complex event processing vendor Agent Logic
Informatica offers data integration software by the hour on Amazon EC2

Business intelligence technology platform
BI project management tips for implementation success
Oracle, SAP and IBM top Forrester's corporate performance management rankings
Microsoft details self-service business intelligence, data warehouse releases
Birst takes SaaS BI out of the cloud, battles data security fears
Hurdles for SaaS BI vendors include data integration, low recognition
IBM launches private analytics cloud
Atlanta YMCA turns to SaaS BI software over 'complicated' Cognos
Choosing BI software: Use your ERP vendor or go with third-party BI?
Data integration for Software as a Service business intelligence software evolves
SAP inks another partnership to enhance NetWeaver BW

RELATED GLOSSARY TERMS
Terms from Whatis.com − the technology online dictionary
corporate performance management  (SearchDataManagement.com)

RELATED RESOURCES
2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems
Search Bitpipe.com for the latest white papers and business webcasts
Whatis.com, the online computer dictionary



Search and Browse the Expert Answer Center
Search and browse more than 25,000 question and answer pairs from more than 250 TechTarget industry experts.
Browse our Expert Advice

About Us  |  Contact Us  |  For Advertisers  |  For Business Partners  |  Site Index  |  RSS
SEARCH 
TechTarget provides technology professionals with the information they need to perform their jobs - from developing strategy, to making cost-effective purchase decisions and managing their organizations' technology projects - with its network of technology-specific websites, events and online magazines.

TechTarget Corporate Web Site  |  Media Kits  |  Site Map




All Rights Reserved, Copyright 2005 - 2009, TechTarget | Read our Privacy Policy
  TechTarget - The IT Media ROI Experts