Home > Ask the Data Management Experts > Data integration Questions & Answers > Optimal data integration performance during the data explosion
Ask The Data Management Expert: Questions & Answers
EMAIL THIS

Optimal data integration performance during the data explosion

Jill Dyché EXPERT RESPONSE FROM: Jill Dyché

Pose a Question
Other Data Management Categories
Meet all Data Management Experts
Become an Expert for this site


Digg This!    StumbleUpon Toolbar StumbleUpon    Bookmark with Delicious Del.icio.us   


>
QUESTION POSED ON: 13 June 2006
A TDWI survey found that 55 percent of respondents expect their company's data volumes to rise more than 25 percent in the next year and a half. With this pending data explosion, it seems that merely boosting processing power is an insufficient strategy for maintaining proper data integration (of course, management disagrees). If you had to outline basic components for maintaining optimal data integration performance in the face of the data explosion, something that could easily understood by management, what would they be?

>
EXPERT RESPONSE
I'm abridging what could be a protracted answer to a complex question. But you're right; it's not all about the addition of linear processing capabilities.

Indeed, there a few different dimensions to consider when planning to scale your BI environment. For instance, there is the "mixed workload" factor -- query processing versus ETL. Adding additional processing power and memory is the most effective means of addressing incremental growth of data. It's important to realize however, that there's a point at which the physical table structures -- not to mention the physics of disk I/O -- will prevent the linearity of queries (e.g., in order to support x percent new data, add x percent new storage and CPU). Because ETL is much more I/O intensive than standard query processing of most BI environments, simply adding more storage and CPU can be impractical. Bottom line, you can't just throw more horsepower at more data.


Digg This!    StumbleUpon Toolbar StumbleUpon    Bookmark with Delicious Del.icio.us   


RELATED CONTENT
Extract, transform and load (ETL)
Integration competency centers centralize data integration projects
Data warehouse case study: Midsized insurer goes enterprise with data warehouse
Enterprise data integration quiz
Data migration planning: Key things to remember
The ETL process and MySQL
Gartner data integration Magic Quadrant 2007: Platforms, market expand
ETL tools and EDR tools: What's the difference?
ETL tools defined
ETL tools: What you do and don't want
Enterprise data integration software: EDI definitions you need to know

Enterprise data integration (EDI)
How to develop a sustainable data integration plan for business intelligence projects
Data integration certifications: Finding the value
Data mashups meet business intelligence: "Bashups" explained
Integration competency centers centralize data integration projects
Informatica to buy identity resolution software maker for $85 million
Understanding the top business intelligence data integration techniques
What's the problem with hand-coding scripts for data integration, anyway?
Integrating unstructured text into a structured environment
Unlocking and integrating unstructured data, with Bill Inmon
Managing unstructured data in the organization

Business intelligence software
Pervasive business intelligence: Are organizations really ready?
SAP acquisition opens up world of opportunities for Business Objects, according to execs
SAP unveils Microsoft Excel transformation tool, provides Crystal Reports to Business One customers
SAS buys revenue management firm to complement its business intelligence practice
On-demand business intelligence takes aim at the midmarket
Rankings of customer data mining market show shift to user-friendly tools
Business intelligence vendors, users embrace location-based analysis
Do business intelligence tools require a data warehouse?
Gartner: Business intelligence software market grew 13% in 2007
Business intelligence, performance management spending to top $57 billion

RELATED GLOSSARY TERMS
Terms from Whatis.com − the technology online dictionary
corporate performance management  (SearchDataManagement.com)

RELATED RESOURCES
2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems
Search Bitpipe.com for the latest white papers and business webcasts
Whatis.com, the online computer dictionary



Search and Browse the Expert Answer Center
Search and browse more than 25,000 question and answer pairs from more than 250 TechTarget industry experts.
Browse our Expert Advice

About Us  |  Contact Us  |  For Advertisers  |  For Business Partners  |  Site Index  |  RSS
SEARCH 
TechTarget provides enterprise IT professionals with the information they need to perform their jobs - from developing strategy, to making cost-effective IT purchase decisions and managing their organizations' IT projects - with its network of technology-specific Web sites, events and magazines.

TechTarget Corporate Web Site  |  Media Kits  |  Reprints  |  Site Map




All Rights Reserved, Copyright 2005 - 2008, TechTarget | Read our Privacy Policy
  TechTarget - The IT Media ROI Experts