Q
Problem solve Get help with specific problems with your technologies, process and projects.

Optimal data integration performance during the data explosion

A TDWI survey found that 55 percent of respondents expect their company's data volumes to rise more than 25 percent in the next year and a half. With this pending data explosion, it seems that merely boosting processing power is an insufficient strategy for maintaining proper data integration.

A TDWI survey found that 55 percent of respondents expect their company's data volumes to rise more than 25 percent in the next year and a half. With this pending data explosion, it seems that merely boosting processing power is an insufficient strategy for maintaining proper data integration (of course, management disagrees). If you had to outline basic components for maintaining optimal data integration performance in the face of the data explosion, something that could easily understood by management, what would they be?
I'm abridging what could be a protracted answer to a complex question. But you're right; it's not all about the addition of linear processing capabilities.

Indeed, there a few different dimensions to consider when planning to scale your BI environment. For instance,...

there is the "mixed workload" factor -- query processing versus ETL. Adding additional processing power and memory is the most effective means of addressing incremental growth of data. It's important to realize however, that there's a point at which the physical table structures -- not to mention the physics of disk I/O -- will prevent the linearity of queries (e.g., in order to support x percent new data, add x percent new storage and CPU). Because ETL is much more I/O intensive than standard query processing of most BI environments, simply adding more storage and CPU can be impractical. Bottom line, you can't just throw more horsepower at more data.

 

This was last published in June 2006

Dig Deeper on Extract transform load tools

PRO+

Content

Find more PRO+ content and other member only offers, here.

Have a question for an expert?

Please add a title for your question

Get answers from a TechTarget expert on whatever's puzzling you.

You will be able to add details on the next page.

Start the conversation

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

-ADS BY GOOGLE

SearchBusinessAnalytics

SearchAWS

SearchContentManagement

SearchOracle

SearchSAP

SearchSQLServer

Close