The data warehouse is no longer the only focal point of the typical enterprise's data integration and analysis efforts -- and that's just one of the trends shaping the future of data virtualization technology,
The original idea behind the enterprise data warehouse was to give users a single source of reliable information for reporting and analysis requirements, said David Besemer, the CTO of San Mateo, Calif.-based data virtualization software vendor Composite Software Inc. But things haven't quite worked out that way.
"I bet nobody in this room can say that you have one single centralized data warehouse that meets all of your visibility and reporting needs," said Besemer, who spoke earlier this month at his company's Data Virtualization Day conference in New York City. "And yet, that has always been the stated role of the data warehouse."
Besemer said many organizations today have multiple data warehouses in place, while at the same time, data virtualization, change data capture and messaging techniques have emerged to give users an alternative to traditional information consolidation techniques.
The attitudes of data management professionals are changing too, said Besemer, who based his remarks on research from IT industry analysts and conversations with current and potential users of Composite data virtualization software.
"It used to be that we couldn't even talk to [data warehouse owners] about data virtualization because it was anathema to the mission," Besemer said. "But now that has changed a lot. Data warehouse practitioners are embracing virtualization as a way to extend warehousing and combine multiple warehouses."
Data managers now see the data warehouse as just one tool in a growing information management arsenal, he continued, and Composite has had to evolve its data virtualization software accordingly.
"We had to [expand] our hardcore technology and query optimization to also encompass things like best practices in governance and, of course, scalability and disaster recovery and redundancy," he said.
Cloud computing having an impact on data management
The growing acceptance of newer techniques for aggregating and analyzing information is just one of the key data management trends that Besemer sees taking shape in the IT marketplace. The CTO said other trends include the rise of cloud computing, the movement toward real-time data processing and the proliferation of data sources.
"Most enterprises are starting to look at how they can leverage cloud computing [and some are] leveraging it accidentally because they're using Salesforce.com or they're using some other cloud-based system that forces them to actually have data and processing in the cloud."
But cloud software users are just "starting to feel the pain" of cloud-based data being geographically separated from enterprise data.
"The typical place [this pain] shows up is when you have something like Salesforce.com and your internal financial system and, of course, Salesforce wants you to put all of the information in Salesforce and use their report," he said. "How many of you want to move all your SAP data to Salesforce? It's just not going to happen."
Besemer said today's most successful companies embrace the idea that information needs to move in real time. The idea, he explained, is to gain valuable business insights faster than competitors.
Companies are also processing more information than ever before, he said, and the types of data being processed are becoming increasingly complex.
"We've got new sensors, we've got machine-generated information, we've now got click streams from Web users, we've got [logs] from cell phone use," he said. “But we're also exacerbating this problem because we keep copying that information over and over and over again, whether it's through replication [or whether] it's through physical consolidation."
Organizations then need to take this huge volume of data being generated and then turn it into usable information.
"You don't just have to deliver it through a report," he said. "You've got to deliver it through five different BI tools and dashboards and you've got to deliver it to mobile users on their iPad and on their Android devices and on their Kindles."
Online travel agency uses multiple data warehousing technologies
Tony Gray, the director of business intelligence (BI) architecture and operations at a popular online travel agency, is one example of a data management professional currently using multiple approaches to data warehousing and analytics.
Gray's company runs an EMC Greenplum data warehouse for traditional financial reporting and analysis and recently went live with a cloud-based data warehousing and analytics platform from Chicago, Ill.-based software vendor Kognitio.
The travel agency is using Kognitio's cloud-based WX2 analytic database to capture and quickly analyze Web-based advertising data and information related to customer demographics and segmentation.
"Our display advertising [activities generate] a large volume of data that we just weren't able to load into our traditional data warehouse," Gray said. "Even when we loaded a subset, we weren't able to get the performance that we needed to make that data actionable."
The cloud service was relatively easy to implement, Gray said. The bigger challenge was getting some people in his organization used to the idea of outsourcing data warehousing and analytics functions.
"Even though from a financial perspective it made sense, there was a tougher battle internally to face in terms of why [we would] use a vendor for setting up an analytical environment instead of just eating our own dog food and doing it in-house," he said. "That was the bigger sort of paradigm shift for folks to wrap their heads around."