This article originally appeared on the BeyeNETWORK.
How can so many magnificent minds be so confused? Despite this collective brilliance in data warehousing, there still seem to be moments of dim-wittedness. The furor involves the handful of innovative appliance (pre-configured hardware with packaged database software) vendors that have the entire market tied in knots. Chaos reigns and everyone is fuming!
The dilemma is this: Where do appliances starting below $150,000 fit in the data warehousing universe where built-to-requirements solutions typically exceed a few million dollars per system?
Five years ago when data warehousing turned red hot, innovators ignited the race to cash in with pre-configured solutions. Euphoric business press trumpeted each new arrival and professed the legitimacy of the appliance market. None, however, detailed where each fits or what criteria to apply in the selection process. What’s even worse is that in their report, Business Analytics Appliances Are Here to Stay, IDC declared that the data warehouse appliances are here to stay, while Gartner in their Hype Cycle Report for Data Management, 2006, shows the data warehouse appliances sliding down the slope of its hype cycle chart into the trough of disillusionment.
Buyer beware! Given the strategic role of a data warehouse in the enterprise, the stakes are high. Diligence in internal requirements analysis and vigilance of vendor claims are critical steps in making the right choice.
Because of our cultural addiction to simplicity – convenience, predictability, speed and price – it would not have been surprising if innovators brought new functionality to the vast populations of users who lack IT skills and substantial budgets. Nor would it have been surprising if these innovators targeted the organizations that would never require the size, volume and complexity of analytical capabilities, elaborate reporting, and integration sophistication of global corporations. But that was not the case. There are no protected territories.
Appliance vendors charged right into the most profitable multiterabyte data warehousing segments – the global trophy accounts – claiming them as their legitimate target market. Their claims of a new price-performance paradigm outraged established data warehouse vendors who invest millions in research and development to earn recognition as best-of-breed by serving the marquee global brands. The gloves came off fast. Sales teams have daggers drawn, and appliance vendors are hopping mad that they are being portrayed as niche solutions, data marts or single function data warehouses. Traditional vendors are responding with a more confrontational approach – attempting to discredit the appliance vendors by describing them as lightweight distractions. This is a full-scale war.
This article is intended to serve prospective data warehousing customers by illuminating common misconceptions and presenting unbiased selection criteria. It is a part of BI Results’ industry contributions and is not a sponsored series, nor is anyone paying for it.
The needs-based evaluation process matches requirements to a data warehouse solution, appliance or built-to-requirements solution. This process also speeds time to value. By embracing it, the vendors will realize reduced sales cycle complexity and expense. By adopting it, the IT teams will benefit from the prioritized requirements list and the support of the associated sponsor. As well, disciplined execution carries substantial learning rewards for all professionals and improved business results for the organizations.
Executives from DATAllegro, IBM, Netezza, Sybase and Teradata, as well as IT practitioners, generously provided their perspectives for this article. In addition, reference customers from IBM, Netezza and Teradata provided their input. The final result, however, was forged through a rigorous review process. The recommendations are tempered with a healthy dose of skepticism toward naturally exuberant sales teams.
We first illuminate the battlefield of contentious claims because it is necessary to sort fact from fiction. To help in this process, we suggest validation questions and a thought process. Next, we build a vision for the strategic nature of data warehousing and its critical value to the enterprise. Finally, we recommend selection criteria.
Anticipate and rapidly steer away from appliance vendors’ hyperbolic performance claims or traditional vendors’ evidence of failed appliance installs. While both sides’ claims are credible, neither can predict performance for your system footprint reliably enough to stake a company’s strategic advantage on it.
Appliance vendors flaunt impressive performance. Proofs of concept (POCs) and benchmarks substantiate their claims. References attest to results: lower cost, low risk, gradual migration, faster integration and reduced burden on IT resources. They claim market dominance by gifting customers the release from a draconian grip of arrogant systems giants. It would appear to be a hands-down win. Reference customers validated this appliance value proposition. However, the appliance customers made available for this article were small companies that fit the stereotype cast by traditional vendors.
The traditional vendors have not thrown in the towel yet. Instead, they have gathered evidence to discredit the superlatives. They list at least half a dozen examples of companies that slowed migration to appliances, or reversed the appliance course and recommitted to the built-to-requirements approach, or discovered performance substantially different in production. As might be expected, they question the candor and integrity of appliance vendor claims. For example, one large non-profit customer has shifted the appliance workload to a data warehouse and is looking for applications to use the appliance. Another prominent telecom customer is now using appliances for independent data marts and weblog analysis. Yet another one is using the appliance to offload the analysis of performance log data from their built-to-requirements data warehouse platform. Its destination is a data mart.
Guard against vendor battles. Be skeptical of all – POCs can be gamed for results, references are arguably positive and benchmarks can be lopsided. Validate risk elements.
Validating the risk basis for an investment is prudent advice anytime. In making a data warehousing decision, it translates specifically into the following actions:
Test Authenticity of Every Validation
- Credibility of Benchmarks
Unless a benchmark is conducted by a third party and the results are attested to by all participants, question credibility of process and results.
- Transferability of Reference Information
Clarify that the customer has no financial or other motivation to provide a reference.
- Details of Upgrade Process
Document upgrade paths to support growth of data, workload complexity, users and applications.
- Portability of POC Results to the Production Environment
How will POC results translate into the production environment? What’s the consequence to the organization? What’s the contractual impact to the vendor?
- Time and Resource Savings for Integration
What cost savings will result in each area: hardware, software, application integration and tools integration.
Our fundamental belief is that the $150 billion worldwide data warehousing market offers tremendous growth potential for all vendors. By staying intensely focused on needs, customers can avoid confusion. Frame those needs in the context of the data warehouse’s ability to serve the enterprise.
Build a vision, detail the plan, map requirements into three tiers and select the solution that best meets the identified criteria. It isn’t quite as simple to execute however. It will be necessary to assess well over a thousand critical success factors for business, operational silos and line of business operations. This tiered approach provides a directional road map.
Start with a Vision
What is the vision and expectation for the data warehouse? How does it compare to the industry’s view of current performance and future potential of data warehousing? Rate the goals relative to best-of breed tools that may be used by the competition.
A highly effective data warehouse is deeply embedded in the enterprise infrastructure. Neither a standalone entity, nor a destination, it is an integral part of the enterprise information infrastructure that perpetually transforms to revitalize the enterprise as its ecosystem (market, laws and competition) evolves. The IDC study referenced earlier in this article showed that 20% of data warehouse users expect the data warehouse to double in the next three years. Nearly half will integrate new capabilities such as radio frequency identification (RFID) that will increase the data warehouse size by another 25%. Data warehouses have never been known to decrease in size or user access. Nearly 70% expect user growth on business intelligence in the next 12 months. The more the data warehouse embodies the enterprise strategy, policies and controls, the greater its value to the enterprise – initially operational efficiency, and eventually competitive advantage. It is never complete nor completely optimized. It is a strategic tool to conduct analysis – past trends and correlations as well as predictive analysis, a defensive shield and a precision weapon. It is a living organism!
Engage either a Six Sigma black belt, a credentialed project manager or an equally qualified facilitator from the organization’s quality office to converge enterprise data warehousing priorities. Skills and experience are critical to earn respect of the participants. Business drivers will seek immediate results. IT will favor error-proof testing and evaluation. The right answer is fact based and tempered with the reality of the organization. It is critical to secure sponsorship and buy-in. The right leader is essential.
The recommended evaluation process is three-tiered:
Tier I: Data warehouse’s core functionality
Tier II: Scalability and infrastructure connectivity
Tier III: Application compatibility
Tier I: Data Warehouse’s Core Functionality
The core functionality is how the data warehouse manages data: indexing, partitioning, computational efficiency, loading, aggregation, integration, quality, and structure for granular access, access speed and schema flexibility; application compatibility: line of business and transactional processes; scalability for users, data, applications, functions, security, authentication, profiling, triggering responses, and supplying data based on a service-oriented architecture (SOA) and service level agreements (SLAs) to the right users according to role-based profiles.
Maintaining an edge in these core functions while optimizing the hardware and software performance is where IBM, Microsoft, Oracle, Sybase and Teradata have competed for decades. R&D labs develop and test methodologies to serve needs and gain an edge. The built-to-requirements systems provide tools and controls to manipulate all the dimensions (as listed earlier) within the data warehouse functionality or through interfacing with additional tools. To fully leverage these, however, the customers pay a premium for highly skilled IT staff.
In contrast, an appliance vendor typically masks some of these core functions or controls to make the appliance more standardized, easier to operate and less expensive, thus reducing the burden on IT resources for functions such as fast loading, aggregating, tuning data and running queries.
Assess the Fit
Assess the fit to determine what is required: a built-to-requirement solution or an appliance:
- Which option’s core functionality is a better match for vision and needs (immediate needs, 1-3 years, and 3-5 years).
- Can appliance-optimized controls enable functionality to exceed competition in precision, speed and quality of decisions.
If an appliance satisfies both criteria, an appliance is likely a less expensive option that will be faster to implement. On the other hand, the inability to adjust controls may be too restrictive for timely business decisions – fraud detection, inventory shortage, intrusion alerts. If the solution restricts the ability to make more precise choices, the tradeoff is ultimately competitive advantage.
Tier II: Scalability and Infrastructure Connectivity
The value of completely revitalized inner cores of real-time computational powerhouses now lies in supplying and extracting data between enterprise-wide line of business silos (CRM, SCM, ERP) and functional applications (billing, scheduling, etc.). Many specialized applications vendors have converged the best minds in mathematics, statistics, psychology and other industry-specific areas to develop decision models and analytical algorithms that rival the knowledge of top industry professionals. The automated decisioning is aligned with industry critical success factors. The
warehouse must transact data with these data models and structures fully and efficiently to power the enterprise.
Assess the Fit
Following are items to consider when deciding whether to implement a built-to-requirement solution or an appliance:
- How well does the solution interface with the existing applications environment?
- Are the interfaces tested and stable? Validation? Risk?
- Are the relationships between vendors mature? Validation? Risk?
- Do the migration teams have experience and a proven track record?
- What is the transition time and impact to operations?
- Line item costs, time for integration, test, deployment (in terms of cost and time.
- Is the solution “certified” or “validated” for the organization’s applications (middleware, ETL tools, and other infrastructure components)?
Certified means that testing and necessary engineering changes have been incorporated and documented.
Validated means that someone tried it and it worked, typically indicating lack of robust testing.
Tier III: Application Compatibility
Now let’s look at the third tier of data warehouse functionality, the application compatibility tier. It interfaces with analytical, reporting and optimization tools ranging in functionality from mining data to extracting data for analysis and reporting. This is the tier that often enables customized data delivery on role- and function-based permissions, formatted and encrypted appropriately for PDAs, pagers, mobile phones and e-mails. These applications conduct real-time and programmed analysis on data in business silos or the data warehouse, providing a basis for action and competitive advantage.
Whether installing a new data warehouse brand or an appliance, the integration need remains unaffected – neither circumvented nor standardized. The criticality of these applications requires disciplined integration and tuning.
Assess the Fit
For application compatibility, repeat the assessment detailed for tier II.
Determine Options Based on Needs
Although the assessment questions and methodology that we use at BI Results encapsulate more than 1,100 line item questions for data warehouse best fit assessment, even the top line view from the assessment suggested in this article will enable determination of a short-list of options based on needs.
Mistakes of reversing course are expensive for clients and vendors. Vendors should help the customers better map their needs. If the vendor’s system does not fit, the vendor should either fix the design or find the customer whose requirements fit. Customers should invest in the three-tier data warehouse analysis and find the best match – appliance or built-to-requirement data warehouse. Keys to success: let the quality office percolate business priorities and prioritize needs by schedule. Let integrated teams match core functionality, infrastructure compatibility and application optimization. Ensure that external vendors/partners are on board.
Download a starter list of evaluation criteria at , and please send feedback to Rajeev Rawat at RR@biresults.com.