When Loyalty Management Group (LMG) agreed to collect and query stock keeping unit (SKU) data for one of its clients, the U.K.-based firm knew it needed more "raw speed, raw performance, raw power" than its Oracle databases could provide, according to IT director Fiachra Woodman.
LMG helps its clients -- retail stores and consumer products companies -- administer customer rewards programs. Each time a customer swipes a rewards card at the checkout of a grocery store or pharmacy, the store collects the transactional data. LMG loads relevant customer data into its own databases for analysis, and LMG clients use the insights to get a better picture of their customers, to maximize sales and to measure the effectiveness of promotional campaigns.
Until recently, LMG collected and analyzed just transactional data -- how much a customer spent and when – also called electronic point of sale (EPOS) data. But in 2007, Sainsbury's grocery stores asked LMG to also collect SKU-level data, a unique identifier for each product, so the grocery chain could get an even more detailed profile of its customers.
Suddenly, Woodman said, LMG was collecting a great deal of information on hypothetical customers like "Woody."
"Now, all of a sudden, instead of having to load two million transactions into our automated system every night, we're now loading 30 million every night," Woodman said.
The Oracle database and data warehouse just couldn't keep up, he said. Queries were taking hours. The company was also eager to launch its new Self Serve application, a Web-based interface that would allow LMG clients to query and analyze transactional and SKU data themselves, which also required more database power. "We needed something that was faster," he said.
LMG considers 'less traditional' database vendors
LMG began evaluating "less traditional" database vendors that offered massive parallel processing (MPP) capabilities, Woodman said, including DATAllegro, Netezza, SAS Institute and Kognitio, as well as reconsidering Oracle. MPP is the partitioning of databases so that large amounts of data can be queried at one time, or in parallel. It is a particularly useful technique with relational database management systems.
LMG eventually whittled the group of vendors down to three -- Netezza, Oracle and Kognitio – and kicked off a proof-of-concept competition in April 2007. LMG gave each vendor two years' worth of its actual data to bring into their internal data centers. They were asked to run a series of queries -- typical of the ones LMG clients might ask -- against the data so that LMG could assess each vendor's capabilities to deliver what it had claimed on paper.
After two days of testing by each vendor, Kognitio's data warehouse appliance, WX2, was the clear winner in terms of raw speed, Woodman said. Compared with the performance of the Oracle database and data warehouse that LMG had been using, "We saw queries drop from around 60 minutes down to about 60 seconds," he said.
LMG also wanted to test each database's robustness, so it turned for advice to WinterCorp, a Waltham, Mass.-based database consultancy. WinterCorp helped LMG set up a series of tests to evaluate how each vendor handled data clustering, partitioning, compression, indexing and optimization. Woodman said that the clear winner, again, was Kognitio, which proved it could scale as LMG's business grew.
Data warehouse appliance powers self-service app
Woodman officially chose Kognitio's WX2 to replace LMG's Oracle databases in mid-July 2007 and informed his executives later that month. "I prepared my management and said, 'We're taking a bit of a chance on this,' " he said.
"They are a relatively small company [and not very well known]," Woodman told his executive team, but he assured them that Kognitio had performed well in the proof-of-concept and boasted a dedicated team of developers.
The staff at Kognitio, led by president and CEO Roger Llewellyn, are passionate about what they do, Woodman said. "I was pretty confident that whatever problems we hit, we could work through together."
There weren't many problems to work through, as it turned out. After adding around 50 HP blade servers, LMG implemented and launched WX2 in November 2007, a process Woodman called uneventful. Sainsbury's began using LMG's Self Serve application powered by WX2 soon after, describing its performance as "jaw-dropping," Woodman said.
Since then, LMG has added around 25 (mostly U.K.-based) Self Serve clients, including Nestle and Unilever, with a total of around 600 users. That's far more customers than Woodman anticipated winning in such a short time. "We're much further along the road than we expected [to be at this point]," he said, an accomplishment which he credits – at least in part -- to WX2.