Web content optimization specialist Cognitive Match is increasing the speed of its reporting and analytics services by replacing Oracle’s MySQL with Calpont Corp.’s InfiniDB columnar database software, company
Cognitive Match, a London-based company founded in 2009, was using Oracle’s open source database management system (DBMS) as the backbone of its customer-facing Web content analytics operations.
The company specializes in helping its clients increase click-through rates and improve sales by instantaneously matching individuals with the Web-based content and advertisements likely to interest them most. Cognitive Match also analyzes and reports on the behavior of website visitors in an effort to help its clients better understand their respective customer bases.
The growing startup found that MySQL regularly buckled under the extremely data-intensive workloads necessary for reporting purposes, said Cognitive Match chief technology officer Andy Lucas.
“We were trying to join across very large tables with tens or hundreds of millions of rows in some cases, and MySQL simply wasn’t working for us,” Lucas said. “We tried the normal approaches [like] adding more memory and then we tried adding indexes. But the problem was that in some cases, even though [we added] indexes, MySQL just [wouldn’t use them]. Ultimately, for an analytics business, this solution wasn’t going to scale.”
That realization prompted Cognitive Match to change its approach, Lucas said, and the firm began looking into the newly emerging world of high-performance columnar database technology. Typically used in business intelligence or analytics applications, columnar, or column-oriented, databases store data vertically in table columns instead of in rows. Analysts say the columnar approach minimizes the time it takes to read information, resulting in potentially faster query times.
“We have two distinct platforms really,” said Lucas. “We have the production platform on which we do the real-time optimization and we have the analytics platform. The analytics platform takes all the data from the production platform and helps us do research and produce new reports for customers.”
A columnar database software evaluation
Determined to speed things up on the analytics side of the organization, Cognitive Match’s IT team evaluated columnar database systems from Calpont and Infobright, a Toronto-based provider of data management software.
“We looked at Infobright, but the cost per terabyte didn’t really speak to us because we do have a large amount of data and in some cases, we can’t aggregate that data,” Lucas said. “For that reason, we chose [Calpont’s] InfiniDB rather than Infobright.”
Lucas said his team was also drawn to InfiniDB because the open source software could be deployed with relative ease.
“The feedback that I got from our data analysts was that [InfiniDB] was mostly 'drop-in,'” Lucas said. “We simply changed the MySQL table type to the Calpont table type, and aside from having to change a few columns, it was a minimal effort for us to do the migration.”
While Lucas has been pleased with InfiniDB, the CTO said he would be happier if Calpont provided him with more information about its plans for the future.
“I’m not really sure about features sets with regard to replication or sharding,” Lucas said. “More information about what their development roadmap is and how they can help us scale beyond a single database on one box [would be helpful].”
In a blog post from last April, Jos van Dongen, principal consultant with Netherlands-based Tholis Consulting, said he believes that Calpont’s approach to indexes could use some improvement.
“Calpont uses the 'no indexes needed' as one of the key benefits of the product; I tend to disagree on that one,” he wrote. “It's nice that you don't need to explicitly specify indexes, but when a DBMS doesn't support any constraints at all, well, that's plain ugly.”
Lucas said MySQL is still in use in the production Web optimization environment, but InfiniDB is now serving as the foundation for the company’s analytics programs. Cognitive Match went live with InfiniDB earlier this year and, as a result, the firm has greatly increased the speed of its reporting processes while offering a wider range of queries for its customers.
Closing advice from a columnar database user
According to Lucas, current and potential MySQL users need to be aware of the limitations of the technology when dealing with exceedingly large datasets. He said this is especially true in situations where the incoming flow of data is essential to the business and cannot be turned off.
“Unless you have a strategy to move beyond MySQL,” Lucas said, “you’re going to be in a lot of trouble because it’s going to be highly disruptive to your business.”