Log data generated by operational systems was once routinely discarded, but now it forms a mother lode of information for many big data analytics efforts. Trying to mine that information can result in what some people describe as ''accidental architectures'' because of the way in which organizations tend to chance upon new uses for data and build a supporting infrastructure on the fly.
One of the early players in the log data management field was LogLogic Inc., which was purchased by middleware giant Tibco Software Inc. in 2012. Tom Yates, product manager for LogLogic at Tibco, said that having grown somewhat organically, the connections between log-based applications sometimes resemble spaghetti code; some centralization would be useful, he said.
"For operational intelligence today, you need to put your machine data in one place," he said, adding that Tibco thinks it's also important to be able to quickly move that centralized data to analytics engines.
Such thinking was behind new enhancements to the LogLogic line that integrate it with some of Tibco's own tools. In June, Tibco released LogLogic Log Management Intelligence 5.5, an upgraded platform for centralized management of diverse machine data feeds. Coupled with Tibco's event processing technology, the LogLogic software supports processing of as many as one million events per second, according to the company. Related high-speed filter-and-forward data routing capabilities are used to feed data from LogLogic into the Tibco Spotfire analytics engine, Yates said.
Log data management is made more difficult today because "the amount of data being created leads to a lot of noise," said Frank Brown, head of business development for cloud services at Versatile Communications Inc., an IT infrastructure services provider in Marlborough, Massachusetts.
Brown said Versatile uses Tibco LogLogic in its quest to harness operations data from cloud environments in which "everything is a measured service and compliancy is important."
Included, he said, are cloud-based health management applications that require effective compliance automation for privacy needs. Security requirements also create a need for careful systems monitoring, and the LogLogic software helps with that, Brown said.
In addition, the emphasis now is on immediately analyzing log data. That is different than in the past, when log data might have been reviewed leisurely. "Security information that's days old isn't good data," Brown said, citing LogLogic's speed of processing as a useful trait.
Although Brown said his group hasn't "dug in" to Tibco Spotfire analytics yet, he credited LogLogic 5.5 for "its ability to filter-forward based on granular policies that we set." He added that the ability to correlate data and events is another favorable point for the software.
LogLogic faces competition, of course. Increased interest in log data analytics and related compliance uses has caused analyst firm Gartner Inc. to place such systems in their own product category, which it dubs ''Security Information and Event Management.'' Besides Tibco, vendors such as EMC, Hewlett-Packard and Splunk can be found in that category.
Learn about some surprising uses for machine-generated data