This article originally appeared on the BeyeNETWORK.
Measuring and managing daily and intra-day business operations have become an important growth area for business intelligence and data warehousing. The marketing buzzword often used for this style of processing is real-time business intelligence. Real-time analysis gives organizations the ability to use business intelligence to rapidly detect business issues. It also allows organizations to quickly react to changing business needs and requirements.
As many published articles on this topic point out, right-time is actually a better term to use than real-time. While right-time recognizes business users’ needs for accessing more current information, it also signifies that this information does not have to be based on current real-time data. A data latency of a few seconds, minutes or even hours, can be acceptable from a business perspective.
Right-time means the latency of the business information delivered to business users must be matched to the business needs of those users. It needs to be correlated to how fast business processes and procedures can handle the information as well. It is pointless to invest a large amount of IT resources and budget for delivering information faster, if the business cannot deal with it fast enough.
In right-time business intelligence processing the time delay between a business event occurring and the business user taking action to resolve issues associated with that event is comprised of three elements. These three elements include: the time to collect the data about the event, the time to analyze the event data and deliver its results to the business user and the time required by the business user to act on the analytical results.
Currently, most right-time processing involves capturing operational events into a data warehousing environment. This also involves processing and analyzing these events as quickly as possible. In these situations, the decision making tasks of the business user can be improved by using rules-driven guided procedures and automated recommendation engines. These rules are defined by the best practices captured from experienced business users.
Close to real-time information is required, however, for certain business situations. An example of this is fraud detection. Several telecommunications companies are now capturing every call data record (CDR) into an operational data store (ODS) for analysis. This information is used not only for fraud detection, but also for tasks like optimizing network performance. The data volumes here are staggering, as is the cost of managing and analyzing this data. But the closer to real-time the data is, the more the business will ultimately benefit.
One issue with many of these near-real-time operational stores is that a large percentage of the data is usually no longer required after it has been analyzed. Only a subset of this data needs to be kept for historical purposes. As data volumes increase, companies will no longer be able to justify the cost of consolidating and analyzing large amounts of detailed operational data. Another approach will have to be found. The large volumes, which are associated with sensor networks that use technologies like RFID, may be the straw that breaks the camel’s back.
The application integration vendors are moving into the business intelligence field to address operational business intelligence scalability problems. Several of these vendors now provide capabilities to capture selected events in their business process transaction workflows for analysis. Some of these vendors also provide their own operational analysis tools and dashboards. Some examples of this are IBM (WebSphere Business Monitor), Microsoft (BizTalk) and TIBCO (OpsFactor). This approach makes the capture process more efficient and reduces data latency. Unfortunately, some required events may not flow through the vendor’s integration software. Most integration vendors, however, provide the ability to make the analytics produced available to external applications.
Another trend in operational and right-time business intelligence is to embed the business intelligence processing into business process transaction workflows. This is becoming more viable as business intelligence and data warehousing vendors evolve their products to a services-oriented architecture. This is also possible if vendors can provide external access to their business intelligence and data warehousing processes via web services. Data integration vendors, such as Ascential (now owned by IBM) and Informatica, were the first to do this. As a result, some companies are now using data transformation web services in their business transaction workflows. Business intelligence vendors are also heading in this direction as well. Microsoft SQL Server 2005, for example, makes Analysis Services available as a web service. Packaged application vendors like SAP are also placing strong emphasis on embedded business intelligence.
The ability to embed business intelligence processing in business transaction workflows allows operational events to be analyzed and processed in-flight, without having to first capture the events in a persistent store like an ODS. This in-flight analysis can also use traditional data warehousing and business intelligence to add additional information, as well as put the in-flight information into a business context. Embedded and traditional business intelligence can work together for in-flight customer scoring and promotion, for example.
Stream-based business intelligence analysis, another interesting technology here, is also developing. This approach queries and analyzes data as it flows through the network. StreamBase, founded by Dr. Michael Stonebraker, is one company that uses stream-based business intelligence. According to the company’s web site:
“The StreamBase stream-processing engine is a revolutionary new infrastructure platform for capturing, integrating, understanding and reacting to business events as they occur when time-based analysis and response are critical. Financial or telecommunications networks, external data feeds, and internal proprietary feed sources can generate tens of thousands to hundreds of messages per second or more, and StreamBase is designed to process, analyze and respond to those streams of real-time data right now.”
In summary, there is significant business potential in using right-time business intelligence processing, and this is a major growth area for the business intelligence industry. When business requirements call for close to real-time data, the data volumes associated with them often prohibit using traditional data warehousing and business intelligence. The solution is sometimes to employ embedded business intelligence approaches. Application integration and packaged applications vendors, rather than business intelligence vendors, are presently focused on this. Unless the two sides work together to provide integrated and cohesive solutions, the applications and vendors may become involved in a direct conflict.