Data management professionals should ditch the "batch mentality" and start focusing instead on real-time data processing and analytics, according to Ted Hills, an enterprise information architecture executive at Bank of America Corp.
Typical enterprises collect data all day through online transactions, retail outlet sales, marketing efforts and more, said Hills, who spoke last week at Composite Software's Data Virtualization Day event in New York City. But in most cases, he continued, the practice of running overnight batch jobs to process that information is no longer sufficient in today's fast-paced business climate.
Whether you're doing analytics or any other kind of data processing you should process data as quickly as possible.
Ted Hills, architecture executive, Bank of America Corp.
"The value of data is highest immediately after it's created [and it] only makes sense that data should be moved, processed and used absolutely as soon as possible after its creation," Hills said. "Whether you're doing analytics or any other kind of data processing you should process data as quickly as possible."
Hills, the leader of the information architecture review board at Bank of America, has developed both batch and real-time systems and says that real-time data processing has improved the way the global financial services firm does business. For example, the company uses real-time data processing to track the progress of hurricanes and make sure that ATMs near evacuation routes are stocked with cash.
"If we have to wait until the end of the day to restock those ATMs with cash, we're going to have a lot of unhappy customers," he said.
To illustrate his point further, Hills recalled the beginning of the mortgage crisis that has rocked the world economy. People think the mortgage crisis began in 2008, but Hills remembers two days in August of 2007 when brokerage house Merill-Lynch couldn't sell any mortgage-backed securities.
"What happened is that the market value of these securities and securities related to them dropped dramatically quickly [and] nobody reacted to that fast enough," he said. "Had they been able to realize what was going on in the marketplace at the time [instead] of when we revalue mutual funds at the end of the day, they could have reacted more quickly, and Merill-Lynch would not have been bought by Bank of America."
More reasons to focus on real-time data processing
Switching from the batch mentality to a real-time culture can be a challenging process -- and it may require a significant up-front investment -- but there are compelling reasons to do so, Hills said.
One of the reasons centers on the idea that technology breeds impatience. Technology has evolved to the point where a person can order a book on Amazon.com and have it on their doorstep in less than 24 hours. A person can sign up for a new cell phone and make phone calls to anywhere in the world in a matter of minutes. A person can order a movie from Comcast and have it playing on their television in seconds.
So why should a technology executive or a CEO have to wait overnight to gain valuable insights from their own organization's business information? According to Hills, they shouldn't.
"We technologists have helped create a culture where we expect everything instantly," he said. "And we should expect everything instantly because the technology is capable of giving us that."
Real-time data processing can also minimize or completely eliminate the chances of "batch waterfalls," failures in overnight processing jobs that cause bottlenecks and result in further delays. This can be especially problematic for global organizations that do business with customers in distant time zones.
"If that failure isn't fixed and the batch stream restarted quickly enough, that last batch job might not finish by 6 a.m. the next morning, and there will be a 24-hour delay in the availability of information about the previous business day," he said.
Besides, Hills added, the concept of "overnight" means very little to most businesses today. Internet shoppers have no concept of closing time and, as a result, useful information flows into an organization at all hours of the day.
"Even if you’re a U.S.-centric business, if you support Internet customers, overnight has no meaning anymore," he said.
Overcoming the barriers to real-time data processing
One of the biggest barriers to adopting a culture of real-time data processing is that most IT staffs tend to be steeped in the batch mentality and lack the skills and experience necessary to develop real-time systems.
Getting IT personnel to change their attitudes toward real time requires an all-out effort that emphasizes the need to process data as soon as a record changes or transaction occurs, Hills said.
"The batch mentality says that I'm going to start [processing at 6 p.m.]," Hills said. "The real-time mentality says that every time a piece of data changes, I'm going to process that change."
Organizations that lack the necessary skills and experience should consider bringing in "just a few" outside consultants or new employees who can jump-start the process. Their goal, Hills said, should be to inject real-time thinking into the group.
A lack of executive buy-in is another major obstacle to adoption. Hills said it's up to data management professionals to sell company higher-ups on the value of real-time systems.
"You need to talk to your executives about the need for real-time processing, get their buy-in and get that message out," he said. "It's really about culture change and about shifting the cultural values of the organization to let people know that batch systems are great, but real-time systems have higher value."
Another big factor standing in the way of real-time or near-real-time adoption is "the myth" that real-time systems are more expensive than batch systems.
For organizations with no real-time infrastructure or skill sets in place, implementing real-time data processing will require a hefty initial investment, Hills said. But over time, he added, the costs tend to even out.
"Once you pass the learning curve, the cost to implement a real-time system and the cost to implement a batch system are actually the same," Hills said. "And if you realize that real-time systems deliver more value than batch systems because they deliver data sooner, then why would you ever build a batch system?"