Best practices don't go out the window when new technologies come in, advises a long-time technology leader working to implement data-intensive cloud applications.
This goes for new database types just as it does for any new technology adoption, according to Marc Firenze, chief technology officer at Eagle Investment Systems LLC, a subsidiary of financial services firm BNY Mellon.
He said that the class of "NewSQL" databases that try to combine the best of relational and non-relational styles have promise, particularly for cloud applications, but they need to be carefully integrated into existing development processes.
"NewSQL DB technology -- it's like any new technology. You have to realize that you are introducing something new," said Firenze. "Whatever practices you use to do that -- keep them in place."
For Firenze and his team, the latest twist on data management architecture coincides with adoption of cloud architecture. He and his Eagle colleagues are implementing VoltDB, an in-memory NewSQL database that supports key relational ACID -- atomicity, consistency, isolation and durability -- traits for data across large-scale distributed clusters. Such distributed clusters are essential attributes of cloud computing.
Need for economical scaling
According to Firenze, Eagle as a company creates investment accounting software, portfolio performance management software and data management software for handling the data needed for portfolio management. A lot of Firenze's efforts help automate adherence to compliance requirements.
Because calculations are done for high volumes of data in increasingly shorter windows of time, a highly scalable cloud environment is required. While the setups still use relational databases, these established databases are not economical best fits for all the new data-intensive cloud requirements.
"To scale in the most economical manner, we had to introduce some newer technologies. We wanted to introduce a shared cache for common reference data, high volume OLTP data and to reduce the load on our persistent data stores. And that is where VoltDB came in," Firenze said.
The project with Volt began about two years ago, with about six months of research and more than a year of development. The first version of an Eagle risk-measuring application is now available, he said.
Putting rigor into the design
Often, new data architectures enter the enterprise via skunk works; developers download innovative databases that may be available as open source samples. That "garage-like" approach is not suited to financial software operations like Eagle.
The Eagle suite is enterprise-level software, requiring considerable precision, Firenze emphasizes. "We feel the rigor we put into the design and analysis is high," he said.
"My development team is over 300 people. It's not the kind of thing we take lightly. It's not like three guys in a garage fooling around with some code," said Firenze, who has established his perspective based on experience gained from his beginnings as a systems analyst during client/server computing days for State Street Bank all the way to his recent works as technology manager for Eagle's move to cloud computing.
Firenze said he strives to ensure his team works cooperatively and avoids disruption. That includes meetings with the business side.
"We bring together our database people, our business folks and our software engineers. And they all work on the design together in a collaborative fashion," he said. Application Programming Interfaces (APIs) become the central medium of exchange.
Focus on APIs
Firenze said Eagle's focus is around designing and building APIs. He feels that the premise of cloud architecture is to take advantage of SOA. By building APIs based on VoltDB into an overall or services catalog, Eagle can make a smooth transition to introduce this new technology. The rollout and integration of the NewSQL databases' APIs can be prioritized, he said.
Most of the effort now is in data infrastructure building that may not be too visible to its users. "Front-end investment analytics are cool. But we do all the stuff on the back-end. It is work that is critically important but not so cool," he said.
Starting slow can be a valuable tactic, he said, because new computing concepts can take some learning.
"Make sure you understand what you have," he said, referring to new data especially. "And make sure that you have a team that knows what you have."
Managers need to give guidance and set up an environment where new technologies can succeed. Managers also need to know when to give the developers some space.
"Sometimes it is best to just to make sure you stay out of their way and give them the chance to do it," Firenze said.
Clearly, technology advances and the continuous onslaught of data are causing changes in the way systems are built. These drivers have led technologists to consider non-traditional approaches to data management. But, as Marc Firenze's and Eagle's work bears out, some established best practices still help those changes take shape.
Find out more about VoltDB 4.0.