michelangelus - Fotolia

New models of processing stalk big data analytics applications

Operations and big data analytics applications are beginning to blend, causing changes in data strategies, Mark Madsen tells a TDWI Boston Chapter meeting. This is part two of two.

As software architect, CTO, CIO and other roles, Mark Madsen saw the birth of modern data warehousing firsthand. Now, as president of consultancy Third Nature Inc., in Portland, Ore., he helps guide projects focused on big data analytics, using the diverse new tools that come along for the ride. SearchDataManagement caught up with Madsen recently, just before he addressed a Boston Chapter meeting of TDWI. This is part two in a two-part interview. Read part one about shifts in big data analytics applications.

There seems to be new interest in connecting analytics with operations. That's often been called "operational intelligence." But there seems to be a lot of new things going on there, at least from a data manager's perspective.

Mark Madsen: When we are talking about operational intelligence in the new tech and big data markets, it tends to fall into the real-time space. I don't mean hard real time like financial transactions in low milliseconds or in nanoseconds -- it can be hundreds of milliseconds up to seconds and possibly into minutes.

Mark Madsen, president of consultancy Third Nature Inc.Mark Madsen

Now, that world, it is outside of every piece of technology we in the BI world have ever worked with. We are talking about messaging backbones, message queues, streaming infrastructure -- and the only way you talk to these things is through APIs, services. It is also tending to come out of customer-facing systems, typically Web apps.

That means a different set of tooling and infrastructure. The problem is that data infrastructure over the years in enterprises has moved very far away from having programmers on staff.

It is not easy to grapple with so much change. But there was a time when developers were in there in the big data analytics applications teams. Are they coming back?

Madsen: Yes, I think that is the case. And for a lot of companies, it is about augmenting what they have with a lot of new capabilities. Spark, Hadoop and the other things are about algorithmic processing of data in addition to transformation work. Just as importantly, they are about executing the code where the data is actually stored.

This is one of the major changes we are seeing. How does the data staff go about relating to it?

Madsen: It is a very different model of processing. It leads you to very different architectural decisions. The moment you introduce these kinds of capabilities, you introduce change. Suddenly, you have to start thinking again. Besides thinking like an architect or a designer, you have to have some level of understanding of software development. At a certain point, I got involved in all sets of problems around which my data warehousing architecture didn't work. The last seven or so years have often been about unlearning.

Next Steps

Read our interview on CDOs with Zhiwei Jiang

Share Joe Caserta's take on Hadoop maturity

Catch up on data parallelism with Curt Monash

Dig Deeper on Enterprise data architecture best practices