Sergey Nivens - Fotolia

Get started Bring yourself up to speed with our introductory content.

DataOps is more than DevOps for data, Delphix CTO says

Data operations is young compared to DevOps, but it is increasingly used as part of projects that put data at the center of development. Here, Delphix CTO Eric Schrock makes observations about the trend.

Eric Schrock worked on the developer side with debuggers at Sun Microsystems and on the data side with distributed data management systems at Oracle. That background helped Schrock prepare for the age of DataOps.

Now, as CTO at Delphix, Schrock is in a good position to talk about how this marriage of DevOps and data management practices can be applied in organizations that want to make data-intensive innovation happen. In this Q&A, Schrock discusses the importance of DataOps today: Essentially, the problem to be solved is data delivery, he said. It's about people, practices and breaking apart departmental silos as much as it's about technology.

How should we look at DataOps in light of traditional software development methodologies?

Eric SchrockEric Schrock

Eric Schrock: You can look at DataOps as an analog of DevOps. A lot of DevOps was about solving a people problem. You needed technology to enable it, but at the end of the day, you had these different groups -- development and operations -- that weren't really working together and collaborating and creating the type of feedback loop that allows you to move fast in terms of building and delivering software.

DataOps has a similar challenge, and while it's gaining momentum, it's definitely still nascent.

I look at it as a question of aligning people, process and technology around the end-to-end delivery of data. DataOps requires you to look at all the pieces needed to capture data; secure data; to curate, prepare and move data; and, finally, to deliver it, giving end consumers not just access to the data, but actual control over it. DataOps is really about the flow of information in the data and how you manage it in a self-service fashion.

How will DataOps play out differently in different use cases?

Schrock: Truly, codified approaches are still being developed. For now, everybody has a different pattern and a different set of practices around DataOps. Some of them are geared more toward analytics or AI or ML [machine learning]. Some of them are geared more toward application development. But we're starting to see practices and techniques start to come together and start to try to solve problems that we see with data in the enterprise.

Do you think we will see these processes adapted broadly?

DataOps is really about the flow of information in the data.
Eric SchrockCTO, Delphix

Schrock: Everyone's becoming a software company. It doesn't matter if you deliver pizzas or you run a taxi service today; you've got to figure out ways to use data to deliver new capabilities and insights. It all comes with competing in the market.

The challenge is how to get data everywhere it needs to be in the enterprise. IT [methods] just aren't cutting it. They're too slow, they're too fragile, they're too insecure. So, something has to give. Too often, when something gives, bad things happen: data breaches or poor application quality or stalled projects -- things like that. DataOps is still developing, but at the end of the day, what it's about [is] looking at not just technology, but also the people and process. DataOps is about finding ways to really manage that end-to-end delivery and lifecycle of data and do it in a way that's fast and efficient and secure.

Automating data delivery seems to be a major focus of DataOps. Is that fair to say?

Schrock: Yes. It's great that refreshing a database now takes three minutes instead of three weeks. But if you still have to file an IT ticket and wait for somebody to get back from lunch and click the button and be alerted by email when it's done, that is just not something you can build automation around or [connect with] a continuous integration pipeline or scale to support hundreds of test runs a day.

It used to be about the information security group doing this part, the DBA [database administrator] team doing another and the storage team doing still another part. Now, with DataOps, you start to think in terms of how you get data to the people who need it, while ensuring it is secured. When there is one set of people who are focused on making that happen, it brings together the managers and the consumers of that data.

Dig Deeper on Data warehouse project management

Join the conversation

5 comments

Send me notifications when other members comment.

Please create a username to comment.

How does your organization use DataOps methods?
Cancel
Rightly said, it's a set of practices. The goal of DataOps is to bring rigor, reuse, and automation to the development of data pipelines and applications. DataOps can help data teams evolve from a wild west environment that spawns data silos, backlogs, and endless quality control issues to an agile, automated, and accelerated data supply chain that continuously improves and delivers value to the business. Wayne Eckerson writes how it's the underbelly of modern data pipelines: https://www.eckerson.com/articles/diving-into-dataops-the-underbelly-of-modern-data-pipelines
Cancel
Thanks for your comment,  Fahadhazara. We'd be interested to know any questions you'd like our reporters to ask on your behalf of vendors, experts and users on the topic of DataOps. - Jack
Cancel
Is Delphix embedding any ML or AI engine, or does it integrate with?

Cancel
Delphix does not embed any AI/ML today, but we have many customers that leverage Delphix to quickly deliver training sets to their AI/ML and accelerate their AI/ML development. 
Cancel

-ADS BY GOOGLE

SearchBusinessAnalytics

SearchAWS

SearchContentManagement

SearchOracle

SearchSAP

SearchSQLServer

Close