More than 60% of IT and business professionals surveyed in early 2013 by London-based consultancy The Information Difference gave the data quality levels at their organizations a grade of "good" or better. That doesn't sound too bad at first -- but it means that more than one-third of the 210 respondents didn't think very highly of the quality of their data. And that isn't such a big surprise considering that more than 40% of the respondents said their companies didn't have enterprise data quality management programs in place.
And with all of the various forms of data flooding into organizations in these big data days, data quality needs are only getting larger -- and more complicated. In a January 2013 blog post, Forrester Research Inc. analyst Michele Goetz wrote that the concept of data quality "is at a crossroads." Data quality tools and processes are "steeped in an old way of thinking about and managing data," one built around the enterprise data warehouse (EDW) and traditional extract, transform and load (ETL) approaches to data integration, Goetz said. Now, she added, more and more data is bypassing the EDW and ETL methods, and data quality measures need to catch up.
In another post last August, Goetz noted that data scientists often can overcome dirty data as part of big data analytics applications. That changes the data quality game somewhat for IT teams, she wrote. But even in big data environments, data quality assurance has its place, Goetz said: "It's about data quality efforts that matter to outcomes."
We've published a variety of articles on SearchDataManagement that offer advice on developing and implementing an effective enterprise data quality strategy. In one, Andy Hayler, CEO of The Information Difference, delves more deeply into the findings and implications of his company's data quality survey. Consultant David Loshin also weighs in with a five-step plan for improving data quality and a formula for estimating the amount of time that data profiling projects will take. In another article, we catalog tips on getting a data quality management program off the ground. And we look at the issues involved in combining data quality improvement initiatives with data governance and master data management efforts. Hopefully, you'll find information that can help open eyes to the importance of data quality in your organization -- and aid you in managing the data quality plan that those open eyes approve.
This was first published in January 2014