Guide to managing a data quality assurance program
A comprehensive collection of articles, videos and more, hand-picked by our editors
The desire to improve data quality in an enterprise is often accompanied by an urge to run out and buy a set of data management tools with a price tag that may run into six figures, if not more. But with the U.S. and global economies continuing to limp along, many organizations don’t have funds available to buy and deploy shiny new toys like that.
The good news is that data quality management is about more than just technology. It’s also about people and processes, and analysts say that by focusing on those two areas, organizations can make significant data quality improvements without spending much money at all. Here are some of their tips on how to improve data quality when budgets are tight:
Make sure employees understand the impact of poor data quality. Data quality problems usually begin with human error, according to Ted Friedman, a research vice president and information management analyst at Gartner Inc. in Stamford, Conn. For example, a call center employee misspells a new customer’s name when entering it into a customer relationship management application or a service technician makes mistakes when filling out an invoice form.
Seven ways to improve data quality, at a glance
1. Make sure employees understand the impact of poor data quality
2. Fill in the data quality holes in your business processes
3. Invest available money in developing data quality skills
4. Learn from the data quality successes -- and mistakes -- of others
5. Get your DBAs involved
6. Launch a data stewardship program
7. Identify and clean up your most important data
The data quality problems get bigger -- and bigger -- as those types of errors are propagated throughout companies via workflows, email and file sharing. To help put a stop to that, Friedman recommends that organizations regularly take the time to educate their employees about the impact that data quality mistakes can have on business operations.
“By and large, data quality issues are caused by people around the data just not doing the right stuff to make sure it gets entered and maintained in a high-quality way,” Friedman said. But he added that if you give workers a better sense of the importance of data and how their behavior affects its quality and, ultimately, its business value, “you can begin to effect some change” in the way they treat data.
Fill in the data quality holes in your business processes. Another low-cost way to improve data quality is to take a close look at key workflows and business processes. More often than not, organizations will find “data quality holes” that can be filled with relative ease, Friedman said.
Such holes might be found lurking within seemingly harmless tasks that often are done manually. For example, he said, a customer emails a file to a salesperson, who then manually “massages the data” and enters it into a business application -- introducing data errors in the process.
In connection with a data quality education effort, you might be able to avoid the errors by setting and enforcing data entry standards for the information that’s coming in, Friedman said. Another idea for improving the data’s quality, he suggested, “would be to potentially take the humans out of the process” by building a mechanism to automate the loading of the data into internal systems.
Invest available money in developing data quality skills. Organizations that don’t have the budget for buying a new data quality platform should consider investing some of the data management funds they do have at their disposal in training data-savvy business users on quality skills and best practices, Friedman advised.
“You need some resources -- some people with know-how,” he said. “You need somebody who understands really well the flow of data in the enterprise and how data is used in business processes.” Those employees could then help educate other end users about the downsides of bad data quality, he added.
For more on how to improve data quality
Find out how data quality metrics can bolster the business case for an increased data management budget
Learn about the impact of data quality issues on business intelligence projects
Read about a gas distributor’s three-tier approach to data quality management
Learn from the data quality successes -- and mistakes -- of others. Do a Google search for “data quality case studies“ and numerous tales of successful projects will pop up in the results -- stories that offer a no-cost way to learn tips and tricks for improving data quality in your organization.
Of course, data quality improvement has its roots in data quality mistakes, which can be instructive as well.
Jaime Fitzgerald, founder and managing partner of Fitzgerald Analytics Inc., a data management and analytics consultancy in New York, recalled the case of one company he worked with that ran into trouble with the way transaction files were being identified within various systems. The problems stemmed from a fundamental mistake: In one system containing 20% of the transaction records, a B value “meant something entirely different than it did in the other 80%,” said Fitzgerald, who didn’t identify the affected company. It eventually resolved the issue by creating a lookup table to correlate the correct values with one another, he said.
Get your DBAs involved. A competent database administrator (DBA) can be an invaluable participant in a data quality initiative in which budgetary constraints are a factor, according to Friedman. One way DBAs can help, he said, is by writing queries designed to expose patterns within data stores -- patterns that may point to outliers or other indicators of data quality problems.
“Some of the basics of data quality measurement and data profiling can be done using things like simple SQL queries,” Friedman said. For instance, a DBA could write a query that analyzes a database column and produces statistics about its minimum and maximum values or the number of times that each value found in the column appears there.
Launch a data stewardship program. Friedman also stressed the importance of appointing a data steward for the company as a whole or for individual business units -- someone who understands data quality issues and can mediate disputes when, say, the meaning of a particular word or a file naming convention comes into question. Data stewards typically come from the business side of an organization and serve as liaisons between the business and IT.
“I’ve seen organizations get a lot of benefit -- even in the absence of introducing technology or changing anything else substantial -- just by putting some data-quality-focused roles in place,” he said.
Identify and clean up your most important data. Many organizations fall into the trap of thinking that data quality is an all-or-nothing proposition, but it’s really an ongoing process in which every little bit helps, Fitzgerald said. He added that enterprises can get a lot of bang for their data quality buck by focusing on their most important data first and gradually expanding the initiative from there.
What should you start with? “It’s often your customer data,” Fitzgerald said. “But another way to look at it is to say that your most valuable data is the data that you most need to drive profits.”
And in the end, even low-cost efforts to improve data quality likely will need to be justified from a financial or business standpoint. Any approach to data quality management -- whether it’s on a tight budget or not -- should be solidly grounded in metrics, according to Friedman and Fitzgerald. That means data quality teams must determine ways to measure the results of their efforts -- for example, to gauge the impact that improved data quality is having on the business.
“You can’t just go to people and say that data quality is the right thing to do,” Friedman said. “You need to measure and you need to show with facts how good or bad the data is and how that is impacting the business.”