Data quality management pitfalls: Three common mistakes to avoid

Digg This!     StumbleUpon     Del.icio.us

RELATED CONTENT Data governance and quality Are there data governance plans, templates or standard procedures? Should we buy data quality management tools or focus on policies? Resolving data ownership issues with external funders, organizations Where to find new academic resources on data quality best practices How to estimate customer data cleansing costs Data quality management for data warehouses What is high quality information? Data quality management tools: Where to get unbiased information Roles and salaries of data quality and governance analysts The role of data governance in unstructured data Data quality techniques and best practices Understanding five major enterprise information management benefits Gartner: Open source data quality software focuses on data profiling Poor data quality costing companies millions of dollars annually Are there data governance plans, templates or standard procedures? Should we buy data quality management tools or focus on policies? Where to find new academic resources on data quality best practices How to improve data quality on a tight budget -- a guide Data quality management tips and best practices Peachtree Data uses SAP BOBJ Data Services to clean up mailing lists Data quality software, including dashboards for non-IT users, gaining traction Data quality management software Gartner: Open source data quality software focuses on data profiling Informatica hopes to bridge the business/IT divide with latest release DataFlux demonstration of product data quality software Poor data quality costing companies millions of dollars annually Should we buy data quality management tools or focus on policies? How to improve data quality on a tight budget -- a guide Data quality management tips and best practices Peachtree Data uses SAP BOBJ Data Services to clean up mailing lists SearchDataManagement.com product directory library Data quality software, including dashboards for non-IT users, gaining traction

RELATED GLOSSARY TERMS Terms from Whatis.com − the technology online dictionary data  (SearchDataManagement.com) data governance  (SearchDataManagement.com) data quality  (SearchDataManagement.com) data scrubbing  (SearchDataManagement.com) fixed data  (SearchDataManagement.com) raw data  (SearchDataManagement.com)

RELATED RESOURCES 2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems Search Bitpipe.com for the latest white papers and business webcasts Whatis.com, the online computer dictionary

Answer from David Loshin, expert contributor and president of Knowledge Integrity, Inc.

Data quality management is not easy -- due to the size and complexity of organizations. There are a number of common mistakes that companies make during a data quality program implementation. Here are three specific pitfalls that can turn a data quality management project into a nightmare:

Data quality management mistake No. 1: Expecting the silver bullet

Some organizations think they can buy a packaged solution that will address all data quality issues and immediately make them disappear. This optimistic hope for a 'magic tool' is evidenced by how often people acquire a data quality tool as the first step in setting up their data quality program. Buying software before developing a program is indicative of a reactive environment -- and a misguided thought that data quality is a technology-driven solution. Too often, senior management gets the ideas that you can "fix" noncompliant data instead of eliminating the introduction of bad data in the first place.

How often has your organization bought a tool, only to have it still sitting on the shelf in its shrink-wrap months later? Although data quality tools are critical components of a data quality program, one must first question the motivation for purchasing a tool, then the process itself, and consider the improvement potential in terms of contributing to the effectiveness of the program.

Data quality management mistake No. 2: Not having the right expertise

There is often an expectation that as soon as a data quality program is initiated within an organization, there should be some visible improvement to the data. This is not so. Developing a data quality management program is a strategic undertaking. Its success depends on having both business and technical expertise. This is complicated by the fact that a large part of data quality management, especially at the enterprise level, is advisory.

Additional complexity is introduced by the close coupling of tools and methods to the process. Too often, the data quality manager is viewed as having responsibility for some data quality improvement action without necessarily having either the knowledge or authority to make it happen. The result is an overwhelming feeling that the size of the problem makes its solution unreachable -- and subsequently the team has no idea where to begin. The mistake occurs in not bringing in the proper expertise to help get the program off the ground.

Data quality management mistake No. 3: Not accounting for organizational culture changes

Even while attempting to improve the quality of data, we often forget that we must work within an organization's existing culture to achieve our improvement goals. No technology in the world will eliminate data quality problems, without understanding of how people's behavior allows the introduction of information flaws in the first place.

The evolution of centralized analytical data warehouses provides a good example. Data sets from numerous source systems are extracted, aggregated and transformed in preparation for loading into the warehouse. The data quality problems emerge when the data sets are merged together (perhaps customer names or account numbers are stored in slightly variant forms, data types might not match on similar columns, values are missing or fields are incomplete). But, without the cooperation of upstream systems owners, data warehousing managers are often helpless to control the quality of incoming data. Stricter data quality needs at the data warehouse demand resource allocation by upstream managers. The problem is that their applications may not directly benefit from the desired improvements -- and this acts as an effective "disincentive" for upstream managers to cooperate.

How to avoid data quality management mistakes

Don't despair though -- knowing the common pitfalls of data quality management programs can help you avoid them. Here are some guidelines to keep in mind:

• Exploit the advisory role of data quality teams and use internal procedures to attach responsibility and accountability for data quality improvement to the existing information management authority.
• Don't forget training in the use of policies and procedures -- especially in the use of acquired tools.
• Hire professionals with experience in managing data quality projects and programs from the start. These individuals will be able to identify opportunities for tactical successes that together contribute to the strategic success of the program.
• Engage external experts to help jump-start the improvement process. This will reassure your team that your problems are not unique and will allow you to learn from others' best practices.

More information on data quality management

• Five steps for weaving data quality management into your enterprise data integration processes
• How to develop and maintain an enterprise data quality management strategy, with Larry English (podcast)