Business groups are empowered by knowledge -- and knowledge comes, in part, from having access to accurate and timely information. It is generally up to IT groups to make that happen.
But it doesn't always work out that way.
Quite often, business groups take matters into their own hands because they're not getting the data they need and business decisions are impacted. Perhaps their requests are too low in the IT group's project queue. Or maybe they simply haven't been able to effectively collaborate with IT to create a solution that meets their needs. Worse yet, if they are not a Fortune 500 company, there might not even be any IT resources available to them. Whatever the reason, their solution is to create a departmental data shadow system.
Data shadow systems are a group of spreadsheets and customized databases -- often Microsoft Access and statistical databases -- created by business groups to gather data for their users. Data shadow systems support business processes such as budgeting, forecasting or other reporting tasks. They include data from enterprise applications, data warehouses, external sources and printed or online reports.
Collectively, data shadow systems support many business users in an enterprise. Often, these users are also using the reporting from their enterprise application and the business intelligence (BI) setup from a data warehouse, but they do their "real" reporting and analysis from the data shadow systems.
This approach is both good and bad. Initially, it's good for the business group because they're finally getting the information they need to contribute to the health and success of the business. And getting information from a data shadow system is fast -- a business analyst has the ability to make urgent requests for information a priority. And, while techies may scoff at the use of Microsoft Excel and Access for data analysis, using these familiar tools can make business users comfortable and more independent.
So what's bad about data shadow systems? It depends on how they're used and integrated into the enterprise's overall BI architecture. One of the biggest weaknesses is if the data shadow system is a silo, it will promote the creation of data that's no longer consistent with other data being used in the enterprise. In order to achieve a high level of quality, data has to be viewed from an enterprise and holistic perspective. Data may be correct within each data silo, but the information will not be consistent, relevant or timely when viewed across the entire enterprise. To make matters worse, each report or analysis interprets the data differently, so even when the numbers start off the same in each silo, the end results will not be consistent.
Inconsistent, inaccurate data will certainly tarnish the image of a data shadow system for business users. But they'll also feel the strain of creating and managing their own departmental systems. Ideally, they should focus on their "real" jobs and spend less time on the care and feeding of the data shadow system. And when the people most experienced with using an undocumented shadow system move on to other jobs, those left behind have no tools to help them learn how it works.
Data shadow systems are a fact of life in companies of all sizes. While they don't offer an optimal solution from a technology standpoint, they have many business-oriented benefits that cannot be ignored. Keeping the needs of business users in mind, it's possible to replace or rework these shadow systems with solutions that dovetail with a company's overall data warehousing architecture.
About the author
Rick Sherman is the founder of Athena IT Solutions, a Boston-based consulting firm that provides data warehouse and business intelligence consulting, training and vendor services. In addition to over 20 years in the business, Sherman is also a published author of more than 50 articles, an industry speaker, a DM Review World Class Solution Awards judge and a data management expert at SearchDataManagement.com. Sherman can be found blogging at The Data Doghouse and can be reached at rsherman@athena-solutions.com.
