Why New Systems Fail: Theory and Practice Collide
Chapter 1
Types of Failure
Not all failures are created equal and there certainly are degrees of failure. This book defines four major types of system failures, one of which may not become apparent until months after a new system has been activated. Each case study is classified in terms of the following failure scale:
• The Unmitigated Disaster
• The Big Failure
• The Mild Failure
• The Forthcoming Failure
The Unmitigated Disaster
The most egregious failure occurs when an organization spends millions of dollars implementing a system and misses deadlines repeatedly. It ultimately junks the new system for a different one altogether or reverts to the legacy system. Relationships between consultancies and clients are often severed. Lawsuits in such cases are not completely out of the question. Fortunately, these abominations are atypical.
The Big Failure
These types of failures are less severe but more common. Perhaps an organization initially budgets $2M and one year on an implementation and ultimately spends $4M over the course of three years, getting much less functionality than expected in the process.
The Mild Failure
Very often, a system failure is so mild that one can hesitate to even call it a failure, especially relative to the two types just mentioned. By comparison, these are rousing successes! For the sake of consistency, however, this book uses the term "failure." An example of the Mild Failure is the company that initially budgets $2M and a year on an implementation and ultimately spends $2.2M over the course of fifteen months, getting slightly less functionality than expected in the process.
The Forthcoming Failure
Sometimes a system failure is not immediately apparent. At first, this notion may seem perplexing. If an organization has met its goals with respect to both its budget and deadline, then how can it consider the system a failure?
Budget and deadline are only two criteria for a system failure, as the statistics at the beginning of the chapter illustrate. The answer to this question lies within the organization's data, documentation, processing, and people. Examples of latent failures include the following:
• The implementation team has made a key mistake that will come back to haunt the organization down the road.
• End-users may not completely understand the system and, as a result, make significant errors or revert to "old ways," negating one of the major benefits of the new system.
• The organization is vulnerable to employee attrition on two fronts:
o End-user documentation is deficient and, if key staff members leave, their replacements will need significant time and/or training to do their jobs.
o Knowledge is not dispersed; only a few employees understand the system in sufficient breadth and depth.
In other words, these are failures waiting to happen. Organizations are not prepared for shocks to their systems.
A Prime Example of the Forthcoming Failure
I am reminded of an organization — Oates Healthcare — that activated its ERP in 2003 with a fundamental but unknown problem with the way in which it calculated employee overtime. No one identified this issue during setup or testing. Only when an ex-employee filed a lawsuit did the problem come to light, five years after Oates had gone live.
For Oates, fixing the problem in the system involved two things, one simple and one very difficult. The first merely entailed changing some flags in the system, allowing it to begin calculating overtime correctly from that point forward. However, checking those flags did not retroactively go back and recalculate overtime for all employees paid incorrectly over the past five years. A breakdown of those errant employee records is presented in Table 1.2:
Table 1.2: Breakdown of Payroll Records Requiring Analysis at Oates Healthcare
| Employees paid per year |
6,500 |
| Average types of pay per week per employee |
6 |
| Weeks per year |
52 |
| Checks per year |
338,000 |
| Payroll records per year |
2,028,000 |
| Years of data requiring analysis |
5 |
| Total records (over a five year period) |
10,140,000 |
The enormity of this task — recalculating employee overtime — was beyond the time and skill of Oates' existing end-users (arguably a failure in itself). Even if an internal super-user knew how to do this on over 10,000,000 records, he or she would not have been able to do it. For ad hoc analyses, Oates provided its end-users only with Microsoft Excel, a very valuable tool but one not nearly robust enough to handle a task of this magnitude. As a result, Oates hired Bishop Consultants to perform this task at considerable expense.
Had Oates' end-users properly tested the system prior to going live, it may have avoided the lawsuit. To be sure, it would have not have had to spend the time, internal resources, and funds on Bishop to fix the problem. Bishop recalculated employee overtime pay but Oates' end-users were not available during the remediation project. This prohibited Bishop consultants from transferring any knowledge during the error-resolution process.
Ironically, Oates did not learn from its mistakes. Despite the recommendation from Bishop, Oates did not seriously consider adding a more powerful reporting tool for end-users to conduct similar kinds of analyses (e.g., Crystal Reports, Microsoft Access, or Business Objects). Oates also failed to actively recruit more technical end-users to use these very tools, should it one day have decided to purchase them. If Oates encounters a similar problem in the future, then it will be at the mercy of external consultants such as Bishop once again.
Consequences of a Typical System Failure
The consequences for a failed implementation go beyond mere dollars and cents. Let's return to the P&P example. Bonham Software may forever have a tarnished reputation within the organization among both end-users and employees who actually do not even use the system on a regular basis. Bonham may always be known as "that system that screwed up payroll." Data could be lost or altered in such a way that it will be impossible to retrieve. Due to lack of training or documentation, employees' jobs may actually become more difficult than they were with P&P's legacy system.
For Bonham Software, as a company, the project was a disaster. Bonham now has a tarnished reputation in the industry resulting from this highly publicized failure. It may have difficulty collecting the hundreds of thousands in accounts receivable from P&P and lose key consultants. P&P may also refuse to provide a reference for its new partner.
As stated before, many types of issues typically haunt system implementations. While each of the case studies detailed in the book differs in terms of the way in which the organizations employed technology and even in the technologies themselves, there is considerable overlap among the issues encountered.
The Expectations Gap
Table 1.1 illustrates that system implementations typically leave many parties disappointed in the ultimate outcomes. Senior management expects the new system to be implemented smoothly, on time, and within the planned budget. Client end-users expect to learn how to properly use the new system and be self-sufficient when consultants leave. Consultancies expect strong client references. For each party, these expectations are often unmet, many times by significant degree.
Disappointments often give way to disasters. A less-than-inconsequential percentage of these projects have their plugs pulled mid-implementation. Organizations sometimes go live when they are wholly unprepared to do so. Issues abound and the benefits and cost savings once promised by the vendor and consultancy may be significantly less pronounced than what clients ultimately see. In retrospect, after system activation many clients opine that the new system is a far cry from what they expected when senior management signed the original contracts.
Risks for Mature Organizations
While this book focuses organizations implementing new systems, the content applies to the maintenance, enhancement, and support of existing systems as well. Mature systems can fail in several ways. First, successfully-activated systems often begin show signs of future problems. Second, a Mild Failure could easily become either a Big Failure or, in extreme cases, an Unmitigated Disaster. In other words, just because a new system goes live on time and under budget does not mean that an organization is out of the woods. There is still significant risk. Systems can and often do begin to experience major difficulties after even successful activation, attributable to:
• Key employee turnover
• System upgrades and the decommissioning of older versions of the application
• The introduction of additional functionality within a system
• Changes to business processes
• Acquisition of a company and the integration of additional legacy systems
• Unwise expansion
A Balanced Approach: Theory, Case Studies, and Examples
Throughout this book, theory and practice are given equal weight with respect new systems. Consider system testing for a moment. The book does not simply espouse the virtues of system testing; to do so would be facile. After all, all consultants and implementation teams intend to run proper parallel tests. How, then, should the importance of — and frequent missteps associated with — testing be illustrated? By drawing upon extensive examples and detailed case studies, the book manifests the essential questions that cause testing to be compromised. These include:
• What causes system testing to produce unexpected results?
• What are the effects of failed testing on the project's timeline, budget, and ultimate outcome?
• Most important, what specifically can an organization do from the beginning — and during — a project to promote accurate, timely, and comprehensive testing?
The examples and case studies in this book stem from actual system implementations but, for reasons of confidentiality, the names of the organizations, consultancies, and individuals have been changed. Specific names are not nearly as important as the lessons they provide. As we will see, many ostensibly different organizations face similar — if not identical — challenges implementing systems.
It's commonly said that one learns more from failures than from successes. To that end, this book's case studies and examples will examine in great detail system implementations that failed, identifying the specific individuals, decisions, and events responsible for the outcomes. This book contains eight detailed case studies:
