Many companies continue to rely on hand-coded scripts for data migration projects, but several trends are driving others to consider packaged tools for the job instead.
In fact, most organizations deal with data migration through hand-coded scripts, according to a recent survey by Towcester, U.K.-based Bloor Research. The survey of more than 700 Forbes Global 2000 organizations with data management project budgets above $1 million found that 30% use hand-coding for data migration, 28% use extract, transform and load (ETL) tools, 13% use application loading, and the rest use data quality tools, data profiling tools or other methods. The survey also found that most projects went over their schedule and budget -- due in part to not properly scoping the data migration effort.
However, according to Rob Karel, principal analyst with Cambridge, Mass.-based Forrester Research Inc., using the custom scripting approach for migrating data to today's data-centric applications leaves one fatal flaw.
"A custom script is not going to do anything about garbage data, it's just going to move it," Karel explained.
Since the goal of migrating data to new applications is better leveraging corporate information, starting off the new system with bad data is not ideal. Most initiatives requiring attention to data quality and governance will also benefit from an emphasis on data migration, Karel said. The data migration process is a good time to consider data quality issues, such as whether old data should even be migrated to the new system -- or whether it's outdated or inaccurate and ready to be retired.
New information management systems are also uncovering previously hidden data problems, Karel said. Projects such as operational business intelligence or implementing a service-oriented architecture (SOA) shine a spotlight on poor data management.
"As more initiatives get under way -- like SOA, information as a service and master data management -- that require trusted data and give visibility to poor data, more and more of these technology processes, like data migration, that touch the data are going to be audited, vetted and improved," Karel said.
That may be why vendors such as Redwood City, Calif.-based Informatica and Paris-based Business Objects S.A. have started talking up specific tools and methodologies for data migration. They may be slightly ahead of the curve on this trend, Karel said, since he hasn't yet heard of widespread adoption of the technology. But he is getting more questions these days from organizations wondering whether it's worthwhile using one of the new data migration tools or an ETL tool. He recommends they do.
"It's bringing in metadata lineage," Karel said, "and allowing you to create better audit trails to actually know what happened with the data, where it came from and what you did to it."
And while some enterprise applications such as SAP or Oracle now ship with tools to help migrate data to their platforms, some companies use additional tools to support the process, Karel said. They might use ETL tools for the more robust transformation capabilities but also use the migration utilities offered by the application vendor to load the new system's target schemas more efficiently, he explained.
Although migrating data to a new large application may be perceived as a one-time or infrequent process, data migration is certainly not going away, Karel said. The Bloor Research survey found that large enterprises completed approximately 4.5 major application implementations per year -- and that's not likely to change anytime soon.
"Companies are constantly trying to consolidate disparate instances of enterprise applications or legacy systems into faster, more maintainable, more secure environments," Karel said. "There's always improved technology, improved storage -- you name it. There are always reasons to move on to the newest technologies. Then you have things like M&A [merger and acquisition] activity that require this consolidation again and again."