This article originally appeared on the BeyeNETWORK.
The medical industry has been slow to adopt data warehousing. This phenomenon is unfortunate since integrated and historical data is especially needed in the medical field. In addition, data warehousing really lowers the cost of information and greatly accelerates information access. These are the two things that doctors, nurses and health care providers want. Clearly, data warehousing should be widely accepted by the medical industry.
So why has data warehousing been so slowly adopted in the medical industry?
There are undoubtedly many reasons for this slow pace. From a business perspective, the medical industry is much like a world of many small fiefdoms. There are the doctors’ offices, the hospitals, the insurance companies and the medical researchers. While each of these organizations has a piece of the pie, they also have their own agendas. Unfortunately, there is no General Motors of medicine that can focus and organize different information needs.
So maybe there is a good reason why medical data warehousing has not become popular. Perhaps medicine’s organizational structure just isn’t ready for data warehousing.
But there is an even deeper issue here. Compared to the data residing in organizations that build data warehouses, medical information is not as transaction-oriented. As you examine the data warehouses of banks, airlines and manufacturers, an important question arises. What lies behind these data warehouses? Essentially, transactions make up the daily fodder for the data warehouse. Most organizations that frequently use data warehousing definitely begin with transaction-oriented data.
But where are these transactions in the medical environment? In truth, there are relatively few transactions in the medical industry. Nurses take notes and conduct examinations. Hospitals perform procedures and monitor the episode of care. Doctors complete diagnoses, take more notes and make prescriptions. Throughout all of this, there are only a small number of transactions. Nearly all important information is locked up in text. Because of this, textual information forms the basis of most of the useful medical information, not transactional information. In the past, textual information been very difficult to capture and organize. Textual data is unstructured. There are no tables, attributes or records—the kind of structured data that data warehouse professionals are familiar with.
From a processing standpoint, unstructured data is similar to an octopus. Just as an octopus has many legs, suckers and tentacles simultaneously going in different directions, unstructured data appears complicated and impossible.
Recent technological advances, though, have made it possible to both access and understand unstructured data. Such advances now allow the text that the doctor writes about an examination to be integrated with text from another doctor. Later, the text that a new doctor writes about a diagnosis can be integrated with text that yet another doctor has written about a patient’s family medical history. Simply put, doctors can now integrate their unstructured information with other doctors and medical providers. The barriers that once existed have been broken by technology. This new technology is specifically designed for integrating unstructured data, and bringing it into the data warehouse in a meaningful way.
Perhaps medical information can be regularly integrated and shared today. Maybe the many benefits of data warehousing are finally available to an industry truly in need of help with their information.
Bill Inmon is universally recognized as the father of the data warehouse. He has more than 36 years of database technology management experience and data warehouse design expertise. He has published more than 40 books and 1,000 articles on data warehousing and data management, and his books have been translated into nine languages. He is known globally for his data warehouse development seminars and has been a keynote speaker for many major computing associations. Bill can be reached at 303-681-6772.