Health care giant UPMC is using data replication software to give doctors and hospital workers quick access to...
electronic medical records in the event of planned or unplanned database outages.
UPMC -- a Pittsburgh-based global health network that includes more than 20 hospitals, 400 doctor's offices and outpatient sites, a health insurance division and about 50,000 employees -- recently implemented Oracle GoldenGate replication software in an effort to quickly pull patient data onto a standby database located in a disaster recovery center.
The organization overcame some tough hurdles and learned some valuable lessons about implementing data replication software in the process, according to William Costantini, associate director of the integrated operations center at UPMC.
It's the little bumps that happen daily or weekly or monthly that knock patient care off balance.
William Costantini, associate director of the integrated operations center at UPMC
"The biggest [challenge] in this kind of project was semantics. When we started off we were all talking the same language but we didn’t really do a reality check [to make sure] that we had the same semantic meaning behind the words we were using," Costantini said. "But once we all sat down and ensured we were on the same semantic page, and also had a clearly defined goal of [the] criteria for success, we were all good."
Need for speed leads to data replication software
UPMC's IT setup includes a suite of about 14 Cerner Corp. medical systems applications that run on a production instance of Oracle Database. UPMC's disaster recovery center also has an instance of Oracle Database that is used as a standby when the primary system is down. UPMC's primary Oracle Database contains over 17 terabytes of data and typically supports about 4,500 concurrent users and 850,000 interactions each day.
"Cerner Corporation does medical information systems for everything from [inpatient] care to surgery, emergency department and bedside medical device software," Costantini explained. "They also do outpatient software."
Back in 2006, Cerner introduced a disaster recovery toolkit that promised to help speed up the process of switching over to the disaster recovery center during planned and unplanned outages. At that point, it took UPMC about two hours to get the standby Oracle database up and running with the most current patient information.
Cerner's disaster recovery toolkit helped UPMC speed the process to the point where it took just 45 minutes. But Costantini and his team believed they could do better and continually looked for ways to speed things up even more. Three years later, Oracle approached Costantini with its newly acquired GoldenGate data replication software.
"Oracle and Cerner and UPMC got together and started talking about how they wanted to do this new disaster recovery process," he said. "On the Cerner side, they were rewriting some of their DR toolkit to deal with the need to keep things in sync code-wise and middleware-wise. And then Oracle said, 'We have [GoldenGate] and it can actually keep both databases in sync.'"
Oracle bought GoldenGate Software in July 2009. UPMC decided to move forward with the project and began implementing Oracle GoldenGate in August of that year. By November, Costantini had a working model, which the company then began migrating into production. The team has been making continual improvements to the system ever since.
"Our initial go-live target was [to provide] fully automated read-only access in under 30 minutes," he said. "We came in at 21 minutes [but] after another two months of working with Cerner we got it down to 12 minutes."
Currently, Costantini said, the data replication software can synchronize the production and standby databases with only sub-second latency while Cerner's 724Access software enables UPMC to activate the standby database.
Costantini has been pleased with Oracle GoldenGate and Cerner's joint efforts but said he is continuing to work with the software giant to ease the pain of completing application upgrades.
"What we actually want to do is real-time or uptime code upgrades [with] some of the capabilities that GoldenGate gives us," he said. "Normally, a major Cerner code release upgrade takes about six to seven hours of downtime. But if it works the way we're painting the picture it could be done in 30 minutes."
From disaster recovery to business continuity
Earthquakes, tsunamis and other calamities don't happen very often. As a result, executives charged with funding a disaster recovery effort may be tempted to delay the initiative when budgets are tight. But this is an unacceptable attitude in the health care industry, according to Costantini.
That's why, when making the business case for the data replication project, Costantini chose to focus on the idea of maintaining business continuity. He said the key to getting the project funded was showing executives that data replication software would help minimize the minor disruptions that tend to pop up more frequently than an all-out disaster.
"In the health care industry, overall the mental picture has to change towards business continuity," he said. "It’s the little bumps that happen daily or weekly or monthly that knock patient care off balance."