New Orleans VA medical center restores data lost to Katrina

The destruction caused by Hurricane Katrina proved the survivability of medical records in digital format. Restoration of these records, however, is not necessarily easy, even with a sound disaster recovery plan. Veterans Integrated Service Network 16, the VA network for the South Central Region, faced many obstacles in making the records of the New Orleans VA Medical Center accessible again.

Operations at the 354-bed state-of-the-art facility in New Orleans ceased abruptly when its basement was filled with floodwaters from the levee breach. The basement contained many of the electrical switches for the hospital, as well as other core infrastructure components. With their destruction, the emergency generators became inoperable, and the hospital was evacuated and closed.

The administrative function and many patients were transferred to facilities in Baton Rouge; Houston; Jackson, MS; and Alexandria, LA. The Baton Rouge Clinic is an outpatient facility, and it was not physically prepared at the time to handle the IT requirements for the New Orleans VAMC's 36,000 patients. The electronic patient record (EPR) information system, therefore, was restored at the Houston VAMC.

The EPR database and software are the primary components of the Veterans Integrated Service Technical Architecture (VistA). It supports specialty disciplines, such as a RIS for diagnostic imaging. Every EPR at every VA medical facility is protected by a robust "recover all" program. As a result, it was possible to restore the database within two to three days of the closure of the New Orleans hospital.

VistA Imaging, the architecture that contains images for radiology, dermatology, ophthalmology, and patient photographs and scanned documents, is rigorously backed up but was not included in the recover all program. Both primary and backup data reside in separate, secure areas in New Orleans. On Oct. 6, 2005, the mean optical density (MOD) and ultradensity optical (UDO) discs-some 1250 in all-were retrieved and taken to Baton Rouge. A week later, Kenneth Allen, technical manager for VistA Imaging systems for VISN 16's 10 medical centers, picked up the discs and took them to Arkansas, to the VA's North Little Rock Diagnostic Service Office, the designated location for rebuilding a replacement system for the New Orleans VistA Imaging. The system would then disassembled, transported to the VA outpatient clinic in Baton Rouge, reassembled, and put into operation. At least that was the plan.

Hardware that had to be purchased and assembled for the new system included two clustered image servers, a single background processor for image processing into and out of a jukebox, a new UDO jukebox, and a storage area network consisting of a large RAID. Because it was necessary to transfer the data on the MOD discs to the online storage RAID, arrangements were made to borrow a legacy jukebox from the VAMC in Jackson. After these data were transferred to the RAID, they would be duplicated and added to the new archive using UDO technology.

The equipment arrived by late November and was assembled.

"That was the easy part," Allen said. "The MOD and UDO discs from New Orleans had to be cleaned before they could be used because there was contaminate on the media. Each one of the 1250 discs was hand-cleaned with a combination of alcohol preps, cotton swabs, lint-free cloth, and compressed air. Both Sony Medical and Plasmon sent experienced staff with cleaning supplies to assist the Little Rock team at no charge to the VA. Not only did they show us how to clean the discs, but they worked with us to get this time-consuming, tedious job done. This was something that wasn't in our disaster recovery plan."

Media transfer began Nov. 28, 2005, and the project was completed Jan. 15, 2006. Pointers to the images were restored in late December, giving physicians throughout VISN 16 access to the images and such important files as advance directives and do not resuscitate orders. VISN 16 is serving as the national beta test site for a new software version providing data search and image access of all VistA Imaging databases within a VISN.

Of the 10 hospitals in VISN 16, only the New Orleans VAMC used a commercial PACS (GE Centricity), which also transferred radiology images to VistA Imaging. There was no way to verify that all of the images acquired the final day of use had transferred until the system could be reactivated.

Limited power for the New Orleans VAMC was restored in early December, but this did not include air conditioning for the hospital. Four portable air conditioning units had to be installed in the hallway outside the PACS computer room. It was necessary to remove the door and cover it with plastic and pump the cool air into the room using ventilation tubing. At that point, it was discovered that a critical uninterruptible power supply (UPS) system for the PACS had been damaged during the electrical failure and needed to be repaired. Several hundred media discs also had to be cleaned by hand, a task performed in conjunction with vendor service representatives.

The PACS was reactivated and tested for two days in mid-December. It appears to have incurred no damage whatsoever.

READYING BATON ROUGE

Even if the New Orleans VistA Imaging system could have been transferred intact in October to Baton Rouge, the VA clinic there could not have accepted it. Additional physical preparations were needed for the site, and staff needed to be relocated or hired to manage the system.

The clinic had enough emergency generators, but a dedicated UPS system had to be purchased and installed to support the new SAN. The New Orleans VistA administrative staff needed to be relocated and housing found for their families. As with many Gulf Coast residents, their homes had been destroyed.

Increased security was needed for the IT systems to meet the Department of Veterans Affairs' national mandate of requirements. The department granted accreditation in November.

In the weeks following the hurricane, it became apparent that the outpatient clinic in Baton Rouge did not have the physical capacity to accommodate displaced New Orleans staff and patients, much less make room for a large hospital's VistA Imaging equipment. Several years earlier, the clinic had moved from another facility, and VISN 16 reopened the old clinic building, which was readied to meet necessary power and security requirements. The only element missing was a broadband communications line to connect it to the "outside world" of VISN 16.

Originally, because the clinic had only one T1 line to communicate with VISN 16, an additional T1 line to prevent network overload was recommended and ordered. Under normal circumstances, an order is filled in 45 days. But throughout the Gulf Coast, the entire telecommunications infrastructure was destroyed or badly damaged, and BellSouth had a huge backlog to restore pre-Katrina connectivity to the area. In early January 2006, VISN 16 substituted its as yet unfilled order for a T1 line with a Metro Ethernet Connection instead.

VISN 16 will not move the VistA Imaging replacement system from Little Rock to Baton Rouge until the broadband communications line is installed. The system is accessible remotely to any physician who needs prior patient files, however.

Radiology images from exams taken at the Baton Rouge clinic are being stored temporarily on CD media on a system donated by GE Healthcare. Once the New Orleans VistA imaging system is restored in the new Baton Rouge facility, these images will be transferred from CD into the permanent archives of VistA Imaging.

The date of this implementation is unknown and outside the control of VISN 16. The lesson to be learned: No matter how good, a disaster recovery plan may encounter unanticipated roadblocks.

"It helps to be flexible and creative with workarounds. With the assistance of our many vendors, we've done just that," Allen said.

Ms. Keen is a PACS consultant and imaging technology analyst with i.t. Communications, headquartered in Palm Beach, FL. Her industry consulting includes engagements with Agfa Healthcare, Eastman Kodak, GE Healthcare, Imaging Dynamics, IDX, Merge-eFilm, Philips Medical Systems, Siemens Medical Systems, SmartPACS, Sorna Corporation, and Talk Technology.