Meditech Crash: A Canary in a Coal Mine
At 4 a.m., Sunday, Jan. 10, Meditech crashed. That crash set into motion a series of events that led to the discovery of a massive water line leak that flooded the mechanical room, shutting down not only Meditech, but also the hospital water and heat.
The events unfolded like an action-filled novel, however Incident Command and staff remained calm, collected and incredibly collaborative. This issue of Engage will highlight some of the amazing efforts of staff.
First, here is an overview of the events of those 24 hours.
Meditech Goes Down
Meditech went down at 4 a.m. IS personnel, Steve Speelman, Janet Finchman, Bob Gausman and Bob Fernandez came to the hospital to bring the servers back up. However, power was disrupted and after doing some searching, Speelman discovered a circuit breaker had blown.
Speelman contacted Facilities. Ray Rodriquez, a new employee, responded to the call. He went in search of the problem and discovered approximately 8 feet of water in the mechanical room, located in the basement, with water cresting that room and flowing up the stairwell near the back hall.
“Meditech going down was the canary in the coal mine,” says Speelman. Had Meditech not gone down, the water leak may have gone undiscovered for several more hours, causing more delay and damage.
Flooding Continues to Be a Major Concern
Speelman and his crew were able to find an alternative source of electricity and got Medictech back up and running in a short period of time. However, the flooding was a major concern and the AC unit to cool the server room was not functioning. Overheating and another Meditech shutdown was possible. The IS team switched the AC unit to a different power supply to help with the cooling and UVA IT brought up another AC unit.
Over the next couple of hours, Facilities crew members Jeremiah Dove, Lee Krueger, Joe Myers, Rodriquez and Mark Utz worked tirelessly to find the leak and stop the flow of water. Utz soon realized that we needed assistance and called the Culpeper County Volunteer Fire Department and Town of Culpeper Public Works to help with the efforts.
Fire Company 1 arrived on site, with 1st Assistant Chief Charles Perryman leading their efforts. Perryman assessed the situation and quickly discovered he could not get the pumper truck close enough to the building to use a system called drafting. Drafting would have allowed the firefighters to suction the water through their hoses, into the pumper truck and out other hoses into the drain. Since this was not an option, the firefighters used stand-alone pumps and called on local business owner of K&M Equipment and firefighter Steve Corbin, who provided industrial-grade pumps.
In addition, Facilities staff contacted multiple construction crews asking for additional industrial-grade pumps. Even with all of this effort, the water was flowing as fast as they could pump it.
Firefighters from Culpeper Volunteer Fire Department who helped that day included: Terry Graham, Wayne Green, Jack Griffin, Lt. Travis Kenney, Dickie Thomas, Mike Williams, Lee Wilson, Chief Kenny Mills and Lt. Eric Neese.
Water is Shut Down and There’s No Heat
Initially, the source of the leak was unclear, causing the hospital to cut all water to the facility until crews could find it. In addition, since the mechanical room was under water, heat to the facility was interrupted. While the day was relatively mild, expected temperatures for Sunday night were in the 20’s.
The immediate need was to deliver non-potable (not for consumption) water to the patient rooms so staff could flush the toilets. All hands on deck retrieved buckets, filled them with water and delivered them to the units.
At the same time, the town, contractors and Facilities staff shut off the water supply at the street.
With no water and heat, Incident Command diverted all non-critical EMS calls. In addition, Chief Medical Officer Tom Reynolds, MD, and Larry Whitehurst, MD, contacted all of the on-call and admitting providers, asking them to admit and refer patients to other facilities until we went off diversion.
Patient Safety Is Continually Assessed
In light of the malfunctioning facility systems, Incident Command cancelled all OR procedures for Jan. 10 and 11. Michelle Strider, RN, Administrative Director of Quality, Maggie Conklin, RN, MPH, Interim Chief Nursing Officer, and Patrick Benton, MD, Medical Director Hospitalists, assessed all of our patients. In addition, Case Manager Margaret Kelly made calls to area skilled nursing facilities to find alternative placement for patients who needed additional care and could be transferred to a skilled nursing facility should the need arise.
“We prepared for the worst-case scenario. If we couldn’t return heat to the facility, we needed to decide when we would need to evacuate the patients to another facility. Patients were prioritized for transport based on their acuity and care needs. When heat was restored to half of the facility, we looked at all patient care areas with heat to see if we could move our patients to those locations, avoiding evacuation,” says Conklin.
Strider, Conklin and Benton created a stratified risk assessment. What patients could be discharged? What patients were so critical that we would have to transport them if we couldn’t restore heat? What areas of the hospital with heat could we move patients to and what was needed in those areas to provide the best patient care possible, for example gases?
The call for transporting patients would be made on Monday morning if heat could not be restored.
Water and Heat are Restored
The gushing water was turned off by mid-afternoon and Facilities staff found the source of the leak in the fire suppression system. With the leak in the fire suppression system and not the domestic water, Facilities was able to turn the water back on to the facility.
Also by mid-afternoon, heat to half of the hospital was restored. It would be hours before the water was low enough in the mechanical room for the engineers to determine the damage to the motors. However, the Facilities team and Moores Electrical and Mechanical assumed the worst and had already found replacement motors in Richmond. Rodriguez drove to Zion Crossroads and returned around 9 p.m. with replacement motors. By midnight, the Facilities team and Moores Electrical and Mechanical replaced the motors and restored heat to the entire facility.
At 3 a.m., Monday, Jan. 11, the hospital went off diversion.
By Monday morning, all of the systems except the fire suppression system were functioning normally. A new electrical panel must be replaced. The fire suppression water line leak, located below the hospital foundation, must be repaired and the elevators must be repaired.
As of Tuesday morning (when this article was written), the crews were still working to replace the pipe and the hospital remained under a fire watch, which includes careful observation and enhanced rounding and diligence.
“With all that could have gone wrong, it didn’t. The teamwork and collaborative efforts were outstanding. It was a remarkable experience,” says Tom McCoy, Manager of Security and Emergency Management.