The Blackout Test: Coordinated Testing for Complicated Utility Infrastructure

by Shaun May, EIT, CEM

This article is part of Wood Harbinger’s newsletter series.

We spend the vast majority of our lives within the built environment. We put our trust in buildings, bridges, and utility infrastructure, and sometimes take for granted that they will safely meet our needs. It’s really only in times of emergency that we actually think about how that’s supposed to happen. If there’s a fire in your building, where do you go? If you’re walking to your car in a downtown high-rise garage and the power goes out, what do you do? We see exit signs, fire alarm call stations, emergency power generators, and we feel assured that in an emergency these safety nets will catch us and keep us safe. But how do we really know?

I would love to walk to work along a grassy dirt path, green tree tops looming overhead, swaying in the wind, my only worries perhaps a falling tree, or a wandering bear twice removed from a warm meal. However, my commute entails driving over the longest floating bridge in the world, the SR-520 Bridge. This bridge was built to last. It is defined as a life safety critical structure; therefore, its systems are designed to strict criteria to assure safety and resiliency.  The bridge is comprised of numerous systems that support life safety, including backup power, lighting systems, fire protection, closed-circuit television (CCTV) for security and roadway safety, bridge pontoon leak detection, safe landing places for stranded boaters, and multiple emergency boater response phones to assist boaters in distress…the list goes on. They’re managed through the Bridge Control System (BCS), a programmable logic controller with a fiber optic communication loop. These systems all play a critical role in maintaining the safety and security of our valuable infrastructure and its end-users; more than a hundred thousand commuters utilize the floating bridge every weekday.

Do all these systems work when the power goes out? You better believe they do. How do I know? I was part of the commissioning team that tested them.

The Blackout Test

As we demand more of our infrastructure to meet increasing productivity, we set the bar higher for it to maintain operations and keep people safe. This often means greater system complexity. As the complexity increases, so too does the intricacy of the testing procedures we apply to assure the integration and functionality of power, backup power, lighting, HVAC, data/comm including specialty components like nurse call, fire alarm and suppression, and elevators. These are common life safety systems found in our utility infrastructure. This highly coordinated, comprehensive testing practice is called blackout testing. It is designed to stress all the systems with a simulated “worst-case scenario,” like a fire during a power outage.

Traditional commissioning functional testing verifies functionality throughout the system hierarchy, from components, to systems, to inter-connected systems, to full system integration. For example, we test individual luminaires, then the road lighting systems, then the lighting power distribution with integrated controls including photoelectric cells and fiber optic communication. Likewise, with blackout testing, we sweep through and verify that everything works, but with one major catch: we first simulate a power outage (hence the name “blackout test”). For critical infrastructure like the SR-520 Bridge, a hospital, or other 24/7/365 facilities and infrastructure, we need to assure that all systems operate as expected on backup generator power, since utility power can be expected to fail occasionally but ongoing normal operations are essential.

The Blackout Test Process

Ready,

Preparation for the blackout test begins by assembling the team. A scripted flow of test sequences and verification steps for each person is outlined. Personnel are distributed throughout the facility/infrastructure, with a test script in hand, ready to test the system.  They will observe and record actions and responses according to each role. The SR-520 blackout test was conducted by a team of 12 people, including four of Wood Harbinger’s commissioning providers, over a five-hour period. We conducted a similar blackout test at the Snoqualmie Valley Hospital before it was opened a couple years ago.  For this test, the testing team was also 12-strong, and conducted the test over a three-hour period.  Radios and phones are pivotal to team communication throughout the test, especially in a large environment like a campus, or a 1.5-mile long bridge.

Set.

Once in position, each team member verifies that systems are in normal operation: no maintenance is in progress that would impede testing, onsite crews are aware and notified prior to initiation of the blackout, the generator has sufficient fuel to endure the test, lighting is on, confined spaces are certified safe to access, systems are not in test mode, local authorities have been notified, user/password credentials are in-hand, monitoring and protection systems are active, and equipment and elevators are operating normally. We haven’t even started the test yet! Coordinating a team around this level of integration requires a high level of preparation, communication, and patience.

GO!

After verifying normal operation, the team is ready to initiate the blackout test.  Everything from here on out is triggering/simulating conditions and observing system responses.

First, utility power is disconnected. Almost every system is deenergized momentarily by the loss of power: on the bridge, this included lighting, fire protection, cathodic protection, CCTV, emergency boater response, leak detection, intrusion detection, weather station, elevators, and air conditioning. These critical systems will come back online on generator power. For SR-520, the Bridge Control System stays online on backup battery power and reports alarms for the lost utility power condition, including an automated message to maintenance personnel to respond to the power loss.

The first system under pressure to perform is the power switchgear. The transfer of power should take no longer than 10 seconds. The switchgear detects a loss of utility line voltage (power) and sends a signal to start the generator. The generator starts up and fires to achieve operating rotational speed.  The switchgear then detects the generator at operating frequency (60 Hz), automatically opens (disconnects) the dead utility feed, and transfers to the live generator feed. The team verifies with a stopwatch that this all happens within 10 seconds from the initial power outage.

The lights turn back on almost instantaneously when the power switches over from utility to the backup generator. Systems are now operating on backup power; this is the time to stress test all levels of all systems, from components to integration, under blackout conditions.

For large infrastructure or campus environments with integrated control systems, there are so many possible points of failure, so many components, and so many system-to-system integrations that have occurred throughout installation and startup of equipment, that a whole system network integration test is necessary to assure zero points of failure throughout. During the hospital’s blackout test, we assured that egress lighting remained lit, elevators recalled to the correct floor, nurse call remained active, and roll-up fire doors, ventilation, and Direct Digital Controls (DDC) performed as expected.

Equipment on critical and life safety power branches add to the integration challenge. When utility power drops and the transfer gear switches to generator power, critical equipment must be powered by the correct switch(es), and automatically reset on the new power source.  Some motor drives experience latching faults that require manual reset under these demanding power conditions; in previous blackout tests, we have discovered and resolved this issue, along with others similar to it.

Fire protection systems are another particularly critical system to verify during the blackout test. Like I said, we’re going for testing all systems in a worst-case scenario, and fire is a dire situation. In our SR-520 bridge scenario, the fire protection pumps were started, and the piping system was filled to ready the system.  Indicator beacons signaled as intended; these beacons are designed to inform firefighters on scene when the standpipe is filled and ready for them to connect the fire hose to fight the fire. These procedures are part of an Emergency Operations Plan that the local fire department has agreed to with WSDOT, the owner of the bridge. Performing simulations of these critical procedures provides necessary training experience and verifies that these life safety systems are ready when we need them.

In a hospital environment, it is critical to assure fire alarm and smoke control systems perform during blackout conditions. Elevators are tested for integration with smoke detectors, so that elevators recall to a safe floor; one without smoke. Additional checks for our bridge blackout test included verifying the corrosion protection system, weather station, and server room air conditioning remained active. Each of these systems plays a valuable role if they can operate under stress for longer periods of time. Within hours without air conditioning, the server rooms would overheat causing potential electronics and controls system failure.  In the event of a natural disaster, the bridge’s weather station remits valuable information to the Traffic Management Center. Corrosion of bridge steel is a compounding process; once it starts in one location, it tends to spread, so minimizing interruptions of the bridge cathodic protection system helps stave off larger issues.

System Reset

The blackout test concludes by returning all systems to normal operation, such as draining the dry-pipe fire protection system, powering down the backup generator and switching back to normal utility power. All systems are verified to return to normal operation on utility power, restored to normal condition, and backchecked against the conditions recorded before the blackout test.

For the SR-520 Bridge, this blackout stress test marked the beginning of a 30-day endurance test for the bridge. Systems were operated normally and monitored for any errors for a month after the blackout/stress test. At this level of infrastructure, redundant safety checks are part of the plan, and all aspects of the process are verified and reverified to establish a deep level of confidence before the infrastructure is opened to the public.

The Benefits of the Blackout Test

Conducting a blackout test is an essential part of commissioning critical facilities such as hospitals, operations centers, and essential community infrastructure. The goal of the blackout testing process is to verify full, integrated system functionality that maintains safety for the users. The blackout test is an opportunity to uncover potential safety oversights, even in fully functional systems, before they become an issue. Operations staff gain invaluable training through their participation in these scenarios. A successful blackout test yields peace of mind for facility owners and operations staff, proving that systems work, under the worst-case conditions, and are ready for action when emergencies happen.

Thinking Beyond the Blackout Test

Throughout our day-to-day life in a metropolitan area, we are supported by critical infrastructure and systems, operating behind the scenes. I drive across the floating bridge every day. I see our local transportation infrastructure expanding, and new construction sprouting up throughout Seattle like seedlings in spring. There are valuable lessons to be learned and extrapolated from the blackout test’s integrated approach. It is just one example of an exhaustive facility/infrastructure test. As we continue to grow, in all senses of the word, we must be mindful of continually assessing our processes, equipment/tools, and operations, to inform future development that sustains safety and productivity in our environment.

This entry was posted in All Engagements, E-Newsletter and tagged , , . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

One Trackback

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*
*