Back-to-Back Storms Prepare Avista’s Incident Command System
By Bryan Cox, Avista
 |
| Lineman working after November storm |
On Nov. 17, 2015, a Pacific Northwest storm bringing near hurricane-force winds blew off roofs, toppled trees, grounded airplanes and severed power to about 180,000 people, or about half of the total electric customers of Spokane-based Avista Utilities.
The storm that struck Spokane and eastern Washington was the largest in Avista’s 126-year history. National Oceanic and Atmospheric Administration (NOAA) began sending alerts a few days in advance of the onslaught. In response, Avista activated its Incident Command System (ICS), briefed employees by conference calls, staged restoration materials and made inquiries to neighboring utilities and contractors for crews.
Within six hours of the storm striking Spokane, 75 percent of Avista’s customers in the most populous counties had lost power. The storm tripped 111 of 300 circuits, damaged 58 out of 300 substations and knocked 29 transmission lines out of service. Toppled trees ripped out gas lines and smashed gas meters and regulator stations, which simultaneously resulted in 63 orders related to blowing-gas or gas odor; in past emergencies, Avista might experience two such calls at the same time.
To restore service to almost half of its 369,000 customers, Avista went on an emergency footing unlike anything it had ever encountered. Avista normally has 10 line crews in Spokane. But for this storm, the utility brought in 132 crews.
Under usual circumstances, Avista’s dispatchers will assign crews based on the damage at a particular location. But Avista’s ICS team decided to focus its dispatchers on simply monitoring circuits and authorizing energization of them when it was determined the area was safe. Field commanders were dispatched to feeders and told to start at a substation with Avista and contract crews and work 16 hours on and eight hours off until they repaired the feeder. The ICS gave field commanders unprecedented operations control to manage the workload and restore power.
While Avista’s response to the November storm departed in many ways from its usual tactics, the utility was working with storm-tested plans and tools put in place more than a year before. The year 2014 was a bridge between a traditional way of restoring power and a new, coordinated and more scalable approach that would be critical for the November 17 storm.
Prior to 2014, in the aftermath of a storm, Avista managers would ask their operation districts: “How many line crews do you want?” It’s an obvious question, but one that lacks context. Beginning in 2014, Avista’s managers posed the question differently. They asked, “If we’re going to restore power within 24 hours, how many crews will you need?”
 |
| Bucket trucks work through the night after August 2014 storm. |
 |
| Big cloud as August 2014 storm approaches Ritzville, Washington |
Managers at Avista know this mindset matters. But here’s how the change came about. In 2011, the utility’s leaders took a hard look at the company’s customer-experience intent statement. That statement, in part, reads: “At every point of interaction with Avista, I feel that I am dealing with people that listen and genuinely care about me.”
Establishing a restoration framework and goals
Avista’s managers asked if everyone was striving to live up to this. Avista recognized that simply marshalling resources in the face of a broad emergency wasn’t enough; the way crews and resources were brought to bear mattered, too. Among those passionate about the way Avista responded to emergencies was Erin Swearingen, Avista’s enterprise business continuity manager. Erin and others knew that a coordinated response, one that mirrored the ICS, would help Avista work more effectively within the organization as well as with agencies outside the utility tasked with emergency response. Part of this work was setting new goals for the safe restoration of customers.
Across its 13 operation districts, which serve nearly 369,000 electric customers in eastern Washington and northern Idaho, Avista established three restoration goals, or thresholds. First, each local area strives to restore power in 16 hours or less with its own resources. If they cannot meet that goal, then Avista initiates an emergency operation plan (EOP). With an EOP activated, Avista’s managers assess whether they can restore power within 48 hours using all company resources. If managers determine they cannot complete restoration after 48 hours, then Avista will call out all resources across the region, including contract and mutual-aid crews, to restore power.
This work took the burden of determining crew needs off the back of the local area leaders. Instead, Avista looked at its emergency operation plans to analyze how it operated, communicated and organized its response to restoration. As part of this self-analysis, the utility formally adopted and implemented the ICS during the first quarter of 2014. The ICS that Swearingen and others lobbied for ensures Avista is better prepared and positioned for rapid restoration and recovery.
Along with implementing the ICS, Avista also made sure its fully automated call-out solution-the system used to locate and assemble crews for major or unplanned events-assigned and archived storm roles for every employee. The call-out solution, developed by ARCOS LLC, would also help Avista track each crew’s location and how long they had been working an event.
As part of Avista’s emergency preparations, any employee who was not tagged as part of a crew in the call-out system was given a storm role ranging from incident commander to damage assessor. For example, someone who normally works as a designer might be called out as a damage assessor in the wake of a storm, and the call-out solution would locate this person and track not only this employee but also anyone else who was working the event. Accounting for employees in this way would also enable Avista to track the number of hours each crew was in service, which would help the utility manage fatigue and comply with service regulations. In addition, Avista’s management agreed that, as part of the ICS, it would periodically call out its incident commanders and support staff to form an emergency operations center (EOC) to rehearse for an actual event.
Once Avista activated its EOC, the utility would send an initial notification to employees listed in the call-out system informing them to take on their storm roles. Upon reporting for storm duty, the call-out system would show each person as working that role.
Storm tests Avista’s EOC and ICS
On Wednesday, July 23, 2014, at approximately 4 p.m. severe thunderstorms with sustained winds of 50 to 70 mph moved through Spokane, Washington, continuing north and east to the towns of Deer Park and Colville and eventually to the panhandle of Idaho and into Canada. Some people caught in the path of the storms told newspapers they felt as if it were a tornado. Eleven Avista substations completely lost power. The winds sheared off numerous transmission structures and snapped poles on Avista’s distribution circuits. By 6:30 p.m., 50,000 Avista customers were without power.
Avista implemented its EOC, which came together as a virtual team rather than at a single, physical location. The incident commander and command staff began analyzing the information it had in order to identify the appropriate response level. Avista used its automated call out solution to bring in office employees to support the line crews. As dawn arrived on July 24, Avista began putting people into the field to assess the damage. The team determined that to restore power within three days it would need 28 contract crews, which was in addition to the 28 line crews already employed by Avista. The contract crews began coming from as far away as Montana and southern Idaho. It took approximately 24 hours for them to arrive whereupon they received a safety briefing and company radios.
As Avista went into storm mode, the utility was running its EOC around the clock. The automated call-out solution became the company’s go-to platform for crew mark-up, keeping track of which crews and employees were undertaking work across the utility service territory. As contractors came on board, part of the check-in and checkout procedure was to sign in to or out of the callout system whether someone was working as a contractor or from a neighboring utility or in-house crew. This centralized system eliminated paper notes and whiteboards scattered across offices and districts. The system also kept a real-time accounting of how long crews had worked a shift, so Avista could avoid pushing anyone beyond their effective limits.
After three days of around-the-clock work, Avista had nearly every customer restored. Complete restoration took a fourth day. Until that point in time, the storm had caused the worst damage to Avista’s power infrastructure since an ice storm in 1996. As Avista critiqued its performance in the wake of the July 23 outage, managers felt they had been successful in some areas, but lagged in others.
Assessing response
For example, the utility didn’t have most of its contract crews working until 24 hours after the storm had passed. And even after establishing the EOC, Avista’s ICS team saw that it was two days into the storm’s aftermath before the group was conducting regularly scheduled briefings for the whole organization.
Initially, the ICS team was holding a daily briefing, but quickly realized the organization needed to connect in this way three times per day. Spontaneously after one command briefing (and halfway through the storm’s aftermath) the team began putting together a cogent, concise set of briefing notes recapping discussions and action-items for the entire organization.
The style and format of the briefing notes were immediately seen as critical. The notes weren’t simply a transcription of who said what. Rather, the content was actionable. For example, the notes summarized messages to be communicated to crews along with the appropriate communication channel and highlighted progress made during the previous eight hours. The comments related to progress and stated specific information on, for instance, the number of customers and substations restored during the previous period. The briefing notes also became a history of the event for future reference.
History repeats itself, almost
Within 10 days, Avista plunged again into a near-identical storm. On the evening of Saturday, Aug. 2, 2014, a storm front with wind gusts reaching nearly 70 mph in some places, knocked out power to nearly 50,000 Avista customers in the span of approximately 45 minutes. Two storms of this magnitude, covering virtually the same geography in the span of 10 days was unprecedented for Avista. The Aug. 2 storm had taken down structures that Avista’s crews had rebuilt less than two weeks before. When managers asked how many of the people affected by the July 23 storm were hit by the Aug. 2 event, the response was: All of them.
Because of the July 23 storm, Avista’s employees were well acquainted with what to do. However, a summer Saturday night is also a popular time for employees to be away with family relaxing or even camping, out of cell coverage. Almost 25 percent of Avista’s field workforce were out of cell coverage when the August storm struck according to reports from the call-out solution. Because it was proving harder for Avista to get employee resources, the utility was much more aggressive about calling out contractors who arrived earlier and began working sooner than during the July 23 storm. Avista pulled 20 contractor line crews from sister utility properties and another eight contractor line crews working on its property. The violent weather across the August skies prompted many employees-initially beyond cell coverage -to come into range, check in and arrive for work.
The ICS sprang into action, too, and immediately went on a three-times-per-day briefing schedule. The call-out solution, again, managed the companywide crew mark-up (i.e., the list of crew resources, crewmember names, crew assignments and locations) along with a status and location for office employees assuming storm roles.
Avista managed the second restoration (returning power to 50,000 customers) within 72 hours, which was a result of quickly calling in resources. Not every customer immediately realized the restoration was so successful. One Avista customer in eastern Washington had left town just after the July 23 storm. Upon his return 10 days later, he saw crews working in the identical location he had left them. When he asked why they still hadn’t restored his power, the crew asked if he was aware there had been a second storm. He apologetically admitted not knowing and thanked them for their efforts.
In reviewing its performance in the wake of the Aug. 2 storm, Avista believed it could have restored power in less than 72 hours. However a critical decision was made during the restoration process to place safety over speed. Normally, in an electric outage when lines hit the ground, it would cause a fuse to blow and de-energize the line. When power returns to the substation, the blown fuse on the downed line would keep the line de-energized. With the Aug. 2 storm, the transmission and distribution lines went out so fast that substations were losing power before Avista’s fuses went out.
If Avista’s substations came back online, the utility might have had hot lines on the ground, fences or cars. Someone standing nearby could be seriously injured or killed. So Avista took the step of opening up all its laterals and hundreds of fuses, which diverted resources away from the restoration effort. The utility then patrolled each lateral before re-energizing its main line.
This safety step cost Avista an extra day, which the utility knew would happen. But the effort was a commitment to stay true (in word and action) to the company’s customer-experience intent statement. Avista’s coordinated response, made possible with its newly employed ICS, helped the utility work effectively and efficiently within the organization as well as with outside contractors, sister utilities and other third parties to safely restore power to its customers.
The experience from these back-to-back summer storms of 2014 set the foundation for the coordinated response that Avista marshaled in the wake of the Nov. 17, 2015, event. With a well-oiled ICS and EOC in place, Avista coordinated communications with police, fire, local elected officials and county agencies. In those earlier storms, Avista was trying to restore electricity. But the November storm felt more like trying to restore civilization, since not only electric service had been broken but also gas, water and sewer service on a wide scale. Hundreds of volunteers and line crews were called in for help.
One way to gauge performance is through the lens of mutual aid. In other words, looking at how long people sit idly by as commanders determine where to put them to work. It’s easier sometimes to get resources than put them to use effectively. But the crews Avista brought in to help remarked how fast they were briefed, issued a radio and guided to their appointed worksite. After the November 2015 storm, 90 percent of customers were restored in seven days, and the remaining 10 percent took two more days. No employees or contractors were injured. Avista’s customers, employees and mutual aid crews knew that they were dealing with people who listened and cared.
Bryan Cox is director of Transmission, Western Electric Operations, at Avista Utilities. Contact him at [email protected].
