Chelan County Public Utility District has invested almost $4 million and thousands of man-hours since 2001 on a program designed to ensure the security of its three hydro projects. The program – which included a functional exercise to assess the ability of the PUD and other affected parties to respond to a security threat – has been well-received by the U.S. Department of Homeland Security and the Federal Energy Regulatory Commission.
By William G. Christman and Richard F. Robert
Chelan County Public Utility District (PUD) in Washington State owns and operates electric, water, wild-stock salmon recovery, and wastewater facilities with a capital value of about $6 billion. This critical infrastructure includes four dams that manage water for hydroelectric generation, recreation, and wild-stock fishery programs. The hydro plants have a total capacity of more than 1,950 MW. Ancillary benefits from the dams include irrigation and flood control. Protecting this infrastructure is a significant job that requires a commensurate investment of money, intellectual capital, corporate commitment, and manpower.
Starting in 2001, Chelan County PUD developed a multi-faceted security program that includes a robust security perimeter with video surveillance, 24-hour guards and patrols, and collaboration with local and regional law enforcement agencies.
To test the effectiveness of its security investment, in 2006 the PUD conducted a functional exercise at its 1,236.6-MW Rocky Reach Dam, in collaboration with other dam owners/operators in the Pacific Northwest, northern California, and Canada. The security response exercise was conducted with or observed by dam owners throughout the Federal Columbia River Power System (FCRPS), the Federal Energy Regulatory Commission (FERC), the Washington State Department of Ecology’s dam safety office, law enforcement first responders, and owners and operators of dams from the western side of Washington State that are not on the Columbia River.
One of the primary outcomes from planning, scripting, and performing the on-site exercise was significant team-building among the PUD, upstream and downstream dam owners, and law enforcement/first responder agencies. In other words, the functional exercise was effective in formulating robust incident-response relationships among the various parties that will be affected by or called upon to respond in the event of a real security threat.
Establishing the security program
After the terrorist attacks on the World Trade Center on September 11, 2001, Chelan County PUD had to determine the best way to protect its critical infrastructure. To spearhead this effort, the PUD hired Dick Robert, a security expert and former Federal Bureau of Investigation (FBI) agent. Robert initially was hired as an advisor but joined Chelan County PUD in August 2002 as a full-time employee to create a new security department.
Robert had to determine what steps were required to protect about 700 employees and $6 billion of physical assets. With a 30-year career in law enforcement and security consulting, Robert knew one significant challenge would be to explain the cultural changes involved in establishing the security program. In other words, he would need to convince utility employees to change their daily habits. As a result of this program, many areas that before were relatively freely accessible to employees – and, in some cases, the general public – were now off limits to the public and to PUD employees who do not have specific security clearances.
For example, the control rooms at the hydro plants were, at best, informally protected against unauthorized entry. Chelan County PUD’s hundreds of employees and occasionally their guests had virtually unencumbered access to control rooms and other locations where significant damage could be done by persons trained with a malicious intent. Consequently, an important part of the PUD’s new security-intense culture is that employees must carry identification badges, log on to their computer repeatedly during the day, check in at guard posts when going to work sites, and stop people they do not know and ask for identification. Some critical locations are off-limits to employees not specifically authorized to enter secure areas. All non-employees are required to obtain temporary passes after going through metal detection, and they must be accompanied by their Chelan County PUD employee “sponsor” at all times.
Images from the 80 video and six infrared-imaging cameras installed at Chelan County Public Utility District’s (PUD) four hydro projects are fed into an image-monitoring network controlled and monitored by PUD security personnel. Click here to enlarge image
In addition, because the PUD’s rate structure is designed to provide affordable services to its diverse stakeholders, budgets had to be revised to allow the security program to evolve. Consequently, Chelan County PUD had to allocate funds to develop the security program that otherwise might have gone to other programs, such as parks and recreation, office facilities, and vehicles. As a result, part of the challenge of implementing a new program and security culture involved responding to questions from the general public about the significant funds expended to establish the security program and how those expenditures were prioritized against other stakeholder wishes.
Elements of the program
Chelan County PUD built a $4 million security network designed to protect the dams, transmission lines, distribution infrastructure, waste/wastewater systems, fiber-optic lines, buildings, computers, and people. The security program involved: installation of security cameras that feed an image-monitoring network; an extensive identification/ badging program, 24-hour guards, and personnel patrols; collaboration with law enforcement and state and local government agencies; development of in-house expertise in monitoring/maintaining the security systems; and responding authoritatively to potential security breaches/threats. In essence, over a relatively short period of time, our security culture evolved from a laissez faire environment to a rigidly controlled culture “owned” by each employee. Remarkably, that culture shift has been well-received by the public and our other stakeholders.
The PUD installed 80 video and six infrared-imaging cameras supplied by Pelco at the hydro plants and switchyards. These cameras scan the facilities day and night and feed the images into an image-monitoring network. Security and operation personnel with Chelan County PUD and first responders can view the images. Digital back-up recordings of the images allow for archive retrieval of events.
Badges and guards
Employees now are required to carry identification badges. Six guard stations were built, night and weekend patrols were instituted, and building entrances were remodeled to accommodate electronic badge readers. Robert signed contracts with local law enforcement to provide security services, such as staffing an airport-style metal detector at the visitor center at Rocky Reach Dam. To avoid hiring dozens of new security staff, Chelan County PUD contracted with a local private security firm to provide guards. These guards staff entry gates at the dams and are stationed at office entrances to monitor visitors.
Collaborating with agencies
Through an ongoing collaboration with local law enforcement agencies, Chelan County PUD shares the images captured by the cameras, in real time, with the local emergency dispatch centers (known as RiverCom 911), Washington State Patrol District 6 Dispatch, and the Chelan County Department of Emergency Management. The cameras can be remotely controlled by PUD security staff or by dispatchers at these first-responder centers.
Chelan County PUD also has worked closely with FERC, the U.S. Army Corps of Engineers, and the U.S. Department of the Interior’s Bureau of Reclamation to identify dam infrastructure-specific security threats and response procedures. One of the best outcomes from this effort is an emerging recognition of how threats to any one hydro project can evolve into secondary threats to other projects when those projects are on the same river system, such as the Columbia River. Consequently, Chelan County PUD has been able to work with the upstream and downstream federal dam owner/operators in a manner that truly is more effective at preventing threats and then at mitigating damage consequences in the event an incident occurs.
Developing in-house expertise
Chelan County PUD recognized that it was important to have the capability to operate and maintain proprietary security systems, rather than relying on contract support from vendors who might be hours away at the time a problem occurred. The PUD convinced its security equipment manufacturers – Pelco, which provided the image monitoring system, and Hirsch, which provided the electronic access controls – to let staff get proprietary training. To date, more than 15 wiremen and technicians have been certified for installation, upgrades, and maintenance on PUD equipment. As a result, Chelan County PUD can immediately respond with its own staff to any equipment problems, such as intrusions into its hydroelectric control system.
Testing the program in a functional exercise
Beginning in 2005, Chelan County PUD and FERC worked together to open a dialogue between the disparate owners with major hydro projects on the Columbia River and its tributaries. This includes the Corps, Reclamation, Grant County PUD, Seattle City Light, Turlock Irrigation District, Puget Sound Energy, and BC Hydro. The goal was for these dam owners to begin to work together and to collaborate on common challenges, such as security and dam safety. The impetus was simple: If a problem occurs at a dam that is upstream of other dams, there will need to be a pre-organized response scenario or at least relationships in place to facilitate the warning of any downstream threatened projects and to respond in a well-communicated and collaborative manner.
Bill Christman led Chelan County PUD’s efforts to convene, with FERC’s assistance, an Owner’s Dam Safety Forum in Wenatchee, Wash., in February 2006. Constantine (Gus) Tjoumas, FERC’s former dam safety director, conceived the idea for this forum. Tjoumas’ vision was that all hydro operators in various regions of the U.S. with similar interests and challenges would convene to share ideas and ensure they knew how to work together. Tjoumas selected the Pacific Northwest for the first such forum because of the many federally-owned, federally-regulated, state-regulated, and even foreign (Canadian) dams on the Columbia River and its tributaries.
During the functional exercise at Chelan County Public Utility District’s 1,237-MW Rocky Reach Dam, emergency responders from the SWAT team regain access to the control room through gates chained by “intruders.” Click here to enlarge image
The forum was very successful in initiating dialogue between the participating organizations, for the benefit of otherwise independent operators learning from each other’s successes and challenges. One major outcome or theme was “combining dam safety and security.” That mattered because the dam safety world is relatively mature and well-established, but the security world was continuing to evolve after the terrorist events of 2001. Combining the dam safety and security objectives has helped dam owners/operators recognize both the value of a high-quality security program and, to a large extent, to evolve dam security programs in a manner focusing on intrusion prevention and on which aspects of hydro infrastructure are most vulnerable to threats.
Consequently, a primary initiative that emerged from this Dam Safety Forum was a clear and widely supported recognition that hydro operators should be closely collaborating, communicating, and coordinating with their security counterparts, both within their organizations and with other owner/operators and first responders. The forum participants established the objective to develop a planned security exercise to be staged at Chelan County PUD’s Rocky Reach Dam. Upstream and neighboring downstream dam owners participated in the exercise, and the other utilities and agencies observed the exercise.
In planning this exercise, the PUD worked with FERC to engage the U.S. Department of Homeland Security (DHS). DHS was pursuing an initiative intended to more closely and effectively coordinate public infrastructure operators in planning for disaster prevention and recovery. Chelan County PUD and DHS discussed several realistic scenarios, including an earthquake, a broken spillgate, and a major flood event. A security breach was viewed as the most likely and realistic scenario that could happen and one that the broadest group of hydro owners and public safety agencies could understand and engage in. In addition, this scenario would allow the PUD to meet a federal mandate to conduct a security action plan at its hydro facilities at least every three years.
Chelan County PUD worked with the Washington State Patrol, local law enforcement, FERC, Reclamation, and the Corps to develop the security breach scenario. They settled on the scenario of an environmental terrorist group attempting to take over the project and open the spill gates. The PUD then developed a script for the exercise. It was important that the exercise be practical and applicable to the Rocky Reach facility. The script also needed to allow the state patrol to test its own responses to security threats.
Once the script was developed, Chelan County PUD needed to determine which representatives would be directly involved and which would be observers. In keeping with FERC’s desire that the exercise be regional in nature, Grant County PUD, the Corps, and Reclamation were invited to be involved as direct participants in the exercise, with the intent the mutual engagement of these agencies/owners would result in increased collaborative partnerships.
Chelan County PUD ran the seven-hour exercise at night for two primary reasons. First, night was viewed as the least likely time that a new, untested event of this magnitude would interfere with regular hydro operations. In addition, the popular visitor center at the dam normally is closed at night, thus reducing the possibility that the public would believe a “real” event was occurring. Second, a night-time scenario seemed more realistic than a broad daylight scenario. The exercise was completed in November 2006.
After the exercise, Chelan County PUD held a debriefing with law enforcement representatives, FERC, Reclamation, the Corps, and the other dam owners. The goal was to reinforce the lessons learned and reinforce the relationships established.
Results and lessons learned
The exercise produced a wealth of lessons learned. In particular, the participants and observers were impressed with the palpable realism of the potential that this kind of security breach and necessary response could occur without notice. Consequently, at the next Dam Safety Regional Forum, hosted in February 2007 by Puget Sound Energy in Bellevue, Wash., a big part of the program centered around Chelan County PUD’s security exercise – how the event was planned, the evolution of the event in real time, how well the response occurred, and all the lessons learned during the event and the subsequent debriefing. Most dam owners within the Pacific Northwest, as well as Dan Mahoney and Bill Allerton from FERC. This was important in maintaining the momentum gained during the earlier forum as a regionally significant colloquy of persons responsible for ensuring ongoing high-quality dam operations and security.
The overriding lesson learned was the critical nature of preparedness. There is no substitute for planning a security breach, staging the event, working through the call-down list, establishing relationships with security and operational stakeholders, observing the response, and then working as a team to determine how to be better prepared in the future.
Chelan County PUD is positive that the investment in its security program, in both time and effort, has been and continues to be a worthwhile investment. There have been no malicious intrusions on the PUD’s critical infrastructure facilities, and employees feel safer because they know security personnel carefully monitor access by outsiders. Anti-hacking computer systems and virus protection have blocked hundreds of thousands of attempts to penetrate PUD firewalls. And guards on patrol have apprehended thieves and vandals in the act of damaging Chelan County PUD structures and equipment. (However, the PUD is encouraged that none of these malicious actions occurred within the perimeters of its hydro facilities.) In addition, the camera system has helped law enforcement personnel monitor suspicious activity near the facilities, capturing images as diverse as water-main breaks, suspects hiding from police, and a prowling cougar.
Despite the success of its dam safety and security programs thus far, the PUD believes these programs require constant upgrading and vigilance in order to continue to be successful. The PUD has learned that combining dam safety and security planning is valuable. However, this full-time effort is only as good as the initial and continuing investment and attention placed into the program.
Messrs. Christman and Robert may be reached at Chelan County Public Utility District, 327 North Wenatchee Avenue, P.O. Box 1231, Wenatchee, WA 98807; (1) 509-661-4283 (Christman) or (1) 509-661-4272 (Robert); E-mail: [email protected] or [email protected] chelanpud.org.
In recognition of its efforts to provide security at its hydro projects, in 2008, Chelan County PUD received the Outstanding Stewardship of America’s Waters award from the National Hydropower Association.
Bill Christman is hydro engineering manager and Dick Robert is director of the security division with Chelan County Public Utility District.