by Rick Geiger, Cisco Systems Inc.
Grid modernization is driving rapid growth in operations technology (OT). Process maturity and best practices to design, secure and manage such large, complex systems, however, are outside the experience of most OT departments. Many utility information technology (IT) organizations have the experience and process maturity these grid operations need. To make this work, IT must recognize the requirements of critical control systems and the knowledgebase, expertise and perspective of OT. And OT must understand the strengths and value of IT expertise in networking, computing and security.
Embracing IT-OT Convergence
Utility IT organizations are expected to have the development and process maturity for handling complex IT infrastructures and networks. To communicate these capabilities, IT may help OT gain insight through questions that should be familiar from process maturity assessments: What is IT’s process model (such as ITIL)? Have all IT personnel been trained in this process model? Were they trained once or is there a cadence of continuing education and professional development? Is there a defined mentoring and education program for new hires? How is business process management maturity or progress measured? What process is in place to make corrections and apply lessons to continuously evolve and improve the model?
Business process models (BPMs) give structure and repeatability to the means used to achieve a business or operational result. Repeatability is key; once a successful way is developed, it can be taught to others and used again without mistakes. Structure is also important. It allows corrections and adjustments to improve or meet changing requirements.
Process development and maturity also must consider mandates for regulatory compliance. In OT departments, particularly, compliance planning, tracking, reviewing, submittal and auditing can dominate process needs and requirements.
Break Down Those Siloes
Few IT departments have had to understand the requirements and priorities of real-time control systems. As a result, nearly every utility engineering department has an IT technician who ran a port scan, installed patches or performed some other normal IT maintenance process that crashed a control system or interfered with operations. Many control systems cannot be taken down until provision has been made for continuing safe operation.
There’s an inclination for IT to be critical of such circumstances. Operational control systems often were designed and installed as stand-alone systems and were not interconnected or available for remote access. There was no requirement for periodic applications of patches or other maintenance processes once correct operation was established. Likewise, there wasn’t a need to respond to new malware or keep updates current. But threat vectors aimed at stand-alone systems and the increasing connectivity required for grid modernization are changing that.
OT system users are different from IT users, and the users’ tolerance for errors and failures is much smaller than typical IT users’. Mirroring the difference in users, vendors of OT systems traditionally have been more limited and specialized. Although the industry has begun to insist OT vendors meet the same standards as IT vendors, this has been evolutionary. North American Electric Reliability Corp. (NERC) critical infrastructure protection (CIP) has encouraged this transition, but the industry is still evolving.
In nearly every utility, IT and OT engage in their own separate strategies, planning and budgeting. Executives might not converge until the chief operating officer (COO) or even CEO level. This autonomy and focus has served utilities well, but now it can result in a lack of dialogue, few or no relationships, and little opportunity to establish trust or the shared context for relevant collaboration.
Collaboration is essential. The goal is development and implementation of best practices, governance and continuing education. This includes ongoing assessment of security, quality and effectiveness. Creating collaborative IT-OT teams is the best way to address the growing complexity of networking and computing brought about by grid modernization.
Building Common Ground
Without executive leadership, the respective organizations continue to focus on short-term goals without long-term change. IT and OT must gain understanding of the value, capabilities and needs of the other. There’s no automatic recipe that works everywhere, but many examples of success exist.
Collaboration begins with fact sharing. In the U.S., IEEE Power & Energy Society (PES) regularly runs regional Plain Talk courses for nonengineering industry professionals. Anyone can learn the basics of electric power systems and the utility’s need for security, communication, telemetry and control. In addition, most major OT suppliers have open professional development courses for utility personnel to learn their technology.
Control system and power engineers benefit from IT for non-IT professionals. Executive education courses are available at colleges and universities and focus on the management of information and communication technology. The aim is to raise awareness of best practices and process maturity that are IT’s stock in trade along with capabilities required to address the enterprise environment, security and operations management. Courses also are available from vendors of business process models, such as the Information Technology Infrastructure Library (ITIL), a widely accepted approach to IT service management.
In addition to cross-training, common ground requires personal relationships. An underlying cause of turf and trust issues is insecurity because of a perceived threat to someone’s job, department or organization. A good relationship builder is working on a cross-functional team to accomplish a short-term deliverable that requires the skills of both departments. The team must be small enough that no one can hide or say he or she doesn’t agree with the goal or how it is met. The first cross-functional team should be built using volunteers and thought leaders from the departments to help establish a new best practice.
One of the best ways to find common ground is to temporarily assign personnel from IT to OT and vice versa. This needs to be full-time. Six months is a good start; 12 months is even better. Both organizations will take a resource and productivity hit because the skill set mix is not completely interchangeable. Expectations should be set aggressively to bring people out of their comfort zones, but not so aggressively to result in failure or hopelessness. Nothing works better than immersion, and both organizations benefit.
Temporary reassignment also should include a mentoring program. Assign a volunteer to train the person from the other department. Working with the respective department directors, managers and supervisors is key so they are invested in its success.
Managing the Risk of Convergence
How should an executive at a regulated, investor-owned utility manage or minimize the risks that have prevented his or her organization from bringing IT and OT closer together? Is it risk from past conflict or poor results? Or is it the perception of risk? Often, perceived risks come down to the very same turf and trust issues discussed. This is why common vision and commitment at the senior executive level is important.
Risk management often is achieved by repeating what works. Repeating the past might work well in a relatively static environment where the business model, regulatory model, technology and expectations remain the same or change slowly. The utility industry is transforming thanks to regulatory mandates and public policy. In this case, managing risk by repeating the past might increase risk in all areas.
Risk management has its own process formalizations that can be found in ISO 31000, Project Management Institute, and other standards. These do not provide guaranteed answers but offer a framework for a methodical approach that can be applied with assessment of results and application of lessons learned. As the industry changes and past practices no longer address needs, ISO 31000-based risk management becomes actively identifying business objectives, assessing where uncertainty arises, and applying resources to manage and reduce uncertainty where possible and actively monitoring and reassessing as progress is made.
What are the IT-OT convergence risks and unknowns? Unsuccessful experience might be used as a reason for inaction when it should be examined for lessons learned. Manage risk based on the suggestions given: Establish common context, provide opportunity for developing relationships and build common ground.
The other key to managing risk is establishing ownership and vested interest in the successful results by the IT and OT professionals responsible for daily operations. Industry consultants may provide essential information and guidance, but internal teams should never lose ownership of goals, process and results. To achieve commitment, internal teams must engage in a collaborative process to develop requirements, architecture and a design for the future state; assess the current state; identify the gaps; and develop a plan to move from the current state to the desired future.
Building the Business Case
Factors in developing a cost-benefit analysis that shows the business case for IT-OT convergence include: ongoing process development and optimization; total system life cycle costs; costs of synchronization and de-duplication; and cost factors for system life cycle management on IT technology timescales.
It is logical to start by looking at the process evolution in IT and follow a parallel path of process development, maturity and optimization for OT. The business case analysis needs to address whether the organization can afford the time and resources required and whether the processes and systems developed to carry OT forward will be the same as those in IT. The cost model must include system costs and implementation, maintenance and other life cycle costs that typically constitute 70 to 90 percent of total cost. Given the much shorter technology time cycles of IT technology, replacement or expansion systems and components might not be available over the normal 20-year time frames expected of power delivery infrastructure. System life cycle planning including new system acquisition, migration planning and decommissioning is prudently considered up front.
Duplicate systems carry the risk and expense of making sure that overlaps stay in synch. The alternative to duplicate process development siloed within OT is to expand IT to provide those process services in collaboration with OT. Although not free of cost, IT departments have been evolving in this direction, building capabilities such as Infrastructure as a Service (IaaS) and Applications as a Service (AaaS).
Implications for Security, Reliability and Compliance
Security and reliability are the strongest drivers of IT-OT collaboration. The industry knows the magnitude and costs of NERC CIP compliance. More changes remain with pending NERC CIP V5, NISTIR 7628, Common Criteria, the February 2013 Executive Order on Cybersecurity, and the potential unaddressed alignment with FIPS.
Integration of renewable and distributed energy resources significantly challenges legacy reliability practices. Demonstration projects funded by the Department of Energy (DOE), research from the national labs, the Electric Power Research Institute (EPRI), universities and others point to near real-time telemetry and active control as essential for reliability of the envisioned ecosystem of distributed energy resources.
The technology solutions that are developing to address security and reliability in the face of these unprecedented requirements involve enormous expansion of network communications for telemetry and control and equally enormous expansion of data centers for the computing power to store, analyze and manage the vast quantities of data.
Author
Rick Geiger is the executive director of Cisco’s smart grid and utilities business transformation team. He serves on the Gridwise Alliance board of directors and is an IEEE Senior Member and member of the Power and Energy Society.
More Electric Light & Power Current Issue Articles
More Electric Light & Power Archives Issue Articles
