This article is the second of a three-part series on the future of hydroelectric control rooms. The authors envision the role of computerized control systems in the operation of hydroelectric generating systems and how smart technology might one day make the most of day-to-day management decisions now performed by human operators.
By Jery R. Stedinger and Charles D.D. Howard
Editor’s Note: HRW asked two of hydro’s creative thinkers to consider how the control room of the not-so-distant future might look. In this article, they extend their sights over the horizon to a day when both humans – and machines – occasionally may long for the good old days of manual operations.
At the Palo Verde hydroelectric plant, the latest update from the Reagan Power System Control Center scrolled across an unattended computer screen in a dark and silent room.
“There is good news and bad news. The good news is that the reservoirs in the North Central System are full to the brim. They are in a good position to meet peak loads for the next two to three weeks without having to purchase from Continental Electric. The bad news is the power system confusion. Power Sales Division has committed our share of the central system transmission line to a wheeling deal, so we can’t use the North Central plants for at least another two days during peak hours. In the meantime, the reservoirs might spill warm water. If that happens, cold water must be released through low-level outlets to meet dynamic instream temperature requirements.”
No one was there to read the bulletin. Diversified Electric Utility Company (DEUC) had completed full automation of the Palo Verde River system in 2011. The monitors remained in the empty room for maintenance personnel making field checks of the equipment.
“An update from sensors implanted in fish in the lower river is due in less than 5 minutes, providing data for determining the appropriate timing and need for low level releases,” the update continued. “It will be important to lower system reservoirs during the next ten hours if the forecasted frontal system moves in. The fish eggs in the stream gravel are almost ripe, and the sonic gravel sensors will detect imminent sediment movement. Any significant disturbance of the riverbed by high spillway discharges would be costly for fish production. A buy-back from Continental Electric could be the best solution. System economic dispatch can be reworked.”
Joe, that computer’s on the phone
Joe Charboneau was just three days into retirement when the voice telephone signaled a call from the control room at Palo Verde. He responded: “This is Joe. What’s the problem, OTTO?”
“Joe, I am having a very difficult problem here,” said the voice of the quality-control computer. “I think there is a virus loose, but I can’t quite track it down. I thought you might have some ideas.”
“OTTO, there can’t be a virus. You would know about it. You have been programmed to detect every possible virus that anyone could ever invent, and you know it. There must be something else that is the problem.”
“Joe, it is definitely a virus. It seems to be located in the turbine temperature-sensing chip. Human personnel may need to replace the chip and purge a portion of the SCADA system. I have no access to the code.”
Joe switched on the videophone and immediately was rewarded with a view of the interior displays in the control center. “OTTO, please switch the display to the main gallery and focus on the main cooling-water plumbing system gages.”
The display switched to show a set of primitive pressure gages, their needles fluttering in response to the transients of pressure. “I want to look at the temperature gages.” Joe turned on the zoom function on the videophone display so that he could read the gages.
“It looks okay from here, OTTO. Show me the oil temperatures and calculate the unit efficiencies compared to their normal ratings at this load.”
OTTO switched the display and instantaneously responded with the numbers. Joe scratched his head. Everything appeared normal. “OTTO, you may or may not have a virus. It does not matter, because everything is fine. Do not become upset. Why don’t you ask Harry to come in early this morning? Perhaps he can go down and test the chips.”
“Thank you Joe. I am glad that everything is OK. I will let your replacement sleep until his normal reporting time. He is inexperienced and his abilities are … limited. In the meantime, if you don’t mind, I will continue to monitor the vital signs and will call you again if conditions warrant.”
“OTTO, Harry should be able to help you. That’s his job.”
“I don’t want to contradict you Joe, but Harry has just completed his simulation school on software that has no human interaction. There is no way he would be able to detect anything more than I have already. The last time we asked for Harry’s help, he made two human errors that canceled out each other. We were unable to detect a problem until it was too late.”
 |
| This is a mathematical representation of how the area-wide load flows of a power system would look on the screen of a system operator’s control console in the not-too-distant future. Based on remotely sensed data, the system would portray how power flows are shifting over a grid. The various colors would show intensity or changing patterns of power flux. |
“I see what you mean, OTTO. I guess it would be better to just work this out yourself. But keep me informed, and if you feel uncertain, be sure to call.”
After Joe turned off the videophone, he reflected on the changes he had seen in 30 years with DEUC. The new computer programs had contributed a lot to the company’s competitive advantage over its larger neighbor, Continental Electric. Downtime for unscheduled maintenance had been eliminated since OTTO’s programming had been perfected for maintaining quality control and quality assurance. His self-doubt programming encouraged him to be thorough and careful in assessing incomplete and uncertain data. “The only thing worse than an obnoxious teenager is an overconfident maintenance computer,” Joe mused to himself as he remembered some of his sparring with earlier generations of OTTO.
Meanwhile, back at the (cyber)ranch…
The image of the river below the Palo Verde hydroelectric plant stood out in the three-dimensional (3D) relief of the hologram that covered the 25-foot height of the north end of the plant control room. Pablo Yepez crossed the room and entered the hologram as he motioned to the computer for a zoom. At a swimming hole downstream from the power plant, three spirited high school youths were enjoying the warm summer afternoon and the low-flow conditions in the river. It took a few seconds for the computer to enhance the revised image transmitted from the cyberhawk. This biomechanical device had automatically launched itself earlier in the day, when dense low cloud cover interrupted satellite observation.
JENNIFER-2, the resident operations supervision computer at Palo Verde, had called Pablo into the control room because DEUC’s Integrated Operations System (IOS) had requested a plant start-up in anticipation of local thunderstorm activity. Either JENNIFER or IOS certainly could handle the start-up, including taking public safety precautions. But human approval from the control center still was required for any major change in operations.
For his part, Pablo easily could manage operations of this plant. In addition to ten years’ experience, he had extensive training and had just completed a refresher course. The company always was concerned that shift supervisors might lose touch with physical reality, forgetting the limitations and undocumented characteristics of the system. They didn’t want a mentality that viewed system operations as just another holographic video game.
Pablo’s training had included instruction on how to manually operate the gates and turbines at the plant, just like in the old days when operators watched the synchroscope lock on to the system frequency as a unit was started. It wasn’t that long ago when a stray squirrel in the communication conduits at one of DEUC’s facilities had selected a key piece of wiring for dinner. When it became necessary to operate the spill gates during a storm, the gates would not respond to commands from the plant control computer. A shift supervisor had to leave the control room, drive over to the dam, and climb up to where the gates could be manually operated. It was a remarkable failure of automation and made headlines in the national press.
The squirrel incident convinced company managers of how important it is to have people on the job in operations centers within a short drive from the dams. Of course, the new 3D holographic imaging technology provided virtual contact with headquarters and centralized coordination of the regional centers. This brings shift supervisors at all of the regional centers together for face-to-face conversations and conferences that include everything, except the smell of coffee.
The possibility that human operators would lose touch with actual operating decisions had worried engineers since the 2010s, when computer control began to dominate hydropower operations. Now, computers could handle all regular day-to-day decisions for the system. And, local plant computers like JENNIFER could operate individual plants and river basins much more reliably and efficiently than humans. There was certainly no need to second-guess computer decisions on efficient dispatch of generating units. However, the company was not ready to let computers completely take over for a couple of reasons.
First, it was important that human operators remain knowledgeable and experienced. Computers could not be set up to deal with every possible emergency that might develop, so DEUC had to have people ready to step in and take command in special situations. Second, liability under full automation had become a real concern for the company’s managers. When something went wrong, a court could rule that operating companies were negligent if they had let machines make the key decision without human supervision – even if the computers had made reasonable decisions.
The need to start a second hydroelectric unit at the Palo Verde plant had been anticipated by IOS and referred to JENNIFER for human consultation. Pablo checked the infrared hologram of the area downstream from the plant, then the visible spectrum hologram. He scanned the displays from instream flow sensors and the biological data being relayed back from the monitored fish. Satisfied with public safety and biological conditions, Pablo put his thumb on the enable button to register approval for the start-up.
While Pablo was getting a cup of coffee, conditions changed again. IOS reoptimized the economic dispatch for all units in the river system. The look-ahead for 168 hours used new Palisraeli petabyte pattern recognition algorithms to estimate the probabilities for loads, hydrology, and weather. Every 3 seconds, as the updated status of loads and units changed, the unit dispatch was optimized again. This action continuously tuned the entire generation and transmission system. Every 15 minutes, the probabilistic analysis re-evaluated the projected streamflow condition and determined how the changing weather was likely to influence electrical loads and thermal plant operations.
Under new Environmental Feedback Regulations, the emission limits for DEUC’s thermal plants were governed by real-time environmental protection requirements that reflected immediately attainable air quality improvements. Stack emission had to be changed in response to the latest changes in weather, so that the physical and biological environment actually was protected, as determined by real-time monitoring. Previous regulations had protected a hypothetical “paper” environment described by deterministic computer models. The new regulations often provided more flexibility for the utilities and more realistic environmental protection. On-line environmental modeling provided IOS with the data for calculating real-time air-quality targets.
Humans do have their place, after all
Two humans were present in the cool, dimly lit, and quiet ambiance of the Reagan control center. They sat in a corner, conversing in hushed voices as they monitored the power transfers and second-guessed the changing prices.
This was important work for people. The computer systems of DEUC and other utilities ran with different software, and not all of the software was equally intelligent or reliable. DEUC’s IOS had been programmed to look for opportunities to buy and to sell power at a good price. But sometimes a trader’s hunch was needed.
In the old days, the computers had monitored the inflow and outflow of power with the adjacent power system, fine-tuning the company’s generation to maintain the penalty-free time limits. It had been a short step to program IOS to exchange information with adjacent utility computer systems while maintain system security for DEUC. Now, IOS automatically negotiated power exchanges with the computers of other companies in response to power system condition and dynamic short-term marginal costs.
Deregulation of the electric power industry had provided many opportunities for enterprising utilities. For less nimble companies, the plethora of independent power producers, transmission companies, and local distribution retailers had meant chaos. Customers on the same street received electric power from different retailers at different prices. Reliability council standards provided a fairly uniform level of service. A host of power exchanges and financial transactions were behind bringing electricity to the customer. The marketing and expansion planning computers had been completely redesigned to continuously determine load location and market share.
The cost to many consumers had risen as environmental and social costs of power generation, transmission, and consumption were added directly to electric bills. Customers had responded by painting or cladding buildings with electrodynamic and photoelectric materials and installing whole-building energy management systems. Some buildings often generated more energy than they consumed. IOS tracked cloud cover motions from satellite images and dynamically computed energy inputs and demands from “experience functions” for each customer class and geographic location.
The computer managed local distribution grids continuously and redistributed and balanced local generation with local consumption, exporting surpluses to adjacent neighborhoods and importing or exporting the difference to the highest-bidding utility. This system of distributed electricity generation was backed up by the generating capacity of utilities for nighttime and peak loads.
IOS closely watched the changing value of water in storage in DEUC’s reservoirs and the value of hydropower at every moment, managing system generation to take advantage of sales and purchase opportunities. Loads and the distributed electricity generation surplus were anticipated by satellite tracking of weather systems, then confirmed by feedback over the cable television/Internet/digital interface that connected the utility with every customer residence and business unit.
Returning to the control room with a fresh cup of coffee, Pablo reflected back to his first day of work with Joe Charboneau, the veteran operator who had trained him. Joe had been awed by the computers that talked and followed his verbal instructions. Those machines were gone now. The old Palo Verde control computer, JENNIFER-1, had been a great help, but she didn’t always put the facts together correctly. Sometimes she missed things completely. Her integration with other computers in the system also had left much to be desired.
An incident that actually had nothing to do with proper plant operations had been the final impetus for getting management to approve purchase of new station control computers. DEUC had been looking at a transmission line expansion plan with a neighboring utility. During the course of the analysis, JENNIFER-1 had provided confidential numbers from that negotiation to a counterpart computer at Continental Electric. Continental used the information to ruin the negotiations and cost DEUC million of dollars in potential profits.
JENNIFER-1 also had raised a few eyebrows by using the aquatic status data and biomechanical hawk to direct staff fisherman to the exact location of large fish within the Palo Verde stream system. JENNIFER-2 and IOS have much better internal security. They knew what they were supposed to know, and they could manage their assignments even when information temporarily was not available to them because of restrictions placed on the system by DEUC’s Administrative Policy, Rates, or Public Participation computer systems. In doing so, JENNIFER and IOS took advantage of an astonishing development in computer science.
The development occurred when the gurus at Stanford University had finally integrated their advanced thinking in management and business administration with computer technology. Now, Controlled Ignorance (CI) was a well-established science. CI allowed a large integrated organization of people and computers to work effectively in the fish bowls of the modern information world.
Without CI, it would have been impossible to carry out business on the information superhighway. The newly emerging field of social science mathematics, developed first at Carnegie Mellon University in the 1970s with funding from IBM, provided the basic ideas and the trained people that kept the information highway from becoming a demolition derby.
IOS reflected a decade of technological advance. It was reliable and much less temperamental than the older computer systems. JENNIFER-1 had suffered from power surges when faced with inconsistent objectives. Not IOS. It knew that its job was to determine how to make the overall system work to meet the economic, social, and environmental objectives selected by human management, no matter how little sense these objective seemed to make. Ego suppression and conflict dissolution algorithms adapted from United Nations manuals ensured that IOS did not get bogged down by an irrational request. IOS just went ahead and did what it was told. The regional and local computers also had been improved.
JENNIFER-1 had caused embarrassment among operations and system professionals by pointing out inconsistencies in her programming priorities. She frequently had requested clarification and authorization from higher levels in the network, often requiring human intervention. JENNIFER-2 was more sophisticated, working equally smoothly with OTTO, the humble quality control machine, and IOS, the superbrain that managed the entire power system.
DDà…xêu+++ogUUUxxà·w … ???
Still, everything doesn’t always work perfectly. Nursing his steaming coffee, Pablo began to have second thoughts about how IOS was scheduling the Palo Verde hydro resources. An agreement with the downstream power supplier seemed to be leading IOS astray. Under the agreement, it would be advantageous to schedule spills at Palo Verde to create high tailwater levels downstream. The overall loss of energy would drive up spot market prices. On balance, it appeared that while energy output would go down, the net revenue to the company would go up slightly.
Logical but wrong. This was just the type of counter-intuitive result that Pablo believed IOS should detect. Obviously, a subtle program change would be required. Leaning forward, Pablo wondered if things had been simpler in the old days, when Joe Charboneau actually controlled hydroelectric operations by telling the computers what to do.
“JENNIFER, please run the old scheduling model … the one that Joe wrote when he worked here,” Pablo said.
“I am sorry Pablo, JOESCHED appears to be in FORTRAN, and we no longer have a compiler for that language.”
JENNIFER displayed the first 30 lines of the code. “Good grief,” said Pablo. “What is the meaning of IF(X)210,20,130?”
“I don’t know, Pablo. What should I do?” JENNIFER whined slightly. “I’m not really programmed to understand this old language. Maybe I can learn it, if there is a book in one of the Internet museum libraries.”
Pablo encouraged her. “Please check the libraries. Then convert that program to E!!, if you can.”
“I have found a FORTRAN 2100 manual in the Rittsliboteq in Amsterdam,” she replied after several seconds’ delay.
The screen immediately changed to show the same code translated to Expressive (E!!) so that Pablo could review the explicit description of the calculations made by the old program. E!! was a combination of global logic, multivariate algebra and calculus, gothic German, and Zen Buddhism. This made it easy to read and very logical. It had replaced D**, which had superseded C**.
The logic of the old program was now clear. Pablo had been wrong about his rate-impact theory. The contract with the downstream independent power supplier had a grandfather clause that included a surcharge. The amount paid for energy purchased from the downstream supplier actually was greater than the market value under the prevailing conditions. Thus, the optimum strategy determined by IOS and Palo Verde included high releases to raise tailwater levels downstream. This would reduce the efficiency of the downstream plants and decrease their energy output. But this conflicted with the intent clauses of the area-wide coordination agreement and with good business practice. No wonder the computers were confused. JOESCHED had cleverly avoided the issue by feeding deliberately false information into the network until the hydraulic situation in the river had returned to normal. Only someone like Joe would recognize this rare situation and know what to do about it.
Pablo reflected on what he should do. He leaned back in his chair and put on virtual reality projection glasses to review a draft report on the coordination agreement. As an operations analyst, he was the senior member of the company’s inter-utility coordination task force of computers and people.
The old-fashioned projection headgear still was useful for flipping pages and reading a book the way books were meant to be read. As Pablo turned the pages, the book’s embedded logic transmitted data to the glasses. The agreement materialized on the blank paper. Clear, crisp text and engineering sketches glowed softly on the page in 3D color. When he touched the caption of a figure, a touch menu popped out on the page so that he could control the scale and perspective of the images hovering over the page of text.
Pablo wrote a few ideas down in the margins. He added comments on the effect of the coordination agreement on decisions made by IOS when river conditions and power exchange prices created a situation like the present. JENNIFER recorded Pablo’s marginal notes for IOS. Later, they would be compiled with those from other reviewers.
Now, Pablo focused on the section of the agreement that dealt with head losses at downstream plants. If he understood Joe’s old FORTRAN program properly, it should be fairly simple to get IOS patched up by rewording this section of the coordination agreement. JENNIFER closely followed Pablo’s notes.
“What should we do right now?” she prodded nervously.
“Just accept the loss of revenue. Max out the energy value for the whole river. I’ll talk it over with Sun Li at the downstream Continental Electric plant, and he can make up the loss of revenue some other time. You can record my instructions with IOS if you want to.”
Pablo sighed, knowing that JENNIFER was programmed to do just that and thinking about how it was going to be explained to management at the end of the month.
“It won’t be necessary,” she replied. “I have an understanding with my counterpart at Continental. He will remind Sun Li when the time comes to repay the favor.”
Pablo smiled and silently praised the gurus at Stanford for their marvelously thoughtful gift of CI. Now, if he and JENNIFER could just get the plan by WERNER, the company’s regulatory relations computer, and PATSY in public participation, whose job was to allow the public to participate and provide input on key policy issues … to be continued.
Reference
Flavin, C., and N. Lenssen, “The Electricity Industry Sees the Light,” Technology Review, May/June 1995, pages 42-49.
Jery Stedinger, PhD, is a professor at the Cornell University School of Civil and Environmental Engineering. Chuck Howard is a principal in CddHoward Consulting Ltd.
More HRW Current Issue Articles
More HRW Archives Issue Articles
