Hydropower

Lessons Learned: Spurious SCADA

Issue 5 and Volume 22.

By James L. Gordon, hydropower consultant

George was annoyed. Almost every hydroelectric power project he had worked on over the past 20-odd years experienced problems with the supervisory control and data acquisition (SCADA) system, always detected when the system was being commissioned. But because he was not an electrical engineer, George had only been able to follow the developments and was unable to provide solutions.

His interest in this topic started in 1992 when a client mentioned that the SCADA system being installed at his old small hydro facility would not operate, despite all attempts to rectify the situation. The 4 MW plant had been flooded, with water rising to just below the generator, from an ice jam in the tailrace. George had produced a report on the damage and recommended that the opportunity be taken to upgrade the governor controls and install a remote control and SCADA facility. The contractor in charge of the rehabilitation had completed the recommended work but was unable to commission the SCADA. It was eventually operational about six months later.

Ever since then, George had advised his clients that difficulties could be expected during the installation of a new SCADA system and that detailed specifications and close following of the installation were essential. Despite such warnings, SCADA problems persisted on every new power facility George worked on.

The latest facility George worked on that experienced the same difficulties was a plant containing two Kaplan units with a total capacity of 50 MW. George was on the owner’s engineering review board and had warned that close attention to the SCADA system was essential. In this case, the owner had recently installed a new SCADA system and control center for the 14 hydro plants in the area and had well-qualified electrical engineers on staff. As a result, the specifications for the SCADA system at the new plant were very detailed, even including manufacturer’s model numbers for the instruments communicating with the central control room and all details on computer screen requirements. The operating engineers for the facility were also consulted and provided valuable input to the specifications.

Nevertheless, the SCADA would not operate, and the power plant had to be staffed and operated manually. It took about a year to resolve all the problems and commission the SCADA system to the owner’s satisfaction.

What is going wrong? One answer was that even though the specification called for instruments installed in a control system only two years previously, they were no longer being manufactured.

A year later, George had another answer from an unusual source. George had upgraded his office by adding high speed internet access and purchasing a new computer. However, he could not get his 4800 dpi, one year-old scanner from the same manufacturer to communicate with the new computer. Discussions with the manufacturer’s help staff were not productive.

George eventually called the computer store where he purchased both devices and was advised that 25 year-old Mike would come fix it. Mike arrived and advised that all would be running within a few minutes. An hour later, Mike was looking worried. He had removed all traces of George’s attempts to install the scanner software and had downloaded the software from the manufacturer’s web site and installed it, with no success. Mike went back to the web site to see if there were any upgrades for the scanner software. None. However, Mike knew of a screen on the web site where technicians posted upcoming revisions. He found two and installed them; still no communication.

The last resort was Google, where Mike looked for comments from technicians having the same problem with the same two devices and found two additional suggestions, one recommending the USB port be changed and the other providing a software patch. Both were done, and bingo — it worked.

A week later, George was working on the review board for a large utility. He mentioned the SCADA problems and asked if the utility had similar experiences. Their reply was that they had trained a group of engineers to design and install the control and SCADA systems for all of their plants because consultants had proven incapable of undertaking such work.

Lesson learned

Over coffee, George discussed the challenging computer problem with Mike, who had 10 years of computer software installation and de-bugging experience. He indicated that the computer industry was advancing at such a rapid rate, it was becoming difficult to keep up with the changes. About five years earlier, Mike had taken a four-month sabbatical from computers to earn some money to continue his college education, and when he returned to computers he had to re-learn the industry.

The same problems affect SCADA engineers, with software and hardware becoming obsolete within months. George had a laptop computer with Windows XP and had no problems with the scanner. The new desktop computer also had Windows XP but with the addition of Service Pack 2, which was produced after the scanner was purchased. If a large computer manufacturer — producing laptops, desktops, scanners and printers — cannot keep up with changes in computer operating systems, what hope is there for SCADA engineers? The only approach is to be prepared for delays in commissioning the SCADA system and include a large penalty in the contract for late commissioning, in order to maintain the manufacturer’s interest in resolving the problems.  

More HRW Current Issue Articles
More HRW Archives Issue Articles