Case studies for teaching

[Albert LaFrance]

I know we have some full-time instructors in this group, and I'm sure many of you teach informally on the job, so I was wondering if there's any interest in discussing cases of control-related mishaps which might offer valuable insights for students.

One of my favorites is the Taum Sauk disaster of 2005, in which the mountaintop reservoir of a pumped-storage hydroelectric plant in Missouri overflowed, undermining and finally collapsing part of the reservoir wall and releasing a torrent of water down the mountain.

It's an interesting case because so many things went wrong. In terms of control systems alone, there were multiple failures. The primary monitoring system for the water level was a pressure transducer at the bottom of the reservoir, enclosed in a plastic pipe which extended to the top of the wall. The pipe came loose and floated upward, so the sensor was no longer at the bottom of the reservoir and thus understated the water level.

The backup level sensor was a pair of conductivity probes, which ended up being higher than the overflow level of the reservoir due to uneven settlement of the wall foundation. And in addition, the PLC to which they were connected was (inexplicably) programmed to require both the HI and HI-HI probes to be actuated before shutting off the pumps!

 

[DeKemp]

The videos of the USCSB (United states chemical safety board) are also very interesting to watch.
https://www.youtube.com/user/USCSB

 

[James Mcquade]

we had a horizontal walking beam system that went south due to a bad prox sensor.

this was a 4 movement system with 6 stations.
extend beam, move beam to the right, retract beam, go home.
the stations were load station, form swedge crimp, spin rivet, blank station, reject station, and good part exit.

there were prox sensors on all movements of the walking beam as well as fault
logic to detect no movement. Worked great for over 10 years.

One day, the sensor that indicated the beam moved to the right had started to go bad and began giving an on signal before the beam was actually there.

as the beam moved the parts to the right, the prox signal turned on after only 6" of the 18" of travel and the machine cycled and almost destroyed everything. We found a piece of metal on the prox and thought the problem was solved. 20 parts later, the prox went bad and was replaced.

we modified the programming to prevent the beam from moving and generate
a series of faults if both proxes were on at the same time for sny movement command (4 in total).

regards,
james

 

[just the cowboy]

Back in the 80's

Back in the early 80's when PLC 5's came out we did some DH+ MSG instructions from compounding to filling for tank filling. We did not know that when someone hooked up their laptop to the network it slowed the highway down while they were connecting. So for about a month compounding would overfill the tanks downstairs in filling dumping gallons of shampoo on the floor, and it only happened once in a while. Once we found out we put handshaking in to detect loss of highway speed.

 

[Jieve]

I had a former co-worker who worked as a programmer before he started teaching. He told me that he was working for a company as part of a team programming an automotive assembly line for a manufacturer here in Germany. 2 weeks after the system was commissioned and up and running he got an angry phone call from the factory. Automobile chassis were transported through the line on skids, and at one point they were transported from a conveyor onto an elevator platform, then up the elevator, and onto another conveyor. It turned out the elevator had decided to start moving up after a chassis had only made it a quarter of the way on. This flipped the chassis/skid combo upside down and it landed on top of the next chassis in the line. Apparently it took them over a day to get it all cleaned up and running again. It turned out he eventually found that it was a bug in his program that caused the elevator to move incorrectly and corrected it.

 

[MrAnonEMoose]

An extreme case of bypassing protective functions, and poor maintenance procedures:

2009 Sayano–Shushenskaya power station accident

These types of catastrophes illustrate the importance of engineering design process (reviews, HAZOPs, SIL etc...), and work procedures in an operating plant.

 

[Albert LaFrance]

The videos of the USCSB (United states chemical safety board) are also very interesting to watch.
https://www.youtube.com/user/USCSB

First of all, my apologies for starting this thread and then neglecting to follow up on everyone's responses.

Thanks for posting that link - I found the videos very interesting, offering plenty of "food for thought" for people who design, maintain and operate control systems.

This one is a good example: https://www.youtube.com/watch?v=41QMaJqxqIo. Lots of good points for discussion there!