loose wires in PLC Cause big problems

[drumgar]

Hello,

I thought I'd share a story about something that recently happened to me. I work for a municipal water treatment plant as an instrumentation worker. Majority of our PLC's are Modicon Quantum. I was having a meeting with a contractor who was trying to troubleshoot his flow indicator. We were done for the day and closed the cabinet door on the PLC. As soon as the door closed we got a bunch of alarms at the plant. The plant operator approached me to check into them as he suspected we caused them, because we were messing around in the PLC.
After some trouble shooting and head scratching we found that the problems were being caused in the adjacent PLC cabinet.This is an older one with some real sketchy wiring going on from years of prior additions and retrofits. Somehow a wire feeding 24v buss had wiggled loose enough to break contact while still being in the terminal block.
The effect was that we overflowed our filters and had water coming into the PLC room and water basically flowing inside the building into places that it was never intended to be. This highlighted some problems with control scheme of the filters, and the fact that the basin weirs that were engineered to not let this happen were obviously never tested.
This was a stressful day for me. Anything like this ever happen to you?

 

[labeledas]

sounds like you just tested those weirs. and I had something similar happen in a lift station sketchy job done nothing was gas tight, in fact some stuff was not even in an enclosure, terminal blocks on din rail screw to plywood. The wet well was in the same room as the exposed controls one of our guys brushed up against some wiring and it crumbled and tripped every alarm.

I am sure I have a million horror stories about scabby stuff done, like overdoing it with spares, worked on one cabinet that had 110 spare #14 wires from old filed device tek cable tucked into the finger duct, and we were trying to add new field devices and the duct covers were bursting off before we even added new wire

 

[Jeev]

Loose terminals, terminations UNDER the terminal clamp instead of in/ontop of it, and poorly crimped wires are a few of my pet hates. Over the years I reckon 5-6 "Major" problems have been traced back to wiring issues like this.

There was an interesting one recently. I had a breakdown call from a site in another state. After a couple of hours on the phone with the sparky/programmer; I bit the bullet got a flight booked and headed down. Turns out all the earth and bakplane connectors in the PLC and I/O cards had started to oxidise. The way it had manifested itself was 2 outputs not working. It's a coastal town, but I have honestly never come across this before, and the oxidisation was only just light surface stuff!

Moral of that story. A/C on cabinets that are near the ocean

 

[Saturn_Europa1]

Hello,

I thought I'd share a story about something that recently happened to me. I work for a municipal water treatment plant as an instrumentation worker. Majority of our PLC's are Modicon Quantum. I was having a meeting with a contractor who was trying to troubleshoot his flow indicator. We were done for the day and closed the cabinet door on the PLC. As soon as the door closed we got a bunch of alarms at the plant. The plant operator approached me to check into them as he suspected we caused them, because we were messing around in the PLC.
After some trouble shooting and head scratching we found that the problems were being caused in the adjacent PLC cabinet.This is an older one with some real sketchy wiring going on from years of prior additions and retrofits. Somehow a wire feeding 24v buss had wiggled loose enough to break contact while still being in the terminal block.
The effect was that we overflowed our filters and had water coming into the PLC room and water basically flowing inside the building into places that it was never intended to be. This highlighted some problems with control scheme of the filters, and the fact that the basin weirs that were engineered to not let this happen were obviously never tested.
This was a stressful day for me. Anything like this ever happen to you?

I worked at a water plant that had remote I/O at the chemical buildings connected to a PLC 5 in the control room. Any com loss would be a single pop up alarm and all the out puts would remain the same. The operator would see the com loss look at SCADA, see everything running and get back to there lab work.

I bumped a loose RS-232 cable that shut the alum off. Bad day at a water plant when you loose the alum.

 

[James Mcquade]

Defective terminal blocks of all things.

we were buying $5,000 if brand x every month, and got this order in for 6 machines. I wired 2 of them and I know they were tight.

everything would work ok for 3-4 days then quit.
loose jumpers on the top of the terminals. tightened them. same thing. and retightened. I got in there and tightened every terminal screw there was and everything was good for 3 days.

Brought in the vendor and brand x territory manager. showed him what we did and the guy refused to do anything!! even after we showed him! returned all terminals to the vendor who agreed that the terminals were defective, no problem with it.

Got a competitive quote from ab and the next week bought $10K in terminals.
the next month, brand x salesman shows up at our vendor asking why sales was down on terminal blocks. he was reminded of what happened.
Begged for our business, and me being the purchasing manager at the time said he could have our business when he paid us for lost time and labor for the bad terminals. The boss backed me up 1000%.
Never saw him again and he was later fired - brand x has a bunch of bad terminals. We never bought brand x again. Didn't have any more terminal problems either.

james

 

[OkiePC]

...
I work for a municipal water treatment plant as an instrumentation worker.

...

This is an older one with some real sketchy wiring going on from years of prior additions and retrofits.

Sketchy? In a water plant? No way... [/sarcasm]

Anything like this ever happen to you?

We had a tire machine with an assembly drum that was servo driven ... like a VFD but with tight response ... analog and enable from a PLC-5.

One day it started making scrap. The drum would halt in the middle of applying a component. Troubleshooting revealed nothing odd going on. PLC code was fine, the relays involved were the correct gold plated contacts for the low voltage signals. New bases were installed, all the wires on the relays and at the PLC were tight and clean. While the panel doors were opened, the machine would run fine...

So the techs decided to just leave the doors open. They thought something must be getting hot...but as soon as you closed the doors, within a couple of minutes, the drum would act up, sputtering starting and stopping and make a wad of expensive waste.

This went on a few days and then we had a visitor to the plant and the boss came to get me involved so they could close those doors since this machine sat on the main aisle where the VIP would see it and raise hell about the panels being open.

I puzzled over it for a couple of hours and could not make it screw up. My partner came along and I filled him in on what I was told, what I'd found, and he called bullcorn on the heat things so we closed the doors. Sure enough, within two minutes it made a mess of a tire so John was convinced there was something going on. The techs on the floor had already replaced the servo drive, the motor, the cables, the relays, the bases, the analog output to the speed reference... everything. So John volunteers to get his little butt inside the panel with the drives (this was pretty big box with plenty of room for him) holding a mirror to watch the drive enable. I didn't want to let him do it, but he was much more senior than me and insisted...this had been going on for a week by now.

So John is inside this live panel...holding a flashlight and a mirror and doors closed on him, the running machine screws up, we let him out and he tells us "yes, the drive enable LED flickered". I follow the wires and all looks good until I get to the center panel (which is nothing but terminal blocks) and the wire falls out in my hand.

The weight of the door (when open) was flexing the wire up into the terminal block making good contact. With the door closed, the panel and wiring relaxed and became intermittent. That terminal block was never tightened and the machine ran for two years following the PLC upgrade before the problem cropped up.

 

[rootboy]

We recently had a problem with a loose block in some Wago IO. Shut the panel door and things drop out.

And since you are at a water treatment plant, and use Modicon and AB, here's something else to worry about:

https://boingboing.net/2017/02/14/proof-of-concept-ransomware-lo.html

 

[jaden]

I had a customer complain that his new grain dryer would not stop the unload auger when it was done. I went out two times and when troubleshooting it, I could find no problem, and the problem never happened. Somehow, on the 3rd call, I left the door shut and tried it first. The problem occurred, but the motor stopped as soon as I opened the door. The problem was caused by an incorrectly mounted Overload relay. When the door was closed, a holding circuit was created because a switch on the door touched the Overload relay.

 

[ASF]

One plant I worked at had an old machine which had about 80 pushbuttons, switches and indicators on the door, all 110VAC. When the panel was built, one of the rolls of building wire the panel builders had used must have been defective. It would have looked fine at the time, but 20 years later, 25% of the wires were discoloured and the insulation would crumble in your hands. So if you ever, ever had to go into that panel to do something, you would open the door veeeeeeeeery slooooooooowly and gingerly, and close it the same way and just hope like hell that nothing shorted out this time.

 

[diat150]

Hello,

I thought I'd share a story about something that recently happened to me. I work for a municipal water treatment plant as an instrumentation worker. Majority of our PLC's are Modicon Quantum. I was having a meeting with a contractor who was trying to troubleshoot his flow indicator. We were done for the day and closed the cabinet door on the PLC. As soon as the door closed we got a bunch of alarms at the plant. The plant operator approached me to check into them as he suspected we caused them, because we were messing around in the PLC.
After some trouble shooting and head scratching we found that the problems were being caused in the adjacent PLC cabinet.This is an older one with some real sketchy wiring going on from years of prior additions and retrofits. Somehow a wire feeding 24v buss had wiggled loose enough to break contact while still being in the terminal block.
The effect was that we overflowed our filters and had water coming into the PLC room and water basically flowing inside the building into places that it was never intended to be. This highlighted some problems with control scheme of the filters, and the fact that the basin weirs that were engineered to not let this happen were obviously never tested.
This was a stressful day for me. Anything like this ever happen to you?

You guys should have a PM program setup. Tightening all terminals should be done at least once or twice a year.

 

[PLC Pie Guy]

Brand new 600 Horse Screw Compressor and 3 vessel system. I came in to tie in some monitoring points in the compressors control cabinet.
My nose told me something was wrong as soon as I opened the cabinet.
Once inside and looking around I was growing ever more curious about the large gauge "Black" wire tied to the Neutral terminal block section. I didn't remember that being there before. I touched it and as suspected it was getting crispy and was just sitting in the terminal block not clamped even a little.

Shut down and repair.

Caught that one just in time! Only 6 months after initial install. Could have lost the whole cabinet or worse the entire machine.

 

[labeledas]

I had a customer complain that his new grain dryer would not stop the unload auger when it was done. I went out two times and when troubleshooting it, I could find no problem, and the problem never happened. Somehow, on the 3rd call, I left the door shut and tried it first. The problem occurred, but the motor stopped as soon as I opened the door. The problem was caused by an incorrectly mounted Overload relay. When the door was closed, a holding circuit was created because a switch on the door touched the Overload relay.

oh man I love those calls some random completely unrelated condition effects a control circuit. I had one where a completely decommissioned carbon skid had to be in auto for a "newly" installed lift pump to run because some idiot wired a few of the HOA's in series.

 

[labeledas]

One plant I worked at had an old machine which had about 80 pushbuttons, switches and indicators on the door, all 110VAC. When the panel was built, one of the rolls of building wire the panel builders had used must have been defective. It would have looked fine at the time, but 20 years later, 25% of the wires were discoloured and the insulation would crumble in your hands. So if you ever, ever had to go into that panel to do something, you would open the door veeeeeeeeery slooooooooowly and gingerly, and close it the same way and just hope like hell that nothing shorted out this time.

Those are my least favorite service calls you simply breathe on the panel and create 10 more issues.

 

[cardosocea]

I had a similar one happen to me once. This was a Motion Reference Unit that had RS-232 outputs as well as analog (+/- 10V) outputs and on the analog outputs every now and then, the value would increase about 0.5 in one axis.

The electrical guy went around thinking it was a configuration issue on the MRU... and to be honest I got sucked into it until the operator asked how come the Serial line values were correct and the "other" values weren't.

A couple of minutes later I was going through the junction box and found a jumper that sits on top of the terminals that was loose... when we tried tightening it, it wouldn't make a contact at all. It seemed like the screw was one size smaller than the terminal... this was in operation without any issue for about 3 years before it happened.

 

[just the cowboy]

IC chips

One of the first NC machines I worked on had 3 doors stacked in a cabinet that were 3ft x 4ft and were covered with IC chip sockets (around 10,000 I think) that were wire wrapped on the back side to form the required circuits. EVERY time you opened a door to work on something, something else would stop working. THAT was the worse design I ever saw. The machine was down more than up.