Worst thing to ever happen!!

After a kamikaze all night session trying to prep a piece of theatrical automation for air shipment from the greater Chicago area to a nightclub in Singapore, I managed t knock out all the lights and other power in our shop.

We were using a step-up transformer to get 460v 3P for testing, as that's the voltage in Singapore. Our shop only had 208v available. We'd wrapped up testing and my co-worker had gone to get the box truck we were going to load this thing onto. I was SURE I'd shut off the disconnect to the testing transformer, and I proceeded to pull the temp power leads off the main breaker's lugs in the shiny new panel.

I hadn't shut off the disconnect.

The three phase conductors slapped together and knocked out power to the entire shop in a large blue ball of fire. To this day, there's still a blot of vaporized copper on the side of the cabinet that's now in Singapore.

To make it worse, it tripped the main breaker instead of the branch breaker (gotta love old buildings), so we couldn't use the shop's chain hoist to get the thing off the build table (about 900#) and onto the waiting pallet. It was 5 in the morning, and I had to call the landlord to come in and reset the main breaker in the electrical vault for our shop space.

Lesson learned - sleep deprivation makes people STOOPUD. I'm lucky I wasn't hurt.


-rpoet
 
First of all, I am sorry to hear Lancie incident. But also salute his dedication to this forum, helping us without any interest.

Mistake:
While learning PLC, I did the mistake downloded the same program on different robot cell which was on different IP address. Without realising mistake, I kept playing with door open/close from HMI. I couldn't understand why the cell has no effect whatever I am doing on panelview. Suddenly one operator came to me, and said, "Why that window is opening / closing by itself". I didn't understand but stopped there and went to that cell. I told him nothing wrong. And went back to same cell where I was programming. And started again with door button. That operator yelled, "it is moving again".

I realised right away, I have programmed wrong IP address in panelview which was controlling the other cell. Luckily doors was linked to hardwired relays of light curtain.

"ALWAYS READ/CONFIRM IP ADDRESS BEFORE DOWNLOADING PROGRAM - WHEN WORKING LIVE"
 
In my early days of programming I retro fitted an old machine with new electrics including a lovely new PLC.

All the wiring and hard work had been done - now for the good bit, the programming.

I started with a blank page - no planning or programming on the bench first - straight into it, online.

I knew the sequence of the machine very well and programmed one stage at a time, testing the program as I went.

I programmed a motor to come on and tried it. I couldn't see the motor but the output went on as expected. Nothing happened???

As I was studying my logic and maybe presuming the motor's belts weren't on I saw clouds of blue smoke coming from the area of where the motor was.

A mechanic had left a spanner in the gears and the dumb dumb programmer hadn't yet put the overload in the logic or hard wired one side of the overload directly to the coil.

Yep, I killed that motor. :(
 
First of all, I am sorry to hear Lancie incident. But also salute his dedication to this forum, helping us without any interest.

Mistake:
While learning PLC, I did the mistake downloded the same program on different robot cell which was on different IP address. Without realising mistake, I kept playing with door open/close from HMI. I couldn't understand why the cell has no effect whatever I am doing on panelview. Suddenly one operator came to me, and said, "Why that window is opening / closing by itself". I didn't understand but stopped there and went to that cell. I told him nothing wrong. And went back to same cell where I was programming. And started again with door button. That operator yelled, "it is moving again".

I realised right away, I have programmed wrong IP address in panelview which was controlling the other cell. Luckily doors was linked to hardwired relays of light curtain.

"ALWAYS READ/CONFIRM IP ADDRESS BEFORE DOWNLOADING PROGRAM - WHEN WORKING LIVE"

A couple of years ago one of our UK plants developed a problem where part of their line was stopping for no apparent reason. One of my colleagues attended and found that the system was receiving a stop command from the downstream system even though the PLC for that system was not sending any such command. After much head scratching he eventually looked at the open connections on the Control Logix ethernet card and found an IP address he didn't recognise. The address was still, however, one of ours so he could get it identified by our IT people. It turns out that the integrator who had installed the system had reused the code in another one of our plants and had not changed the IP address in a message. The UK plant was responding to an interlock with the downstream section from a plant in the USA.
 
Ha, not the worst I've ever done, but something along with the above post that happened last Friday. Was working on our relic Square D SyNet, which was down, no communication to seven important pumps. Did some Network Interface Module swapping between the head end gateway and one of the modules in one pump. No problem, I checked all the DIP switches, everything looked ok. Swapping the module did not help, so I gave the pump module back to the tech to put back in the pump. Many hours later on Friday, network still down, but they were able to run in Hand over the weekend. Finally during more troubleshooting on Monday thought to myself: Did I change the Network Address back to the Pump Address after I gave the comm module back to the Tech? A quick check confirmed I had not, thus two comm modules with address '00' were on the network. The comm lights on the modules kinda looked like communication was taking place - as it turned out it was - but with 100% collisions. As Andybr and yasirkhi noted ALWAYS check the network address....
 
After a kamikaze all night session trying to prep a piece of theatrical automation for air shipment from the greater Chicago area to a nightclub in Singapore, I managed t knock out all the lights and other power in our shop.

We were using a step-up transformer to get 460v 3P for testing, as that's the voltage in Singapore. Our shop only had 208v available. We'd wrapped up testing and my co-worker had gone to get the box truck we were going to load this thing onto. I was SURE I'd shut off the disconnect to the testing transformer, and I proceeded to pull the temp power leads off the main breaker's lugs in the shiny new panel.

I hadn't shut off the disconnect.

The three phase conductors slapped together and knocked out power to the entire shop in a large blue ball of fire. To this day, there's still a blot of vaporized copper on the side of the cabinet that's now in Singapore.

To make it worse, it tripped the main breaker instead of the branch breaker (gotta love old buildings), so we couldn't use the shop's chain hoist to get the thing off the build table (about 900#) and onto the waiting pallet. It was 5 in the morning, and I had to call the landlord to come in and reset the main breaker in the electrical vault for our shop space.

Lesson learned - sleep deprivation makes people STOOPUD. I'm lucky I wasn't hurt.


-rpoet


Yeah, those are spectacular - glad you weren't hurt!

Years ago I worked for an OEM making electric furnaces. They ran usually 300kVA to 500kVA, although my personal best is a 1000kVA glass furnace. Anyway, we had Square D bus duct in the shop and we'd just move the disco box to where ever the furnace was and wire it up. The duct runs were rated for A LOT as you might guess. So, one time the electrician wired in a furnace comming up towards testing, and closed the box (thank god!) and pushed up the disco to connect the box to the oven enclosure. Noise and light. Briefly. Entire building went down.

He opened the box and found what was left of his screwdriver lying across the top of the lugs... IDK if he got two or all three phases but it was memorable.
 
The UK plant was responding to an interlock with the downstream section from a plant in the USA.
Great balls of fire! :eek::eek: I dread when the bad guys learn how to do that.
 
Last edited:
I had one like Andy's too ! A PLC-5 in Texas was inadvertently sending messages to the PLC-5 I was working on in Arkansas.

It wasn't quite as elegant as an interlock, though. By coincidence they were writing to the command data file for a twenty-ton crawler I was standing next to when it started moving, without anyone at the control console.

I think of that crawler (and the space between it and the concrete wall I was standing in) whenever I explain why programmable control systems need independent emergency stop circuitry.
 
I was also in a robot cell and it was the first time I used multi move system. There I was trying to jog a robot and could not understand why it was not moving until I heard a crunch,
Donnchadh

This hits close to home. I did the same however the robot was plasma spraying. It lit the HVOF torch in an adjacent cell, cut a $20k jet engine part in half. To make matters worse, this was a rush part for an engine that was on a stand, waiting for it. I still received a raise that year, don't know how.
 
... but I do wish I could say a few words to that Army guy who told me "go ahead in, that old building is safe because we tested it out completely".

Sometimes we yield to the "experts" a little to quickly. I had a toxic gas valve that was leaking down in a pit. Overhead are two large furnace vessels at 2000-degrees F. Inside the reactors (right over my head) were flowing Hydrogen and Hydrofluoric Acid. The exhaust to the scrubber is a 6" steel pipe. It had become clogged because water from the scrubber was making it past the venturi and into this pipe. Over time, the pipe became completely blocked so we were experiencing plant evacuations because of HF gas being released into the atmosphere. My boss said "the Oxygen sensor is reading 20.1, you can go take care of that sticking valve now". I was in the pit for less than 15 seconds when my face, lungs, eyes, and even ears, began to burn. I didn't think I was going to make it up the stairs to get out of the pit. Turned-out the technician that said the Oxygen Sensor and display had been properly repaired, lied. He didn't even certify it and no one questioned it.
 
Oh, I forgot about this one:

I was replacing an AB MCC Panel door for a size one starter while the machinery was running. Hey, my boss told me to! Well, as I was removing the door hinge pin, my right hand pinky knuckle touched the screw of the overload block, and for a second I could feel all the muscles in both my arms and chest tighten up, since I was holding the panel door with my other hand. We're a 480v plant, so I had 277 to ground.

Lucky it was just a glancing poke and not a solid hit.
 
Oh, I forgot about this one:

I was replacing an AB MCC Panel door for a size one starter while the machinery was running. Hey, my boss told me to! Well, as I was removing the door hinge pin, my right hand pinky knuckle touched the screw of the overload block, and for a second I could feel all the muscles in both my arms and chest tighten up, since I was holding the panel door with my other hand. We're a 480v plant, so I had 277 to ground.

Lucky it was just a glancing poke and not a solid hit.

Next time tell the bass to do it! :mad:
 
I'm sorry to hear what happened to you Lancie. Everything I have done so far doesn't compare to some peoples stories, but then again I'm fairly new so give me a couple years and I may put a plant on lockdown.

There was one time I was wiring up a relay and I thought to myself "I wonder what would happen if I shorted one of the normally open contacts", so I did just that so I turned on the power and BOOM! the relay dissapeared. So... it did what I would expect it to and then told myself it's not a good idea to do something like that again.

Another thing I have done is do programming changes offline, then downloading that program with all the updates. From that day on I found out how awesome online editting or just uploading all current data can be. What I did was whenever the last time that program was saved offline, I just sent a whole bunch of bad values to one of our plating lines. Took them about 5 minutes to get back up and running again.

You always learn from your mistakes!
 
Got 12 azonix touch screens and server sent from our US office to the UK, all EX equipment to be installed on a oil rig so not cheap.... Plugged the server and 4 screens into a 5way extension to test ... flipped the switch and poof.... forgot they were all set to US voltage for the rig supply....
 
I took down 2 production lines changing a lightbulb. Now that takes skill

Thank you dbh6 for initiating this thread! When I need a laugh, I just open this thread and seeing how others have been there and done that, it enlightens me. I am still waiting to read the specifics about the light bulb bringing down 2 production lines. This is by far the funniest.
 

Similar Topics

Ran a firmware update on a datalogger and this is the horrific splash screen it showed when the update is running. Genuinely thought I'd bricked...
Replies
8
Views
1,250
Posted just because it's has an Allen Bradley keyed selector switch https://www.youtube.com/watch?v=YeFevEGoPF0 LockPickingLawyer
Replies
6
Views
2,672
Can anyone show any worse example for this indescribably tacky marketing approach for industrial components? Control Techniques make GREAT drives...
Replies
1
Views
3,741
For years I believed that Siemens Simatic Basic panels where the worst panels ever with no reason to exist. That was until I was forced to work...
Replies
8
Views
3,122
Hey folks; I have been a long time lurker and an occassional poster and was curious if you think that PLC.net has got to be the best or worst...
Replies
17
Views
9,747
Back
Top Bottom