Worst thing to ever happen!!

A colleague working on a SEL protection relay accidentally triggered a Remedial Action generation shedding Scheme (RAS) that tripped over 1300 MW of hydro electric power generation. joined the gigawatt club!
 
A colleague working on a SEL protection relay accidentally triggered a Remedial Action generation shedding Scheme (RAS) that tripped over 1300 MW of hydro electric power generation. joined the gigawatt club!

Wow, I'm going to say that you probably don't want to be in the gigawatt club. Unless of course you own a working flux capacitor.
 
About a year into doing field service work, I get called into looking at this machine that the customer says is discharging into the door. Well when I get there and it gets about 3/4 into the cycle a bolt Flys into the side door, about a foot of space between these two components. Should have walked away then. But I kept poking around, I was still learning right..
so no documentation no prints anything to tell me what this piece of equipment does. Turns out I can't get any voltage readings off of this thing whatsoever. So I called in backup, co-worker, about as green as me so he took a stab at it. Put his meter leads on a portion of the discharge unit when we thought it was powered down, and just melted both meter leads onto this thing. Literally looked like he had two welding sticks in his hands trying to measure voltage of some kind.
After that we told the customer we have no idea how this functions and left.
To this day I still honestly don't know what that machine was.
Some type of High frequency generator I think, with multiple glass bowl/tubes that had glowing red filiments in them. I think I still have photos of the glass bulbs.
Nothing bad happened but about 15 more minutes on site and I'm sure it would have. Lol
 
Not one of mine - but I was sent to fix it.

A Crown Court where major criminal cases are heard.

The judges have a button on their desk that will electrically lock the door(s) in case a defendant decides to 'leg it.
There are a few other buttons in critical places that will lock all court doors and others.

The company I worked for had refurbished the court and up until now the buttons had never been used.

A judge had pressed his desk button and locked the entire building! (re-pressing did not unlock anything)
My first problem was the police would not let me park in front of the court. (They had all been locked in for nearly 2 hours by now)
so 1/2 an hour extra was wasted in finding a parking spot and walking back.

The control room was electrically locked too - so I couldn't get to the panel.
They sent for a battering ram.

As I was walking around the corridors there was a criminal sat on a bench handcuffed to two jail wardens. He asked me if I had come to fix the door problem.
'Yes'
He told me in his criminal language that I should make haste to remedy the situation as quick as was humanly possible as he was waiting to go into court 4 and if it was cancelled he would have to go back to jail until another date could be set. Furthermore, if that happens, he will personally make sure that I expire. (had it been filmed and shown on TV every second word would have been bleeped)

I finally got to the control panel and was faced with an absolute mess.
No wire markings, no drawings and all wires the same colour.
Of course I eventually worked out how to temporarily open all the doors with a linking wire.
As I put the link in, there was a big arc as 50 door locks energised and thud thud thud around the building as they did.

I left it in and told my boss to get the lads that wired it back to sort it out.

I walked around the building checking all was well and occasionally looked out of the windows into the rear security area. I saw the criminal being loaded into a prison security van obviously not having his case heard.

It got in the papers and local TV news: ' Today at court, everyone was locked up including the judges'
 
Late in the startup of a plant I was tidying up the loose ends, and early that day the maint guys asked that I clear the line if product from last night in an area so they could do some PMs/training of the new guys. Sure thing, no problem. We've run the line dozens of times so far full and empty of product. about a half hour before they were to do their training I started firing up the line to empty it out, leaving the infeed off so as to not load more in. It takes a little bit for the line to start up, motors have to spin up, you can hear them quite well. After 30 seconds or so I realize it doesn't sound right. It spins up to a pretty loud high pitch.. it didn't reach there. Stayed lower with the wrong harmonic. SO I look over. I don't see my piece of equipment I expect to see. Its about 28ft long, 8ft high, 5 ft deep and RED... What I do see is a cloud of smoke about 40ft long, 20ft high... and probably gotta be 15ft deep.

F***! Slap the Estop to kill everything, then throw all fixed run switches to stop (just in case someone pulls the estop back up) . About a minute later guys with fire gear, supervisors, etc come pouring in from outside, we check and verify there is no fire so we can relax a bit. Lockout time and start investigation. Fortunately we only burned some belts. While we're talking it pops into my head.. I saw a mech walk over to the air supply lockout and put his ganglock and lock on it. But there was no sound... Normally when you let the air loose from this system there is about 5 seconds worth of a seriously loud venting. So while the mechs, supervisors check over the equipment itself further for possibly more hidden damage (none was found), I found the source of the problem.. Remember I said the air was locked out? The tags for the Air pressure switches... were reading as True. So I track them back to... nothing. They were mapped no where. Nothing wrote to them to inform them of the status of the inputs. Continued to dig and found what those actual Inputs were mapped to... a slightly different name. Instead of being Zone2, they were mapped to Zone3. Not uncommon as its based on what motor drives the chain.

This should have been caught before we ever ran product. No one ever tested whether it would fire up WITHOUT AIR. The permissive are to prevent startup without air. Problem is when your reading a BOOL that you can toggle and nothing is writing it to a certain state... your permissives are no use. Corrected it, found other places the wrong tags were used and corrected them. Made some screen caps of before/after of each wrong tag location for my incident report.


Why did the belts get burned? Well with the air off, the blades drop down into cut position, however by chance someone had previously stopped the line with the blades embedded in wood. the blades could not turn with the lumber holding them. Which is why air is a requirement for power up of that motor. Air on is blades up, logic forces all blades up during startup.
 
These two are from my previous employer and just witnessed by me.. Maint Superintendent (guy over the supervisors at this place) who claimed he knew/could do it all.. Took some meter readings on a dumper control box, decided since power was on one wire but on another and that connecting them together would make it work to complete the circuit.. so he did it. it began working again. 15 minutes later all hell breaks loose when the building fire alarms go off and they report the MCC is on fire.


we use a FLIR to find the hot box as we could smell a paper burn smell but thats it, no smoke to be seen. Find it was the MCC bucket for the peice of equipment he worked on. what did he do you might ask... what he wired down below.. jumped the fuse. What is a fuse supposed to do guys? Yea.. so what happens when you BYPASS the fuse? You burn the transformer.

the second one... same guy..

Troubleshooting a vacuum sealer while the rest of the maint team was battling other problems. He Jumped one contact to another and couldn't figure out why it wasn't working... He jumped a 24v supply contact to a contact with 24V on it already and couldn't figure out why did didn't work. Pointed that out.. he said really? Looked again and went.. well thats not gonna work now is it...


Why was he on the floor? Trying to help fix problems as we'd had a couple call outs so there were only two maint techs that day and we were overwhelmed.
 
Used to work at an ordnance plant that made 25mm HEI (High Explosive Incendiary) rounds. All of the filling and presses are in a remote building/bunker. I was sitting in my office one day about 2 miles away and all of the windows in the office blew out and everyone's ears were ringing. It even blew out windows in town 5 miles down the hill. Turns out the filling station had gone off High Order with about 70 lbs of PBXN5 explosive (7 oz will blow a whole in the side of an armored tank). We found the 1" thick aluminum indexing plate turned inside out and on the top of the hill 1 1/4 miles away. Amazingly no one got killed, although a couple of the operators lost some hearing.
 
There's been a few times I've wondered why my values aren't changing and logic isn't doing what it's supposed to do until I realize I'm off line. Really.
Back in the day before I got totally into electrical, along with controls, I had been a diesel mechanic. Since I worked out of the Northwest, I did a lot of marine application including generators, so got pretty good with power generation. I was the lead operator and chief mechanic at a brand new plant in Galena Alaska. 6 V-12 Mitsubishis. There was a day tank in the engine room and the valve feeding the engine supply fuel manifold was dripping a little. Since I wanted a pristine engine room floor, I started to look at why it was leaking. Didn't know too much about those valves, weird that the valve wheel was made of aluminum and had left hand threads. I took it off, and noticed the fuel was coming from around the packing gland nut. 3 engines on line BTW, and minus 50 outside. Put that crescent wrench on that packing nut and loosened it to give it a chance to seat and then I was going to tighten it and BANG, that valve stem dropped immediately and it got real quiet in there in about 3 seconds while I was frantically trying to open it again. Turns out it was a fire safety valve--a fire would melt the wheel, and the stem would drop, thus the packing nut a little loose. We were training young folks as operators and teaching them electrical theory, and they were always a little edgy around that plant. You should have seen the look on his face when we went black. I ran into the office and said "it was me, start the stand-by!!" The superintendent was in his dark office along with the Mayor of the village. He asked me what happened so I told him, and he said, no problem, at least you were trying to do something.


Lastly, a co-worker of mine when I worked at another company a few years ago did a download at Ford final line in Oakville, Ontario and it went down for 4 hours. The boss had to catch an emergency flight out from Detroit and get it going. The co-worked never did tell me exactly what he did. Ha ha, nobody's perfect.
 
Used to work at an ordnance plant that made 25mm HEI (High Explosive Incendiary) rounds. All of the filling and presses are in a remote building/bunker. I was sitting in my office one day about 2 miles away and all of the windows in the office blew out and everyone's ears were ringing. It even blew out windows in town 5 miles down the hill. Turns out the filling station had gone off High Order with about 70 lbs of PBXN5 explosive (7 oz will blow a whole in the side of an armored tank). We found the 1" thick aluminum indexing plate turned inside out and on the top of the hill 1 1/4 miles away. Amazingly no one got killed, although a couple of the operators lost some hearing.

I did a project at a facility making air bag inflation igniters, basically a blasting cap used to ignite solid rocket fuel, which is what inflated the air bags. The machine I was working on was a robotic turn table with about 15 stations on it to pisition the little aluminum cups, check them, fill them with explosive pellets, cover them with foil tape, insert the electric igniter probes, with check and double check stations in between. For safety during our testing, we used peletized baby powder as a substitute for the explosive. When things went wrong, the pellets would spill and bounce around and a supervisor would come on the intercom pager from his observation room behind a blast door and say “You’re dead!” Finally after a week of testing and troubleshooting we went an entire 12 hour shift with no spills, so they decided to do a “live test run”. I was in a side room behind a “bullet proof” thick plexiglass window, but everyone else when into the other room behind the 3ft thick blast door to watch via camera. I asked why there was no blast door for me, they said I was in a “sacrificial” zone! Nothing happened obviously, but about 6 months later an A-B technician was doing a similar project in another part of the plant where they did the larger canisters of rocket fuel, and it did go up, killing him.
 
Last edited:
A colleague told me a story from a previous job. He had an EE student working with him on an internship. They were preparing the control panel of a large machine. It contained, among other things, quite a significant number of VFD's. The student did quite some of the wiring. He was supposed to have his work checked, but my colleague was out on a service call. He was quite confident to have done it right, so he decides it is time to switch on power for the first time. Poof. VFD's blow out the infamous magic smoke. All of them.

No problem, student diligently takes out all the blown VFD's. Gets 8 brand new VFD's from the stockroom and wires these in. Power is connected, in order to troubleshoot once more. Poof.

Back to the stock room, another batch of VFD's. My colleague came back from his service call just in time to prevent a third round. All VFD's were wired incorrectly. Very consistently.

That was an expensive day in the shop. No, I was not the student.
 
Literally my first day working for Allen-Bradley doing telephone technical support, I took a call from a guy troubleshooting a 1336 Plus VFD.

A senior support engineer had volunteered to supervise; he had a second phone and was listening in. I had my user manual out, and a notebook to draw diagrams and take notes, and was ready to prove how good a troubleshooter I was. I had been hired largely on the strength of my experience with the 1336 Plus.

The customer seemed a little overwhelmed but I walked him through the basic troubleshooting. It seemed that the drive's DC bus was not charging, so I asked him to verify the DC bus voltage.

"Be sure your meter leads are set up for 1000V range."
"They are."

"Double-check. We were just measuring a 4-20 mA loop, and voltage is in different plugs on that Fluke you have."
"I've got it."

"Okay, I want you to measure between the DC- and DC+ terminals at the bottom of the drive."
"OK...."

BAM !

There are some scrabbling noises on the other end of the line. Eventually the customer finds his glasses and his phone.

"It blew up !"

"The drive or the meter ?"

"The meter ! It just blew up !"

"Are you OK ?"

"I can't hear you so good. Yeah... oh, you said VOLTAGE."

I looked up and the senior engineer had tears of laughter in his eyes and had muted his microphone. He was waving another manager to come over and listen. I was horrified.

Since then I've worked in places with rocket fuel and lagoons of pig **** and giant rotating pulverizer hammers and the occasional kidnapping by armed insurgents, all without incident.

But it was a pretty strong impression on Day One.
 
The worst thing I recall doing was finding a loose wire by a relay that it fell out of. As I moved it back to it's terminal it touched another terminal just enough for a little spark.



Turned out the loose wire was +24VDC power and the other terminal was 120VAC HOT. almost every 24V device that was powered up fried. Luckily it was a small line and I swapped everything out in less than an hour. Haven't made that mistake since.


My favorite horror story is I had a maintenance tech working for me thought he could fix anything, and his favorite answer was "Sure, I fix photocopiers" - cracked open one the size of a small car & caused $5,000 damage without finding the $20 part that went bad. NOTE: Never hire a maintenance tech with a celebrity's name (especially if it matches one from Miami Vice)
 
I remember a bad day with an old Hayssen bagmaker. The registration unit was bad. I grabbed a used unit and put it on - bad. Not terribly unusual to have bad used parts. Tried another used unit - bad. Grabbed a new unit - bad.

Now I finally had a clue something might be up. Tried all four bad ones on another machine - bad. Did someone destroy a new one and put it back? Unfortunately that was possible. Checked out power at the connector and 24VDC seemed fine. Got another new unit and tested it on another bagger to make sure it was good. Temporarily connected to the first machine - good. Yay! Finally! Mounted it to the frame and powered up again - registration unit bad.

After more voltage testing and some searching, I finally found the problem. An operator had adjusted the film carriage farther than normal, all the way tight against one side. A screw mounting the low film switch had pierced the insulation of the wire to that switch and tied +24V to ground. Most machines had the 24V negative grounded. This machine didn’t. The registration units apparently didn’t like a positive ground. I think they were about $800 each.
 
Last edited:
Had a guy switch a 600A 480V breaker OFf and nearly deafening all of us at an aerospace plant. The dead front cover screws were missing on one side and he pushed the thing into the bus ! Tripped a 1600A ahead of it, and took out two breakers and damaged all the wire in the cabinet.

But my most recent experience was on a stupid residential job, walking through a large attic on the framing. Slipped on some insulation, and tripped shearing off the fire sprinkler 1”pvc pipe at about 140lbs of pressure !!!! Was down the ladder circling the house Couldn’t locate the water shut off for about 4 minutes and after trying to freeze up the water main at the street !!! Worst thing that I caused in about 30 years ��
 
I have found that after many many years in this game, if you are going to blow something up or destroy something there will always be an appreciative audience.

I blew an inverter up in an engineering shop while they were all sat at their machines having a cup of tea. The cheers and banging of metal cups was louder than the explosion.

I put a short on a water cooler in a canteen while 200 women were having lunch.
The bang, sparks and smoke made them all scream as if they were going down the first hill on a roller coaster.

In an industrial museum I connected a motor the wrong direction without checking with the belts off.
It was an 1800's weaving loom made from cast iron and it collapsed into itself.
Of course there was a school outing watching my every move.

And finally, one wintery rainy day, I was walking across a brick manufacturers yard loaded up with a laptop bag and tools in either hand.
Huge puddles were forming but they didn't look very deep. (I didn't know there was about 3 inch of brick dust mud at the bottom of them)
I got to the middle of one and one of my legs slipped away, I recovered but then my other foot lost grip and I started to do the splits.
I went down very slowly until I was completely sat in the cold very muddy water.
I tried to get up without putting the laptop bag and tools in the water and splash- I and everything went in again.

When I finally got up coated from head to toe in mud, I looked around to see if anybody had witnessed it. There was no one around.
Then I saw a lorry with a driver reading a newspaper covering his face.
The shaking of the newspaper betrayed him...... he was laughing his socks off behind it
 

Similar Topics

Ran a firmware update on a datalogger and this is the horrific splash screen it showed when the update is running. Genuinely thought I'd bricked...
Replies
8
Views
1,251
Posted just because it's has an Allen Bradley keyed selector switch https://www.youtube.com/watch?v=YeFevEGoPF0 LockPickingLawyer
Replies
6
Views
2,672
Can anyone show any worse example for this indescribably tacky marketing approach for industrial components? Control Techniques make GREAT drives...
Replies
1
Views
3,741
For years I believed that Siemens Simatic Basic panels where the worst panels ever with no reason to exist. That was until I was forced to work...
Replies
8
Views
3,123
Hey folks; I have been a long time lurker and an occassional poster and was curious if you think that PLC.net has got to be the best or worst...
Replies
17
Views
9,747
Back
Top Bottom