Ever done this?

I was called out to our new 2000-ton press today and found something very interesting. The press has a hydraulic ram that feeds metal blanks up into a feeder. The ram has two sets of sensors that shoot across the top of the blanks. One set is for material in position and the other is for safety, so the ram doesn't continue pushing on the blanks ramming them up into the feeder. The ram feeds the blanks up into the feeder by preset timing from the particular job that’s being run.

Well, as I walked up to the operators, I could see the look on their face and new something was wrong. The ram for some reason didn't stop and continued to push on the blanks and bending some very expensive parts. This being a multi-million dollar press brought out all the Japanese and upper management. The boss told me to investigate why this had happened. I hooked up to the Mitsubishi plc and started looking for the sensors in the program. After finally finding them in the program and hardwiring prints, I found something that I could not believe.

There is a line in the program that has the two sensor bits for safety, "Keeping the ram from pressing itself through the feeder", and it turns on a fault when the sensor senses something. This fault line is used to kill the hydraulic ram solenoid, before it does damage. Well, on the fault line was a normally open "ALWAYS OFF BIT" before the fault output. It seems that when the manufacturer was here they disabled this fault from ever happening by putting the always off bit in the fault line. I know that I use this "ALWAYS OFF BIT" to do some troubleshooting, or when I want to "TEMPERARLY" disable something in the program. It's just a little easier then deleting the whole line, but of course don't forget to take it back out. Anyone ever done this and find out later on down the line that you forgot to take it back out? The manufacturer is coming in tomorrow and I'm curious what they'll say when I show them their screw up.

I have burned myself with that in the shop, but never had it happen in the field! Er, at least not to my knowledge... utoh

Since I occasionally use "ALWAYS ON" or "ALWAYS OFF" bits in the final program, I have made it a habit to use a DIFFERENT bit for debugging. I just pick an unused bit and call it "DEBUG". I use this "DEBUG" bit to override/disable code during the debugging stage.

A quick search for this "DEBUG" bit before shipment, and stuff like you describe doesn't happen.

I'll bet if YOU didn't find it, the manufacturer would deny it was their fault. Guess who would eat the repair bill?...

Time to ask for another raise, Tim!... :nodi:

beerchug

-Eric

I use Siemens S7, in statement list representation exclusively, and I often need to "jump out" some logic, or toss in some test code for some reason. In that case, I'll always insert the text line "//Test!!" before and after the code. Then, because it's sometimes hard to remember where all the patches are, I'll generate a source file for the entire project, and then just do a search on "text", and I can easily find all of my test code and remove it. It's the safest, surest method I have found.

And congrats on your find.

Tim, I hope you made a copy of what you found. In the health care field, it is pretty hard to prove a cause without documenting everything.

It good when you find something like that. You stare and cant believe what you are seeing.
You then double check to make sure you aren’t missing something before the full realisation dawns on you.
It good that you have posted this because all reading will be thinking ‘ I hope I haven’t left something like this out there’ A jog against complacency always helps.

A programming fault I once found (not in the same league as yours though) was on an industrial spin dryer. It had 2 sensors to detect if the spinning cage was out of balance.
The first one detected small imbalances and the second really bad ones.
The machine was about 5 years old and until this day I went, it had never had a big imbalance.

It had had one and would not restart. When this had happened a retentive internal bit had been set with absolutely no way to reset it without some reprogramming. I could never make my mind up wether the manufacturers had done this on purpose (so that they were called to it) or wether it was a programming fault.

Goody,
You’re absolutely right. I looked at this for about an hour trying to find some way to not believe it. I guess I couldn't come to the realization that someone who sells, or programs a multi-million dollar piece of equipment could make a mistake. I guess were all-human whether we do this stuff full time or not. I'm meeting with the manufacturer today and will post their response.

BDKuhns,
Yes I have made copies of the program. It’s over 10,000 steps long. They also have their own scribble marks all over the program where they were doing some changes and the page where the error is at actually has some program corrections on it in their own writing.

I alaways do this method of testing in the commissioning stage.it's normal to want to disables the fault conditions or protection during this stage using a NO Contacts Or deleting the sefety condition.
Usally i do this modifications away from the original program i save a new copy of my program make all what i want to it and after finishing i download the original program.
I agree with eric that the manufacture will denay it was thier fault and may be they accuse you and telling you that you are who did that in the program not them.
BEST LUCK

I had same problem so I decided no more "Always On" or "Always Off"
bits in my programs. Some platforms dont even let you do a crossref
to see if these bits are used or not (see AFI in RSLogix5000).
My solution is to create small file just for debuging. Whole file is
cleared on powerup by "First scan" (FLL in AB, or FMOV in Mitsi).
Then I use those bits as normally open contacts for "Always Off" or
normally closed for "Always On".

Benefits: This way, I don't have to edit online to disable or
bypass logic, just toggle bit. And if you are on your way home and
customer calls you with horror stories, just tell them to cycle power
to equipment.

Yep,
They denied it and tried to pull a fast one on me today. They didn't deny that they didn’t put it in there, but they said it was supposed to be there.
The story goes like this...
They,"one American and one Japanese" came in today to look at the problem. The American guy says to me,"Yep your right it probably shouldn't be there". He later changes his decision, after we told him about his company replacing any bent parts. The Japanese guy has to be showed numerous times that we had a programming problem. After, a few hours of them crawling through the program the Japanese guy say's he needs to use the phone. He goes outside for about a half an hour and comes back in, looks around a bit more and says to me that the always-off bit was supposed to be there. I say to him.."OK so what keeps the ram from ramming itself through the feeder?" He says.."I need to make another phone call", and goes outside again for about a half an hour.

By this time, I know I'm getting scammed. I don't think he realizes that I've been study this program for some time now and I'm ready to fire back with any bull**** being throwed my way. He comes back in and looks around a little more and tells me that there are numerous places in the program where they have the "always off bit". I say to him.."Why?" He tells me that there are several different applications of this particular feeder that use some ladder rungs and some of the rungs are not needed, or will not work with a certain application. I tell him,"so what your saying is, when you install a feeder on a press the programmer will go through and take out any always off bits that need to be taken out for the particular application?" He promptly agrees and I tell him,"Ok, so someone forgot to take this one out, thus not allowing the fault to occur and stopping the ram before it does damage."

I leave them alone for a while and come back about an hour later. I asked them if the problem was fixed and they say, "yep got it all taken care of". I had them show me what they changed and all they did was put a damn bypass on the bypass. I looked at that and said, why didn't you just take out the always-off bit and it would do the same thing? They again, reminded me that rung was for a different setup. I'm getting mad at this point and say to my boss that this is a bunch of bull****. He doesn't know much about programming, so he goes by what I tell him.

Anybody ever been through this kind of ****? This is my first time and hopefully the last.
Tim

Hi Tim, Yes I have been through the same BS, just keep you cool, and take the time to ma sure that all the ducks are lined up and sitting correctly, and document everything, with a film camera if you can...some PTB still will not accept a digital picture...'you did something to that picture'. Good luck , I hope you can get some satisfaction out of the fact that YOU figured out THEIR problem!!!!
David
beerchug

Eric Nelson said:

Since I occasionally use "ALWAYS ON" or "ALWAYS OFF" bits in the final program, I have made it a habit to use a DIFFERENT bit for debugging.

Click to expand...

Just curious. I've never used an "ALWAYS ON" or "ALWAYS OFF" bit in a final program. When would it be needed?

I had a machine that I had commissioned nearly kill one of the operators I trained. One of the witnesses claimed that the machine had simply "opened up by itself" while she was unjamming the machine, dropping several hundred pounds of plywood on the poor girl, then "closed up again". When I got to the site there was still blood on the machine frame.

I was told that I would be prosecuted as liable for this mishap, and I was scared as hell... I was only 23 at the time and on my own in a foreign country where I'd already been harassed by the local cops. I couldn't believe that this was possible... the machine was E-stopped when the injury occurred and I would bet my own life on the E-stop circuit removing all output power.

When I got to the machine, there was a new button on the control panel, hand-labeled "open". It didn't do anything when I pressed it. The checksum on the PLC program didn't match, even though the maintenance manager said he'd swear in court that it had not been changed.

I traced the wire from the "open" button to the PLC-5 input module, then checked my archive program and had their manager check the running program in the controller (I was not allowed, obviously, to touch the system). The input wasn't used in either version of the program.

I was finally able to prove to the customer that the PLC had nothing to do with the operator's injury. She was a slender girl; maybe 110 pounds. The arms that held the wood panels are pneumatically actuated, and have a lockout valve that vents all the pressure out of the machine. She'd pressed the plunger on the lockout valve, but had been unable to get her lock hasp through the bottom of it... because she hadn't been strong enough to press it all the way. She left the lock off, climbed inside the machine frame, and proceeded to attempt to un-jam the machine. Several minutes later, enough pressure had leaked through the partially-engaged lockout valve to allow the weight of the wood to overcome the pneumatic cylinders. Once relieved of their load, the retained pressure snapped them back up into place "automatically".

Two years later, I met one of the contractor electricians from that job in another state at another mill. He told me that the maintenance electrican had added that "open" button to the controller, and associated some logic with it. He had removed it after the accident to make anything that had gone wrong with the PLC controlled machine look like my responsibility.

COVER YOUR BUTT !

paul,
I'm not sure about Eric, but I've used them in certain situations. One is if I have a particular ladder rung that is constantly looking for a changing situation. Example, if I want to move some data to a register at all times, "every scan" then there would be nothing before that command. Unfortuantly, doing this in the plcs that I've used is not allowed.

        | not allowed                                   |

        |--------------------------------[mov k1 d1]----|

        | Always on                                     |

        |----| |-------------------------[mov k1 d1]----|

Tim

Guest,
Wow, interesting story. I know I've come home many a times thinking,” did I cover every possible situation" on a new program, or machine build. I'll tell you that this has got to keep a lot of you programmers up at night. I couldn't imagine having my own business doing this stuff. It would drive me crazy. My hats off to those who do this stuff full time and can keep a sane mind.
Tim

Paul the "Always On" or "Always True" bit is used pretty widely. Let's say you want to temporarily get something going and intend to remove this "jumper" later. If you branch around it, it becomes unsearchable in your program. If you branch around it with an Always True bit, you can search that bit later and remove it.