So --- What do you test for?

After reading the "Control engineers in over their heads" thread, I agree with Eric Nilsen et al, you do have to program a lot of diagnostics into your code.
I normally program the following alarms for diagnostics:
- Circuit breaker/overload
- Contactor fail to close/Fail to open* (although I am not sure the fail to open is actually workable)
- Fail to run, this is via a run proximity, pressure sensor, or something to show that what you want done is actually being done.
- E-Stop activated
- VSD fault
- Pneumatics/Hudraulics Actuator failed to reach position
- Bin/Tank too high/too low.

I was wondering what other alarms or diagnostics everyone else uses?
Also how to display them, I generally have a Citect PC or Touch panel close by, allowing alarms to be displayed via text. However, sometimes a simple push button panel is specified with illuminated lights for alarms, how would you handle this, do you common up alarms, or maybe kick and scream for a text display?

What are your thoughts?

it depends on the project. I can't see writing the code for an automatic fence/gate (plant entrance) with tons of alarms, bells & whistles...

Add these to the first list:

• Input devices that should never be turned on at the same time.
• Conveyor/pallet traffic control: release a pallet and it should shortly clear the stop and should make the next stop.
• Rack or module faults.
• Power supply faults.
• Momentary operators (PBs, Switches) that are active too long

Troubleshooting is one of the reasons that I really like installing and programming HMI's. I will always create a "Input Verify" page along with a "Motor's Verify" page along with some cool animation so the operator can easily understand what he's looking at.

The Input verify screen is a screen that is a quick way of the operator to know if a limit is stuck or somthing of that nature. If everything is ok all the inputs should be green. The limit switch that is stuck will be red.

Of course there is the "Alarms" page but I have found that operators have ahard time with this (maybe because there is no pictures).
Ernie

It really all depends on the specific application. As far as inputs go, you should try to cover all the possible scenarios. Things like disconnected sensors, sensors stuck on, etc... Anything abnormal. I discussed this in Hester's thread HERE.

I monitor the master control relay in the program. The PLC knows when the MCR is on via an auxiliary contact. It also knows at least some of the conditions when it should be off (E-stop pressed, guard open, etc.). I wire one PLC output into the safety circuit through a relay, which allows the PLC to 'have a say' in the matter. If the PLC sees that the MCR is still on when it should be off, I trigger a fault message that this occurred. I also turn off the PLC's output that's in the safety circuit, in hopes that it will drop out the MCR. This won't help if the MCR is welded on, but the fault message will at least inform someone that this condition exists.

Note that this is IN ADDITION TO the hardwired safety circuit. A PLC is not a safety device, but if sure can't hurt to let it at least make a valiant attempt to shut off the machine if the hardwired circuit happens to fail.

Another simple example... You have a pushbutton that starts automatic operation. This pushbutton is broken and is stuck in the pressed (closed) position (yes, this CAN happen). When you turn on the machine, what happens?

Lousy programing result: Machine goes right into automatic operation. The operator thinks "Hey, that shouldn't happen!" and pushes the 'automatic stop' pushbutton, which stops automatic, but as soon as he/she releases the button, it goes right back into automatic. Operator turns off the machine and calls the mechanic, who then has to go get his laptop, find the right adapter/cable for that PLC, dig through the (lousy, undocumented) program to finally discover that the pushbutton is stuck closed. Mechanic replaces pushbutton.

And let me add... Lousy control design result: The PLC is powered AFTER the MCR, so in order for the mechanic to go online with the PLC, he has to have the operator sit there with their finger on the 'stop automatic' button so the machine will not run while he looks at the program...

Good programming result: Machine does NOT go into automatic (thanks to a simple one-shot on the pushbutton input), and the display indicates that the pushbutton is pressed during initial powerup. Mechanic replaces pushbutton.

Which machine does the mechanic like?...

Even if you have no HMI, that's still not a reason to NOT have the PLC tell you E-X-A-C-T-L-Y what's wrong. A single lamp can tell you everything by flashing error codes. Just like the 'check engine' light in a car. "Flash, flash, flash, pause, flash, flash, long pause, repeat" = Error #32, You look up error #32 on the 'fault decriptions' sheet at the machine, which explains the problem.



-Eric

Eric Nelson said:

And let me add... Lousy control design result: The PLC is powered AFTER the MCR, so in order for the mechanic to go online with the PLC, he has to have the operator sit there with their finger on the 'stop automatic' button so the machine will not run while he looks at the program...
-Eric

Click to expand...

Hate that, and its brother - Machine Isolator cuts the power to the PLC. Cannot check for problems while the operators clean up the mess....

If I'm displaying alarms on an HMI (eg Siemens OP7) I find that it helps if I put the Marker number on that caused the alarm. I normally have a separate section of code that has a couple of marker words for the alarms being set by the various conditions needed to bring them on.

Displaying the relevant marker can really help speed up fault finding if you need to trace that alarm in the code.

eg.
MOTOR 10
OVERLOAD
(M56.0)

Cheers

JT

Johnny T reminded my of something that is OT but at the same time closely related to this topic. I HATE the way that most control components are labeled/designated. Example, LS67---PB32--input1. I like the way that alot of our European machines are labeled. LS2A--LS2B--PB2A--PB2B--Input2 and Output3--K3--Motor3. Makes troubleshooting a heck of alot faster and just seems to make more sense when circuits have some type of common designation.

Dale

Another good way to display error or alarm codes on the cheap is to use a Durant Counter. You can pulse one with a PLC output 32 times and it will display "32"... Then when the operator presses reset another PLC output can clear the counter and it will display "0".

Other things to check for:
anything moving/active that the PLC isn't telling to move/be active (ie, conveyor is moving, but PLC isn't telling it to move; flame is on, but the PLC isn't telling the gas valve to be open)

My simple rules for knowing what to test for:

1) Every input should have at least one fault associated with it. For example, a "cylinder extended" limit switch should be made whenever the "cylinder extend" solenoid is energized. (within a few seconds, anyway) If it doesn't "make", it's a fault.

2) Every input should not be made under opposite conditions.

3) Safety monitoring (Example: Guard door not closed)

REMEMBER: This should always be hard-wired using a dedicated safety circuit. I just like to monitor the conditions with separate circuits as inputs to the PLC so the operator knows what's wrong and why no MCR can pull in.

Then I brainstorm...

What faults really matter? (Example: Things that inhibit going into auto when the operator presses "auto start")

Perhaps add "Ready" pilot light? Use simple fault screens or fault codes with "real world" descriptions in addition to just "LS_123456 not made". Unless LS_123456 is marked with a 1' x 4' sign, the operator may not know where that cryptic device is located.

What is awful good to know? (Example: Parts hopper is getting low or PLC battery is low)

Make these just simple "warnings", don't blast the operator with a 100dB alarm horn mounted on the pushbutton console. Save that for "Reactor Vessel Exploded".

Just using common sense and putting your mind into the mind of the operator, maintenance man, safety inspector, etc. always worked for me.

In addition to what others have mentioned already, a couple of things that I add to my alarming/diagnostics code:

Rate of change calculations: If my analogs are moving more rapidly than physics/chemistry (or just the process itself) says they ought to move . . .

Deviations from theoretical: If a "measured" value deviates by XXX from the calculated value . . .

Communication faults



For reference, I usually do my analog alarms at four levels (low, low-low, high, high-high).

I also have these alarms typically come in at three "severities":

Error (the instant that the "bad condition" is detected)
Warning (once the error has been continuously active for XXX ms)
Fault (once the error has been continuously active for YYY ms)


Marc

I do alot of what I've read here so far. To prevent unexpected motion from stuck pushbuttons, I use a short duration timer for each button (except jog buttons) and use the timer timing bit (enabled and not done) to trigger the logic. If the done bit gets set, a warning appears on the HMI. If the timer stays done, it will no longer be timing and no new motion will be initiated.

For sequencers with SQI instructions, I XOR the input word(s) with the SQI file word (indexed for the step) and MVM the result through the SQI mask word for that step. That produces a word with a bit set for each item that needs to change states in order to advance the sequence. There is usually a time limit for each step. When the timer is done, I send this "sequence diagnostic word (or words)" to the HMI where each bit is used to condition the visibility of a message. It will show each and every input condition that is preventing the sequence from advancing. The beauty of this approach is that if you "re-peg" the sequence bits, the diagnostic code still works. The only time you have to mess with it is when you change the definition of a SQI bit. Then you must change the associated HMI message. On one particular machine, I send this diagnostic word all the time so the operator can tell which part of the fast moving sequence is the "bottle-neck" of the process even when everything's running normally.

For testing the final code, I can usually find one of those operators who will try everything for me. I have one guy here who almost daily brings me issues like "When I cover this photoeye and push JOG FWD, AUTO, and FEED at the same time...this happens". I try to think of everything, but someone will usually find something I never thought they'd try.

As for using lamps for alarms, I have taken the time to write code that would flash a single lamp in a pattern (like the CHECK ENGINE light on my old chevy). I did this as a test to narrow down the causes of a problem that was too fast and too seldom to identify otherwise. I ended up leaving the code in place. It worked well with only three or four alarms. Any more than that, and the operator needs a cheat sheet to interpret them. It's doable, but I like the method propsed by marksji using a counter even better.

Another thing that I always test is entry limits on the HMI. I make sure that the values entered are validated and limited if necessary by trying to enter numbers that will cause problems. I got called in at 3 a.m. once because someone accidently entered a 0 for accel rate on a servo controlled machine. The servo controller would not fault because the command and actual position will never change with 0 accel. The maintenance guys had no idea what to look for. They knew the speed value was good, the axis status was "moving". They'd tried swapping amps and motors. The total downtime was six hours by the time I got involved. Simple stuff like limiting HMI entry values can save tons of headaches later.

Like the others, I segregate alarm code into critical (stop the machine) and non-critical categories. I check for unexpected motion (ie. the brake is on, but the encoder is moving), impossible switch states (ie. cylinder extend and retract limits both made), and deviations from the modelled expectations for analogs.

Hope this helps!
Paul C.

OkiePC said:

For testing the final code, I can usually find one of those operators who will try everything for me. I have one guy here who almost daily brings me issues like "When I cover this photoeye and push JOG FWD, AUTO, and FEED at the same time...this happens". I try to think of everything, but someone will usually find something I never thought they'd try.

Click to expand...

Always good to have one of these guys on staff. My boss happens to be very good at this task...



-Eric