Philosophy of Equipment Error Recovery, Alarm recovery, and Re-initialization

Hester

Member
Join Date
Apr 2004
Location
Tacoma, WA
Posts
61
Scenario: It is a nice sunny 73 (F) day outside. Maybe with a slight temperature inversion. Toni Gibbitz is operating his production line, day dreaming of surfing perfect Lower Trestles, with nobody but himself and just a couple of close friends out. Suddenly Toni smells something funny, and not being much sharper than a marble, reaches over and punches out the EMO.
Maintenance investigates the "smell" and informs the production group that the Burrito factory up the street is making Garlic and Bean Burritos today, and the smell has blown down wind into the "Make up" air port, and being blown all through out the factory - all personnel Please disregard.

The next day, Production Manager Lazie Crotcha, wants to know why 15 minutes were lost recovering the piece of equipment that got EMO'ed.

Automation Engineer Milkey Licker, gets assigned the task of investigating this issue. He finds the Operator must completely clear the line of all product and restart it from the beginning in order to get the line running again. Something has to be done about this. It is "ridiculous¨ crows the VP that recovery should take so long. FIX IT!

And here is where I am at: Could you share your basic philosophy to a PLC greenie, regarding Alarm recovery, Error Recovery, and re-initialization?

What are your general rules or thumb, or design guidelines regarding error recovery. Any links appreciated.
Does one need to make a full matrix of all Discrete Inputs and generate code for testing state of each and every one of the inputs?
How do you guys think about this issue?

Thanks to those who wrote here:

Machine Safety: Pausing Index Table? Thread 6717
How To Start A Ladder. Thread 333
Safety Bypass ????? Thread 2293
RSLogix User Fault Routine Thread 891
 
Last edited:
There are some different philosophies.

The first one I used as a PLC newbie was that the sequence should resume where it was stopped. Of course there were some factors that required consideration, such as "has the part moved?" or "has the machinery moved?", or in the case of a batch "has the batch gone sour?"

Initially, I set it up so that the operator need only press the on and off button to operate. This required some complex code that did all the detecting required. And in the case of the concrete mix, there was a 20 minute timer with a dump valve. You let the mix sit for more than 20 minutes and it was dumped.
This ended up being quite reliable with fast (aroung 30 seconds) restart times, no clearance of the line usually necessary, and a happy workforce and management.

Looking at other machines, I have seen two other philosophies. One is to have the sequence operational during manual operations. In that case, the sequence would step forward if the operator created conditions in manual that required a forward step. It also has a sequence step set function, allowing the operator to select which point in the sequence to jump to. This also works fairly well.

One piece of equipment I am working on now requires a full re-start if the sequence is interrupted. The person who wrote the code is convinced that this is the only safe way of doing it. He is also convinced that the operators are a bunch of knuckle dragging morons (they don't like him either). This machine causes nothing but trouble. However, this trouble is not caused by having to reset the sequence for every interruption. Clearing and resetting this machine takes only a few minutes, so I am not going to change this function.

So to answer your question, in theory it is do-able.

It would also be a nice challenge.

Why don't you post some more descriptions of your problems, including some operating constraints, consequences of restarting in the wrong position and similar. This way you can get some more ideas.

Doug
 
I see you've dug through the archives, Hester. Those are good links about your topic. Others may not know how to get to them, so here's 'clickable' versions:

Machine Safety: Pausing Index Table? Thread 6717
How To Start A Ladder. Thread 333
Safety Bypass ????? Thread 2293
RSLogix User Fault RoutineThread 891

Since you're dealing with an 'already built' piece of equipment, the best advice you're gonna get is from the operators. They LIVE WITH the machine, so they are the ones who can tell you the good and bad points about the machine. Ask them their likes and dislikes about the process, and use that feedback to help plan your changes. Explain to them that you want to make the machine easier for THEM to operate. That way, you'll get HONEST answers.

You have to look at all the 'what ifs' in your process. There may be a very good reason why "the operator must clear the entire line of product", or there may not. You also mention they pressed the EMO. I assume that means "Emergency Machine Off"? Do they have to empty the machine after a NORMAL shutdown (not E-Stop)? Do they even HAVE a 'normal' stop button?

Keep a few things in mind when analyzing the process. Above all, operator safety. Then look at stuff like ease of use (for the operators), crash prevention (for the machine), etc...

One 'tip' I'll mention is that E-Stop should NOT kill power to the PLC. It SHOULD kill power to the outputs, but NOT the PLC. I see too many machines where pressing the E-Stop is the same as throwing the disconnect. There's just no reason for this. Think about it. How is the PLC supposed to know what's going on when it has no power?

As Doug mentioned, more details from you = More ideas from us... :nodi:

beerchug

-Eric
 
Challenge? Definitely! Some of the regulars can probably quote chapter and verse on this one easily. I would think that some shift registers would work, but depending on the product, some proximatey sensors might be needed for verification. I would have a fault reset push button input to the plc. If the machine had an alarm condition, then the reset could start with the data stored in the shift registers. Maybe with more sleep, I could explain this one a little better.

If you get a chance, post more info on machine, or sequence of events.


There is always the Sara Lee Help Line, 1-800-FAT-CH..... oh never mind.

bestest regards DH, casey
 
There are two philosophies in programming a production line such as you have described.

One is the "Supply Side Philosophy". The other is the "Demand Side Philosophy".

The "Supply Side Philosophy" says... "Ready or not, here comes the next part... hope you can handle it!"

The "Demand Side Philosophy" says... "Don't send a part unless I ask for it."

I always write my code, as much as practicable, using the "Demand Side Philosophy".

The very foundation of the "Demand Side Philosophy" is MODULARIZATION!

Careful examination of your system will show that there are distinct sections of your process that can be uniquely identified. Each of those sections should be coded separately as distinct Modules.

There are two types of modules in a "Demand System"; the Passive Module and the Active Module.

In a "Passive Module", once a part enters that particular type of module, the part is "committed" to moving through that module without stopping.

An "Active Module" can hold parts. That is, when necessary, parts can be held in these modules. The parts are accounted for. That is, the process is aware that they are there, and, if necessary, can resume running after a stop of any kind. That is, unless that particular module is the cause of the stoppage. If this particular module is the cause of the stoppage, then the module, and the process, "knows" that the fault occurred in this module and only those parts must be manually handled to bring to system back to a good re-start point.

The "Demand" aspect of the "Demand Side System" centers on "Calls" being made by "Active Modules".

On start-up, a "Call" originates from the last "Active Module" at the end of the process. That "Call" is passed by "Flag" to the previous "Active Module". A "Call" is passed from one "Active Module" to its prior "Active Module" until the process finds an "Active Module" that is holding parts. On start-up, that would be the first module in the process.

Now, if, during the course of normal running, a fault, jam or whatever, occurs at a particular module, then all down-line modules should be allowed to continue running... there is nothing wrong with those parts and they should be allowed to continue.

Meanwhile, up-line, from the faulted module, towards the source, the process should be allowed to continue moving parts until they run into "Active Modules" that are NOT receiving a "Call" from the next down-line "Active-Module".

"Active Modules" are "boundary-modules". A "boundary-module" is one where a part or parts can accumulate while waiting for a "Call" and doing so without causing a fault.

If the line is halted, for reasons other than an E-Stop, then parts should continue moving through those modules that are unaffected by the cause of the stoppage.

That is, except for the case of an "Aw-$hit-E-Stop", all modules should come to a "controlled -crash-stop". This leaves all modules ready to resume running - except for the one that caused the stop, if any.

In the case of your "EMO", there needs to be a distinction between "Aw-$hit-E-Stop" and "Stop the Line". In the case of "Stop the Line" ALL "Calls" should be disabled. This will allow parts to move to and fill, to the extent that they can, all "boundary-modules".

That is a "controlled-crash-stop".

In a "controlled-crash-stop" those parts that are already moving through "Passive Modules", where they should not stop, are allowed to proceed to the following "calling" module.
-Unless, of course, the particular "calling module" or "passive module" is the module at fault.

If any particular module is at fault then the down-line portion will continue to run and clear itself. Meanwhile, the prior "Active Module" (boundary) will hold parts until a "Call" is made on that module. As long as the line is "halted" that "Call" will not be made.

So... in a simple "Stop the Line" 'cause I smell something funny... the line will continue running until all "boundary-modules" are holding parts in anticipation of a "Call" which will not be coming.

If it turns out to be a False Alarm, then simply pull the "EMO" push-button and the process will resume without having to physically handle any parts.

In that case, the "Call" will again originate from the end of the line and the process will begin fulfilling demand, from "Active Module to Active Module" as fast as it can.

Just as in the American Capitalist System, better results are obtained through the "Demand Side Philosophy"!


MGD has done what MGD does... and so... I'm done.

I hope what I've said has turned on some lights.


(151)
 
Amen to that, Terry... I've always used the "Demand Side" philosophy... :nodi:

Too bad not every machine builder subscribes to it... :rolleyes:

beerchug

-Eric

P.S. to Hester. What's with the "Welcome to McDonald's drive-thru, may I take your order?" avatar?
 
Last edited:
I just realised that I have been unconsciously programming using the demand side philosophy (mostly).
Another light has just gone on.

Thanks Terry,

Doug
 
excellent points, all

Terry makes his usual great reply. No arguments there.

The key for me in these area has always been the limit of the
operator interface to allow the operator to know just what he
is agreeing to or ignoring.

The process is important on this point, as well.

In a commercial bakery, the product is going to come in on you, regardless of how hard you do not want it to.

OTOH, in a conveyor or material handling application, you might
be able to simple 'stop things' until you can continue.

Lastly, you have to think about 'sensory input'.

I just think: WATERGATE. What do I know, and when did I know it?
 

Similar Topics

Hi all. This is a very specific issue. My first time with a Modbus ProSoft. Customer setup doesn't make any sense but we're stuck with it...
Replies
8
Views
290
See code at the bottom. Hi all. For the story, this is my first complete project using a Rockwell PLC, I've been mostly working with Unity Pro...
Replies
7
Views
764
Hey! I have an .acd and .apa files from a facility (RSlogix5000) and I need to see their programming. I can't find a way to do so.I am happy...
Replies
5
Views
1,981
Had a disagreement today with a colleague. The situation we discussed was something like this: "Is it ok to use an instrument from a SIF to...
Replies
3
Views
1,687
A big heated discussion today at work about whether to use bits or integer numbers for error/alarm messages. I would be interested in reading...
Replies
11
Views
2,784
Back
Top Bottom