machine faults

[OkiePC]

Guys, thanks so much for the detailed and informative responses!

Regarding these faults, under what circumstances would you actually stop the process and wait for a repair vs. simply mute the sensor? If a sensor signal fails that is required to go to the next step in a sequence, do you just program in a time delay overrride which can be activated via the hmi or just have the operator acknowledge the fault on the hmi then mute the sensor from that point on?

I'm not really sure i understand how you can write the program to mute the sensors in advance in case One consistently times out.

Thanks!

It depends on the systems tolerance to expected errors if you substitute expected sensor information.

I once wrote some sorting logic that mapped the contents of a conveyor with encoder feedback into a data file, and contolled the merge of 8 machines onto a belt at various drop points, and then being discharged among four output lines. In the office, testing my pointer based conveyor map logic, I wrote a pseudo encoder program with ramping matched to the drive command.

In the field, when the encoder failed, I had added a method to switch on the encoder emulator math and it worked flawlessly, even when stopped and started, (because my test code was modelled after the real machine to accurately test my design)

It ran in that mode for a week while they got new drive parts ready, and I went ahead and added logic to switchover within a few seconds of failure, and switch back when the encoder signal was restored automatically with a Fault: Encoder Loss making them acknowledge the fault and restart the line. When the encoder signal was restored, it became a Warning (but no stoppage) to the Panelmate when the encoder signal returned to normal.

These were fully assembled green tires being dropped on a moving belt and they needed to be close together but not touching or they would stick to each other.

If you ignore a sensor will someone get hurt?
If you ignore a sensor will your machine be damaged?
If you ignore a sensor will your product be compromised?

After answering no to all the above may qualify some sort of fallback or failover mode.

 

[MikeW]

Lots of goo info presented. "Evil Monkey" - I like that term and will use it in the future for randomize input testing.

Also consider verifying not just digital that the contactor has closed but is the motor actually drawing the expected power/ current - the contactor pulled in but a fuses is blown/ there is a broken coupling or belt. On pumps, use a flow switch and then debounce it once it's initially made so that if it falls to a low flow state for a second or two, it doesn't trip the process. Add a counter for these nuisance events and you may then be able to use this info for preventitive maintenance to see what's causing these events to occur more frequently.

As always, the risk assessment will point to what you can do and not do in a safe manner

 

[iant]

I might ask here, How many Inputs can we afford to use and how many hours of programming will be involved.
Some of these 'Ideas',as good as they are, are for the Ideal world with heaps of money and panel space available.
This is only possible If you are quoting against other companies to build this machine.
And the scope of work clearly states each fault condition to be displayed.
In a competetive market - a prospective customer may not choose the cheapest quote, But, definately not usually the dearest.

 

[Ken Roach]

Ian's right, of course; you can't afford to provide logic and testing and documentation and hardware for every possible thing that might go wrong with a machine.

This is why we do this sort of planning in tandem with the risk assessment; we're all thinking about probability and consequences.

This is my verification technician:

 

[JesperMP]

About limiting the number of sensors and alarms:
For each actuator, drive etc.. there should be one check that all is OK.
It is then possible to add many more sensors and alarms. The reason for adding all these alarms is to provide the operators with as precise an alarm description as possible. This is a nice-to-have rather than critically necessary. But it maybe makes the difference between a primitive and a modern machine.

An example: We have a machine where a cup has to be filled with material before the cup is moved to another position.
In the old design, the number of sensors had been reduced to the bare minimum. So there was simply one sensor detecting if the cup was succesfully filled. The alarm text could say that the cup failed to fill, but little about why it didnt fill. So the operators would have to spend much more time pin-pointing the actual error. Downtime equals lost profit, so it was a problem.
In the new design there are many more sensors, and many more alarms. So it is possible to say for example that the cup didnt get into position before filling, or the material that should fill is spent, etc.

I have seen some registering the time spent for all steps in a sequence, and then doing statistics on the time data. It is then possible to issue a warning if suddenly the time for one step jumps up. The operators can then maybe adjust something, or replace a part that is worn but not yet defective.
I am planning to implement this for some of our machines in the future.

It is a lot of work, but it actually has a value. We dont try to be the cheapest in our market. We make most of our sales on reputation.

 

[Andybr]

I might ask here, How many Inputs can we afford to use and how many hours of programming will be involved.
Some of these 'Ideas',as good as they are, are for the Ideal world with heaps of money and panel space available.
This is only possible If you are quoting against other companies to build this machine.
And the scope of work clearly states each fault condition to be displayed.
In a competetive market - a prospective customer may not choose the cheapest quote, But, definately not usually the dearest.

This is where standardisation comes in. we have a standard arrangement for every commonly used device. For a DOL started motor we use a Rockwell E3+ Overload on DeviceNet and all IO for the motor is on that device. We then have a standard UDT for a DOL Motor and a standard sub routine which we pass the UDT into and out of. All we need to do is create a tag, generate a required bit and call the sub routine when we add a motor into a programme. The sub routine takes care of all of the alarm generation and other indications. We also have standard objects in our SCADA applications so adding a motor is just a case of attaching the Tag name to an instance.
All of the above requires a lot of work to set up but once done it pays for itself in no time. It is also the reason we absolutely insist that suppliers adhere strictly to our standards, even for simple jobs.

 

[daba]

......so the decision has to come from above. It can be a bad thing if not managed properly.

Rheinhardt

"from above" is often WCS, they have different goals

 

[bimini3]

If you have sensors that are not needed to still run the machine then you may want to consider getting rid of the sensor and any other device that does not add real value to the system. (If it is not there it cannot have a failure). I've noticed some industries like lots of sensors that are not really needed and others that have only the sensors and devices needed to get the job done. Things like end of travel limit switches on high speed machines (that if the switch would ever trip there is no way of stopping the motion in to to prevent a crash). One place I worked we refered to these as "just to late" switches.

You did not state what PLC you are using. If it is one with Sequential Function Chart capabilities you may want to consider using that for you sequencer. Most have a Time limit you can set for each step of the sequence. If this time limit is exceeded an alarm bit is set. If something gets stuck or goes wrong that prevents you from getting to the next step then you can alert the operator or take other measures. It is a good idea to show what step you are in in some mannor on the HMI. If there are repetative problems people learn pretty quick where to look for problems when they know where in the sequence the machine is stuck.

Guys, thanks so much for the detailed and informative responses!

Regarding these faults, under what circumstances would you actually stop the process and wait for a repair vs. simply mute the sensor? If a sensor signal fails that is required to go to the next step in a sequence, do you just program in a time delay overrride which can be activated via the hmi or just have the operator acknowledge the fault on the hmi then mute the sensor from that point on?

I'm not really sure i understand how you can write the program to mute the sensors in advance in case One consistently times out.

Thanks!

 

[DamianInRochester]

As with everything else in life you have to strike a balance. It is easy to get carried away and take things far beyond practical value.



I am always asked questions along the lines of what is "normally" done in this or that and I nearly always find myself answering with what inevitably is an unsatisfying answer to whomever asked. That is simply that there is no normal. It depends. Every job must be analysed on a case by case basis. As soon as you start getting carried away with "standardizing" everything to death you end up with tons of nonsensical situations where you spend more time trying to work around the standards than just doing the job.



I find a great deal of machine faults often occur as a result of the actual sensor itself malfunctioning. To that point, the more sensors you add the more points of failure you add as well. Sure, you can make a rule that all double acting air cylinder must have a EXT and RET sensor. Well, what if it is just a simple press application where once every hour the operator puts a part in, presses a button, and lets it extend and retract? Depending on what is being done there may be zero value to actually confirming that the cylinder even cycled.

What if the stroke of the cylinder is only 1/4 inch? Most likely you can probably get away with just sensing the ON and OFF of one switch.



I see situations where a "part present" sensor is added to a station where that condition could be very easily deduced based on another condition.



That being said, I still think the programmer should detect and annunciate all reasonable means of failure or alarm. And by reasonable I mean if some milk-breathed needle head decides to see what would happen if he pressed all 10 pushbuttons simultaneously on the operator panel then if something nasty happens as a result I say GREAT, they got their just deserts!



As is mentioned many times in other threads as well, if the system is relying on the program to be "safe" then we also have another problem. Unless it is a "safety" controller then risk assessment part of it should have taken care of the hazard with either proper guarding or hardwired circuitry.



The amount of effort spent protecting a machine from itself should be proportional to the worth of the machine, cost of downtime, and whether or not it would truly cause damage. Consequence and likelihood should always be part of the decision making process.



And at least with sequences, it really isn't that difficult/time consuming to account for a fault for every step of the process. Depending on how the sequence code is written the "fault detection and determination" part can almost come along free for the ride. For me it is just a matter of typing in all the text per step.



And for my final rant ............... I also have to say that it would help if more effort was put into making sure the machine isn't faulting out in the first place. I see way too many poor designs out there these days. Cheap parts, undersized actuators and motors, shotty workmanship, etc. A lot this is coming out of the big Automation houses that should know better but have resorted to cutting corners and rushing things out the door before they are ready just so some manager can look good for coming in on time and under budget. Then it is up to the controls guy to try and mask over all this.



We have to remember that in almost all cases all the operators of the machine care about is that the machines just runs so they can do their job. I've seen many cynical remarks regarding this over the years, but the vast majority of operators I meet are the most happy when their machine is working right. If the machine is going down every 5 minutes because of a jam somewhere that has already happened 18 thousand times this past year it is difficult to chastise them for trying to defeat guarding or circumvent the safety features of the machine. I swear that well over 75% of the cases I personally know of where people have lost a finger or similar were because they were trying to clear a jam, and in almost all those cases it was a jam that occurred frequently to the extent that after the 38000th time they had to clear it they start getting too comfortable and confident and it bit them. Although it can't be condoned a large portion of the blame also rests with the machine builders, the maintenance staff, and the production managers that fail to resolve frequently occurring problems but instead turn a blind eye or put another band-aid on it. The SAFEST machines are the ones designed well enough that they don't go into alarm all the time.

 

[Jieve]

Guys, thanks so much for the insightful comments. Will definitely be downloading this thread for future reference.

 

[arturnok]

Hello Gentelmans,
Was reading all comments without breathing!
Thanks Jieve, it was very interesting topic, (as all in this forum) and all yours answers Gentelmans, with all examples, were very clear and easy to imaginate and to understand for a person who is trying to start the adventure with programming PLC's.
Forgive me to connect me to your topic, Jieve, without createing any new topic (tell me please if I should to start a new one).
I'd like to ask about creating alarms in S7 300/400.
Do you usually create UDT-alarms and UDT-notices with all alarms and notices inside (I'am thinking about small application - let's say about 50 alarms) and later trigger it in FBalarms where you describe all conditions? How do you usually do this?

 

[DamianInRochester]

I've noticed some industries like lots of sensors that are not really needed and others that have only the sensors and devices needed to get the job done. Things like end of travel limit switches on high speed machines (that if the switch would ever trip there is no way of stopping the motion in to to prevent a crash). One place I worked we refered to these as "just to late" switches.

Two things here I forgot to point out.

It drives me insane when I am asked to startup a servo system and find they designed the machine such that the mechanical actuator is about a fraction of an inch longer than the travel they needed to do the job. There seems to be this mis-conception that limit switches magically bring the mechanism to an instantaneous stop. Again, in most cases the real problem was not that EOT were not required but that the design did not accomodate the appropriate extra travel necessary to stop it in time. Usually when I start asking questions I find that simply nobody bothered to think about it. Often in these instances I recommend that they added rubber bumpers or some sort of soft stop to the ends to protect the carriage from damage in the event of a failure. Then they start making assinine responses impying that the problem is with the equipment or my ability to get it to stop quick enough.

Also, regardless of whether they are capable of preventing the machine from hurting itself, they are almost always still a good idea because

a) When in manual mode (which hopefully moves at a much slower pace than normal) it prevents the operator from jogging beyond the travel limits.

b) They still serve the purpose of confining the motion for the homing routine.

One thing I see often is the setup where you have an internal magnet triggering your switch and it is adjustable on a piece of extrusion or similar. Rarely do I see a switch/magnet combo that has enough range to remain on for the entire length of a high speed deceleration. Then you end up with the situation where the next time the machine is homed it is already beyond the switch and then just crashes. The machine itself of course has no idea what side of the switch it is on because .......... it's not home yet.

We usually abandon using the packaged switch kits in favor of external flags that remain ON for the entire range of overtravel. We ususally overlap the HOME switch range slightly with the end of travel such that you can only ever be to one side of it and make the homing more predicatable.

With absolute feedback coming down in price and becoming more ubiquitous I find myself paying the extra bucks to install it in almost all non-continuous applications. Absolute feedback can make what would normally be a very complex tooling return scenario a simple affair.

 

[OkiePC]

One thing I see often is the setup where you have an internal magnet triggering your switch and it is adjustable on a piece of extrusion or similar. Rarely do I see a switch/magnet combo that has enough range to remain on for the entire length of a high speed deceleration.

Magnetic reed switches suck. External inductive proximites and long flags are the way to go.

Then you end up with the situation where the next time the machine is homed it is already beyond the switch and then just crashes. The machine itself of course has no idea what side of the switch it is on because .......... it's not home yet.

Homing to a positive stop (torque mode, or torque home) is one sure way to get a precise and repeatable home. Always use a verification sensor as confirmation when doing this, lest you home and set zero due to a chunk of scrap in a belt drive or something.

Keep the drive logic power applied to avoid having to home the drive and keep it powered, using output contactors for safety. (Home only after a full power outage or encoder fault.)

Unless you would break too many other rules, always keep the drive and encoder signal alive during e-stop conditions and break the connection between it and the motor power. An early break contact from these safety power relays breaks the servo amp enable.

Having worked with both styles of drive e-stops, I find power cycling VFDs and servo amps to be much more destructive to them than unexpectedly switching their outputs off and on. Done right output contactors are far more reliable and offer many other advantages like faster restart time, ability to megger test entire motor circuit from the bottom of the output contactor without having to determinate a drive...

We usually abandon using the packaged switch kits in favor of external flags that remain ON for the entire range of overtravel. We ususally overlap the HOME switch range slightly with the end of travel such that you can only ever be to one side of it and make the homing more predicatable.

With absolute feedback coming down in price and becoming more ubiquitous I find myself paying the extra bucks to install it in almost all non-continuous applications. Absolute feedback can make what would normally be a very complex tooling return scenario a simple affair.

On lots of material converting machines used to make tires, we put resolver feedback on the tail shaft or undrive end of a belt or roller. We then used poor man's servo logic in the PLC-5 to control the +/-10vdc and enable signals to the motors.

The machine angle could change over about a 30 degree range, and having unique starting and stopping points for the cutter carriage for each angle optimized the cuts per minute.

We set the vfd for minimum accel and decel times and the PLC5 did a really nice job once fine tuned. I found with the cheap scalar drives, I need a minimum speed to avoid stalling, and had an adjustable decel distance and top speed. About 6 math instructions is all there was between the sequencer and this positioning logic.

On other machines with real motion control, using absolute feedback on the tail shaft as a virtual axis with its own position error tolerance for the fastest response to a failed drive train or changed home position.

I liked the AMCI products...talk about reliable sensors. We used to find them mangled and gouged and still working. Standing the the cables would eventually break a pin or something, but they were otherwise ball peen hammer proof.