How to handle PLC failures

Join Date
Dec 2007
Location
Clovis, CA
Posts
41
Firstly I don't have a lot of experience with PLCs.

We have a customer that we are putting together a project for and one part of that project is updating the controls for part of their cooling system. We'll be controlling 2 condensers (total of 3 pumps and 3 fans), 6 evaporators (one contactor, 1 RTD and 4 SOVs per evaporator) and 2 recirculator pumps (one recirc pump will be primary and if it fails/trips the other pump will take over.)

We'll have a 4-20 pressure transducer on the condenser as well as an outside air temp (RTD). All of the emergency controls are outside of our scope and are existing. Though we will try to get an output from that system in case we need to do anything.

The customer is worried about using a PLC and having it fail and being stuck with about $5M worth of produce in the cooler. They've had an HMI die on them on a vacuum tube and it took months to get a replacement (from Australia with a price of about $15,000.)

We are looking to use an AB ML1400 1766-L32AWA with (1) 1762-IF4 (2) 1762-OW16 (2) 1762-IR4. We may also be adding the 4-20 output module if the customer approves VFDs on the condensers. This is subject to revision as we get further along. Our A-B supplier recommended to use Modbus RTU to control the VFDs but when looking up information to get ideas it seemed to be that is a Bad Idea^TM. If we go VFD I'm planning on hard wired start/stop from the PLC with 4-20 speed control.

How do you handle a situation like this? I suppose we could provide them with a complete backup PLC setup programmed and ready to go ($2000-3000 extra) if they need to swap it out. What kind of failures usually happen if any? It will be in a hot climate (AZ/MX/CA border area with summer temps in the low 100s-120) on the south side of a building but they are not usually in production the summer. The enclosure will be cooled by an A/C unit. I'm also going to suggest a canopy over the panel.

It seems that designing a backup hard wired system would defeat the purpose of using a PLC in the first place. The idea of splitting half the controls to one PLC and the other half to another was also brought up.

What kind of UPS protection would you recommend? The customer states that the utility voltage is really high (+/- 500VAC is the summer normal) and can swing. We'd want something that would be an online type I believe.

My boss has seen some output terminal boards that have small switches for H-O-A on them. I can't seem to find any. These are usually in custom panels (Frick Quantum LX I believe).

If more information is needed I'll be happy to provide what I know. I'm looking forward to hearing everyone's experiences and helpful hints. Hope everyone has/had a great holiday (if it applies.)
 
I suppose we could provide them with a complete backup PLC setup programmed and ready to go ($2000-3000 extra) if they need to swap it out.
Yes, that is a good practical idea. Also, provide a battery-backup UPS, mostly to make sure the PLC supply voltage does not go too high. Decide how long the average power outage lasts, and get a UPS/battery rated to handle the control power for that long. This will allow the PLC to ride through an outage without going offline and having to be restarted or reset. It could provide readouts and a temperature record on the the cooler so maybe could be used to decide if the product is still good after a power failure. Other than that, you might as well let the PLC die when the power dies.

What kind of failures usually happen if any?
PLC faults are fairly common, caused by failing PLC power supply, bad I/O module, or undiscovered errors (divide by 0) in the program. There are Status settings in the Mircologix that can handle a lot of the fault situations. Having a complete standby PLC unit will help with any other PLC-related problems.
 
Last edited:
My boss has seen some output terminal boards that have small switches for H-O-A on them. I can't seem to find any. These are usually in custom panels (Frick Quantum LX I believe).
How would these be used?
with 4-20 speed control.
You could use the built in 0 to 10v
you could use a 1/4 din temp controller as a backup alarm and if the site is not maned 24/7 just hook up an auto dialer to it. also email or text is possible for the alarms
I dont think the heat will be a problem if you put it in the shade! good luck
 
I did an alarm system for a pharma, who has beau coup bucks in inventory that has to be temperature maintained or it's garbage, it can't be sold. At $5M in inventory, no wonder your client has concerns.

If it were me, I'd be selling them a PLC based alarm system in addition to whatever control they want. Any and all mechanical devices AND controls can potentially fail. The key is knowing about failures to be able to take remedial action in time.

We had several meetings brainstorming failure modes. Obviously a pump can fail, its motor can fail or the power to the motor. They were concerned about 'human error' faults too, the "I can't believe someone did that" problems. I'm guessing that a produce warehouse is not employing the upper strata of the work force. They had stories where someone propped a freezer door open and then left the site. No refrigeration system in the world can cool an open system. As ridiculous as that sounds, it happened there.

In the system I was involved with the controls that deal with alarms are battery backed up with a large UPS.

All major alarms get emailed or texted to a cell phone and critical alarms dial out on a standard POTS dedicated phone line and the dialer only stops dialing when it gets acknowledgement. (auto dialers are primitive, I thought I was back in the 1976 era when I configured it; but we refused to rely on a PC based autodialer, too much risk of Windows hanging).

For what your system sounds like, I'd be monitoring for
- proof of flows, water (pump) and air (fan)
- temp in the storage area
- controls enclosure temperature (need to know if AC fails)
- loss of site AC power
- loss of a AC phase
- watchdog: the controller has an internal watchdog circuit that trips a dedicated relay if it detects a fault. That same relay is fail-safe, so it 'alarms' if power to the controller drops out.

I'd insist on proving flow with flow switches because I've seen systems that can be valved off where one would still get a pressure indication, but have zero flow available.

There's a critical alarm test circuit, a momentary pushbutton switch mounted behind the panel door. When it's held in for more than 5 seconds, it trips the critical alarm circuit, emails the list and starts dialing out (dialer has its own battery back-up). Their protocol calls for testing the alarm circuit and its outputs weekly.
 
It seems that designing a backup hard wired system would defeat the purpose of using a PLC in the first place.
Although I don't know the full scope of your project, I would hope it is still possible to manually operate the machine to at least salvage the product. You may not get all the 'bells and whistles' that the PLC provides, but minimal control is better than none.

Think of how the ECU in a car works (or is SUPPOSED to work). When a sensor is determined to be 'bad', it uses a default value for that sensor to at least allow the car to be driven. Sometimes called 'Limp Home' mode.

Although, if the ECU itself dies, you're out of luck... šŸ™ƒ

Another thought... Do you really need ALL the I/O on the main PLC to operate the basic functions? Critical I/O could be paralleled to a second, smaller PLC that would run the system, in a limited fashion, when necessary. Years ago, I did a material handling system with a backup PLC. This just ran a default program that cycled the system on a timed basis. Crude, but it would keep them running in the event that the main PLC failed. I don't know if they ever needed to use it though... :confused:

šŸ»

-Eric
 
I'd strongly recommend some sort of trend/alarm data logging on alarm points and using an autodialer with a state log. The trends and the autodialer log have both been pulled out to prove that the alarms were sensed and output accomplished, the failure to respond was on the human side. When large dollar inventories are lost, fingers will point in any and all directions looking for a scapegoat. The logs are vindication.
 
...
PLC faults are fairly common, caused by failing PLC power supply, bad I/O module, or undiscovered errors (divide by 0) in the program. There are Status settings in the Mircologix that can handle a lot of the fault situations. Having a complete standby PLC unit will help with any other PLC-related problems.

I respectfully disagree with this in general. I have seen PLCs run flawlessly for more than a decade...Of course the design is critical and spares are certainly necessary. We have one running a refrigeration system that has been bulletproof since around 1996. It's a PLC-5 though, not a Micrologix.

I agree with using a UPS for power backup and providing manual overrides. The value of the product may justify a more robust PLC than the Micrologix.

In any case, PLC control tends to be much more reliable than relay logic and hard wired pushbuttons.

Definitely provide spares, non-volatile backup (memory module in a Micrologix) and make sure the design of the controls takes every possible precaution to prevent failures.
 
I respectfully disagree with this in general. I have seen PLCs run flawlessly for more than a decade...
Yes, I have too, but in our potassium carbonate plant (due to the corrosion and contamination), PLCs in sealed filtered-air rooms rarely made it past 6 months. Pot carb is an insidious material, sneak-creeping up and inside the stranded copper wires going to the control room to contaminate the PLC. That is why we got big bucks for running a plant where no one else wanted the hassles. We had to rebuild the plant about every 12 months.

The reliability depends on the environment. Warrior did say it was hot, and I would guess that there is not much reliable maintenance.
 
Last edited:
We have an AB1200 mounted in a non ventilated cabinet out in the gulf. I think those are nearly indestructible. No failures whatsoever in the 5 years it has been installed.
 
While I have never had a micrologix 1400 fail they have not been around for that long. That being said I think they are good plcs. The built in data log feature could be helpful. I have also had click plcs in the field for just as long no failures!.It really does mater about the design of the system, proper fuses, surge protection ect... Dan makes really good points. For that much liability a few extra bucks for all the bells and whistles is well worth the money IMO.
 
Would a safety PLC be an option here? Pricey, but cost might be offset if you have to have two identical PLC's running in tandem.

I'm under the assumption that safety PLC's are redundant in some way.
-Dave
 
Last edited:
I would start by seeing what they do now.

Also, risk assessment with cause and effect analysis would also be a good idea. PLCs do not fail that often, but they do still fail, and I had a 1400 go on me due to a single drop of water getting past a faulty seal and landing on the wrong spot on the PLC.

Obviously, a PLC failure will not result in instant destruction of the entire stock, so what you should start with is acceptable down time. Then see what could be done if a failure occurs. I have also seen in the mining industry around here a lot of circuits built on relay logic override. If the PLC goes down, the device can generally be switches over to relay logic and started and stopped manually. You cant get good control, but if it is just someone starting and stopping a pump to protect 5m of stock, then it may be a good solution for the few hours that the PLC is down.
 
Only rule in electronics as far as length of service. It will last somewhere between 0 seconds and forever.

You hope for the best and plan for the worst.

Even though you have given alot of info you really do not have enough to go on. A risk assessment is needed.

Questions I would ask to assess risk:

If system failed and all cooling stop, how long could it stay in that state until material loss occurred. How long can you have a failure before recovery is impossible. IE at what temp if you brought all cooling online will it be a futile effort.

What local support will be given the system. Can the locals spell PLC.

Is a partial failure an issue. Can one part of the system fail and cause the same loss that a total (PLC crash).

How will this system be monitored. Is it in the corner and ignored until everything goes south or does it have scheduled checks by a operator. Is it a 24/7 operation or does it run and stop.

My blanket statement:

I trust a PLC a lot more than relay logic and I am old enough (thanks for reminding me PCkid ) to have done both. Looking at your spec I would rather poke myself in the eye with a blunt stick than do that with relay logic.

That said it comes down to design. Garbage in will always equal garbage out. Lots of ways to add redundancy and have the system limp along till repairs are made. Knowing the system you may find that everything does not have to be redundant just critical parts. As you said you already have a redundant pump for recirc.

To be honest I would be more scared of other components failing before the PLC. Well the CPU anyway. I/Os get fried almost as much as the parts they are attached to.
 
If your customer is concerned about failure of the the PLC and loss of product, create a redundant system. Should the first one fail, the second one would resume all operations. You could create notification regarding failure of one controller.

With a redundant system, everything is already wired and installed. No need for the customer to swap out a controller should it fail.

or

Utilizing the two controller scenario, you can do what you would do for a lead lag pump system and rotate the controllers so that you will know that "the back up" is functional.

Keep in mind this will cost the customer more, but it will provide them with assurance of a back up should the first system fail in any way.

You have stated that you are looking at the ML1400. If you are not totally committed to it, I would like to throw our hat in the ring for Entertron's Elite-2000:

http://www.eternity-sales.com/Entertron/elite2000.htm

If you have any questions, feel free to pm or email

Hope this helps
 

Similar Topics

I have a machine which has 1 keyence optical sensor (digital output) , 3 servo motors (Two rockwell MPL series motors and one rockwell TLY motor)...
Replies
4
Views
2,783
Hello all: I'am new to CIP protocol.Currently I want to read tag value from AB PLC not using RSLinx.When I put the tag name as address into the...
Replies
3
Views
4,083
Hi there Can it be said as a rule of a thumb that: No. of AI or AO which a particular PLC can handle = No. of PID's a PLC can handle
Replies
6
Views
3,003
Hi, I graduated from college about a year ago with both my Computer Programmer Analyst diploma and Computer Systems Networking Technican one...
Replies
16
Views
5,364
I am working on a project using AB PLCs and PanelView HMIs. There is a desire to allow control loop setpoints to be provided through the HMI or...
Replies
11
Views
3,646
Back
Top Bottom