How to handle PLC failures

If the stock is that critical I'd look at High-Limit temperature controllers, have a setup so that if the temperature approached critical then the controller itself bypasses the PLC and pulls in the fan/pump/evap contactors as well as sending out alert messages that something is wrong and someone better get there to look at it right away.

As far as PLC failure, with the cost of their inventory I'd look into what The PLC Kid (The PLC Adult now??) has posted, for the money involved here, a fully redundant system doesn't seem like that much of a stretch.
 
Would a safety PLC be an option here? Pricey, but cost might be offset if you have to have two identical PLC's running in tandem.

I'm under the assumption that safety PLC's are redundant in some way.
-Dave


Redundancy in the control system may be an option but you need a redundant PLC not a safety PLC. Safety PLCs are set up to fault and lock out the system if there is a fault in either Processor. A true redundant PLC can be set up to switch over to the good PLC if there is a fault on the other.

Typically processors are not the problem, but field wiring and devices are. You can double up on inputs & outputs and external devices. Set them up so that if one device is functional and the other is not the system will still work. Log when any device fails so that maintenance can fix the device before the other one fails. Communications networks are always a weak link. If using a communication system go with a redundant one.
 
if you are looking for a ML1400, using redundant plc will be way over the price of giving a spare programmed second ml1400...
In this kind of case trying to find an easy and cost effective way out with redundancy just create more parts that could fail than using just 1 plc because you won't look at everything you need to build a good one and save...

You must first start with good incomming protection, surge, line filter etc. then maybe have analog signal isolator for sensors going outside or in a different building etc...

Then you could say that an ML 1400 will be easier to find as it's locally availaible(not from australia)

You may also think about local manuel controls for emergency situation to keep going with few temporary operators instead of locking everythings out by the plc code
 
Price is dependant upon the cost of lost product.

Having a system designed with built in redundancy may cost more than having a spare PLC. However, the significant advantage is no down time. When one system fails, the system is still in operation on the second controller. The failed controller can be serviced while the second controller is operational, meaning no down time or loss of product as a result of down time.

It a customer is that concerned about loss of product, they will likely pay for a system that offers built in protection from failure.
 
you're right Stephen but if he has already a hard time convincing the customer to use a ML1400 instead of ordinary relay and timers, i cannot see how you could sell a 15 000$ upgrade to get a redundant system
 
Some customers are pennywise and pound foolish. They want something your opinion on what to do, then reject your recommendations due to cost. Then if your ML1400 solution fails in some way, they'll blame you for not providing a more reliable system. If downtime or the safety of their product is paramount, they need to put thing into place to guarentee it. A redundant system would be where I'd start.
 
How would these be used?

The H-O-A terminals I'm talking about have small switches on them that allow you to manually control the output like you would a normal HOA switch. I'm not sure how they would be used except for the pumps & fans. The solenoid valve groups would be very tricky, or at least a PITA.

You could use the built in 0 to 10v

I hadn't thought about that. I will investigate using the 0-10 rather than the 4-20mA output.
you could use a 1/4 din temp controller as a backup alarm and if the site is not maned 24/7 just hook up an auto dialer to it. also email or text is possible for the alarms
I dont think the heat will be a problem if you put it in the shade! good luck

We're going to check to see what we can do with the existing auto dialer. They have an existing AB SLC 5/0X (not sure which model) that they use for monitoring (not sure if they monitor temp, but I'm sure they monitor for ammonia) and it's hooked to an autodialer.

I did an alarm system for a pharma, who has beau coup bucks in inventory that has to be temperature maintained or it's garbage, it can't be sold. At $5M in inventory, no wonder your client has concerns.

If it were me, I'd be selling them a PLC based alarm system in addition to whatever control they want. Any and all mechanical devices AND controls can potentially fail. The key is knowing about failures to be able to take remedial action in time.

We had several meetings brainstorming failure modes. Obviously a pump can fail, its motor can fail or the power to the motor. They were concerned about 'human error' faults too, the "I can't believe someone did that" problems. I'm guessing that a produce warehouse is not employing the upper strata of the work force. They had stories where someone propped a freezer door open and then left the site. No refrigeration system in the world can cool an open system. As ridiculous as that sounds, it happened there.

The customer is concerned with the "human factor" as well. Currently they have to manually change over to high compressor head pressure so they get 150 psi to their ice house so that it will defrost correctly. This will often get left on for a few days needlessly wasting power running the compressors that hard. It is one of the issues they are looking at. I'm hoping to convince them that the savings they'll get by putting some controls in will override their reluctance on using a PLC.

In the system I was involved with the controls that deal with alarms are battery backed up with a large UPS.

All major alarms get emailed or texted to a cell phone and critical alarms dial out on a standard POTS dedicated phone line and the dialer only stops dialing when it gets acknowledgement. (auto dialers are primitive, I thought I was back in the 1976 era when I configured it; but we refused to rely on a PC based autodialer, too much risk of Windows hanging).

For what your system sounds like, I'd be monitoring for
- proof of flows, water (pump) and air (fan)
- temp in the storage area
- controls enclosure temperature (need to know if AC fails)
- loss of site AC power
- loss of a AC phase
- watchdog: the controller has an internal watchdog circuit that trips a dedicated relay if it detects a fault. That same relay is fail-safe, so it 'alarms' if power to the controller drops out.

I will see what is possible for proof of flow. I can see the condensers not being an issue, too much. I'm planning on monitoring the overloads to see if something kicks out. I figure if they are not tripped then they SHOULD be running, but then assuming anything isn't the best route.

I'm planning on a temp transducer in the cabinet to make sure things don't get too toasty in there. I know many of the panels close by became nearly too hot to even touch in the middle of the afternoon without gloves on.

They have some existing power monitoring on the switchboards. I'll see if that's still active.

I'd insist on proving flow with flow switches because I've seen systems that can be valved off where one would still get a pressure indication, but have zero flow available.

Understood. The problem with assuming there's flow just based on an overload not being tripped.

There's a critical alarm test circuit, a momentary pushbutton switch mounted behind the panel door. When it's held in for more than 5 seconds, it trips the critical alarm circuit, emails the list and starts dialing out (dialer has its own battery back-up). Their protocol calls for testing the alarm circuit and its outputs weekly.

I will discuss this with them.

Although I don't know the full scope of your project, I would hope it is still possible to manually operate the machine to at least salvage the product. You may not get all the 'bells and whistles' that the PLC provides, but minimal control is better than none.

This is what the HOA terminal blocks I was talking about would do. At least for some of the things. Pumps and fans.[/QUOTE]

Think of how the ECU in a car works (or is SUPPOSED to work). When a sensor is determined to be 'bad', it uses a default value for that sensor to at least allow the car to be driven. Sometimes called 'Limp Home' mode.

Although, if the ECU itself dies, you're out of luck... 🙃

Another thought... Do you really need ALL the I/O on the main PLC to operate the basic functions? Critical I/O could be paralleled to a second, smaller PLC that would run the system, in a limited fashion, when necessary. Years ago, I did a material handling system with a backup PLC. This just ran a default program that cycled the system on a timed basis. Crude, but it would keep them running in the event that the main PLC failed. I don't know if they ever needed to use it though... :confused:

🍻

-Eric[/QUOTE]

The issue I think that would come up with paralleling the outputs going to the SOVs for the refrigeration. Maybe running these through a 4 pole relay would solve it. When the secondary PLC takes over it would activate the relay so that it's outputs went to the SOVs. Having the switch over happen during defrost might cause some issues. Liquid ammonia into the compressor would be a Bad Thing^TM.

I'd strongly recommend some sort of trend/alarm data logging on alarm points and using an autodialer with a state log. The trends and the autodialer log have both been pulled out to prove that the alarms were sensed and output accomplished, the failure to respond was on the human side. When large dollar inventories are lost, fingers will point in any and all directions looking for a scapegoat. The logs are vindication.

The plus side is they would really like to have data logging and that helps the PLC proposal. I need to discuss how often and what they want to log. They'd like to be able to keey a whole season (six months) in one place. There current system requires them to manually download data every couple days. I think this is the ammonia and possibly room temp data.

The reliability depends on the environment. Warrior did say it was hot, and I would guess that there is not much reliable maintenance.

I know they hired a new maintenance guy last year. He at least is honest and says he isn't an electrician and doesn't like to mess with it. Considering some of the plants I've been in (including this one) that's a good thing. The boss told the customer last year "Your code violations and code violations." At least they understood they had a problem and want to address it.

Luckily the only physical contaminants there are dust (LOTS of dust), a tiny bit of water if it condenses and makes it's way there and of course, heat. LOT's of heat.

What local support will be given the system. Can the locals spell PLC.

They have an existing PLC so there is someone at least locally to deal with it if the need does arise.

How will this system be monitored. Is it in the corner and ignored until everything goes south or does it have scheduled checks by a operator. Is it a 24/7 operation or does it run and stop.

I have a feeling they want the least amount of interaction as possible.

I trust a PLC a lot more than relay logic and I am old enough (thanks for reminding me PCkid ) to have done both. Looking at your spec I would rather poke myself in the eye with a blunt stick than do that with relay logic.

The boss doesn't even want to attempt this with relays, defrost time clocks and what not.[/QUOTE]

Some customers are pennywise and pound foolish. They want something your opinion on what to do, then reject your recommendations due to cost. Then if your ML1400 solution fails in some way, they'll blame you for not providing a more reliable system. If downtime or the safety of their product is paramount, they need to put thing into place to guarentee it. A redundant system would be where I'd start.

I understand that. West coast farmers are the cheapest collective group.

Thanks for everyone's responses. I'll try to keep you guys updated.
 
I have not had an Omron PLC fail in years. That being said, I am having to put in more Omron CS1D PLCs which have redundant processors and power supplies but single I/O. They also have complete dual I/O systems available but you then have to duplicate I/O with a whole pile of relays and introduce a whole pile more single points of failure.
I have provided systems in the past with an output from one PLC inputting to the other and that input turns off all the outputs in the second PLC. First PLC fails second PLC has been scanning all the time with the outputs turned off. This system can be at best called warm transfer as there is a small break in control.
Once again a whole pile of relays and analogue splitters are required adding more single points of failure.
I do generally use DC power supply type PLCs for critical systems with 2 DC power supply systems combined through bridge rectifiers and then fed to the PLC through a 24VDC/24VDC switch mode power supply. If one of the power supplies fail the other takes over. The 24VDC/24VDC switch mode then keeps the voltage to the PLC power supply and I/O at a constant voltage reducing strain on the PLC power supply. the negatives are NEVER grounded but allowed to float and that way there is no chance of a 24VDC positive shorting to ground and turning the switch mode power supply off -0 works for me.
 
Luckily the only physical contaminants there are dust (LOTS of dust), a tiny bit of water if it condenses and makes it's way there and of course, heat. LOT's of heat.
You mentioned earlier that the cooling system would only be used in the produce season. Will the PLC and other controls be in an air conditioned humidity-controlled space year-round, or only during the produce season? Are they going to kill power to the plant when it is shut down?

Long periods with no humidity control, dusty air, and no power can be rough on electronic equipment. Damp dust can easily turn into a short-circuit path across a printed-circuit board the next time that power is applied. Maybe in Arizona, humidity is not as much a problem as it is here in Alabama, and your dust is always dry. Our current-flow-attracted dust on PC boards is NOT always dry, going from dry in the summer and fall to wet in the winter and spring.

Last week I took my 2006 Dell desktop computer out to the garage, opened the cover, and did the annual dust blow-out with an air hose. I had a big pile of dust bunnies when I finished. It is amazing how much dust is attracted to the circuits, even in a filtered-air environment. All it takes is a little moist air to change that collected dust into short-circuits.
 
Last edited:
You mentioned earlier that the cooling system would only be used in the produce season. Will the PLC and other controls be in an air conditioned humidity-controlled space year-round, or only during the produce season? Are they going to kill power to the plant when it is shut down?

Long periods with no humidity control, dusty air, and no power can be rough on electronic equipment. Damp dust can easily turn into a short-circuit path across a printed-circuit board the next time that power is applied. Maybe in Arizona, humidity is not as much a problem as it is here in Alabama, and your dust is always dry. Our current-flow-attracted dust on PC boards is NOT always dry, going from dry in the summer and fall to wet in the winter and spring.

Last week I took my 2006 Dell desktop computer out to the garage, opened the cover, and did the annual dust blow-out with an air hose. I had a big pile of dust bunnies when I finished. It is amazing how much dust is attracted to the circuits, even in a filtered-air environment. All it takes is a little moist air to change that collected dust into short-circuits.

The control panel will be outside with it's own side mounted A/C unit. We will probably request them to leave it running 365 days a year. They may need it up to run maintenance on the equipment. They do turn some things off, we'll see.

Humidity there can run from bone dry to 100%. The fluctuations can happen in just a few days, even over night. We were there last August. It was just plain hot the first two days. We walked out of the hotel on day three and it was like getting hit with a board walking out the doors. Spent an hour doing some last minute research and jumped in the plane out of there. (Next stop was Camarillo/Oxnard, CA... a welcome change.)

Sand in everything is a huge issue. We cleaned, well vacuumed out the main switchboard when we were there.

The first rain of the season (September to November) here (Fresno, CA) the dirt on the power lines turns into mud and things start blowing up.
 
The H-O-A terminals I'm talking about have small switches on them that allow you to manually control the output like you would a normal HOA switch. I'm not sure how they would be used except for the pumps & fans. The solenoid valve groups would be very tricky, or at least a PITA.
To create a similar function, you could use interface relays that have a "lockable test button". Basically just a manual test button that can also be used to hold the contacts closed indefinitely. They have their uses, but I'm not a fan of these because untrained personnel can really screw stuff up. Plus, you have to open the panel to access them. Many manufacturers offer relays like this. Here's one example with a decent explanation:

http://www.ia.omron.com/data_pdf/data_sheet/mk-s_ds_csm1382.pdf

🍻

-Eric
 
Our refrigeration systems all have small HOA 3pdt toggle switches wired to bypass their respective PLC outputs. They are cheap and compact. They are also mounted inside the enclosures to help prevent unauthorized operation. Eric's lockable relays look pretty cool too, but the toggle switch levers make it plainly visible when one or more of them is in "Off" or "Hand" mode.
 
Last edited:
Our refrigeration systems all have small HOA 3pdt toggle switches wired to bypass their respective PLC outputs. They are cheap and compact. They are also mounted inside the enclosures to help prevent unauthorized operation. Eric's lockable relays look pretty cool too, but the toggle switch levers make it plainly visible when one or more of them is in "Off" or "Hand" mode.

This is along the lines of what I was thinking, but with the units had a switch like that mounted to a DIN rail card. Basically they are like the AB 1492 Programmable Controller Wiring System with this toggle bypass switch.

2+ hours later...

I've found some DIN rail mounted switches.
 
When I suggested monitoring flows I was thinking glycol chilled water cooling loops, not refrigerant lines. I would not monitor flow in refrigerant lines.
 

Similar Topics

I have a machine which has 1 keyence optical sensor (digital output) , 3 servo motors (Two rockwell MPL series motors and one rockwell TLY motor)...
Replies
4
Views
2,844
Hello all: I'am new to CIP protocol.Currently I want to read tag value from AB PLC not using RSLinx.When I put the tag name as address into the...
Replies
3
Views
4,091
Hi there Can it be said as a rule of a thumb that: No. of AI or AO which a particular PLC can handle = No. of PID's a PLC can handle
Replies
6
Views
3,029
Hi, I graduated from college about a year ago with both my Computer Programmer Analyst diploma and Computer Systems Networking Technican one...
Replies
16
Views
5,374
I am working on a project using AB PLCs and PanelView HMIs. There is a desire to allow control loop setpoints to be provided through the HMI or...
Replies
11
Views
3,687
Back
Top Bottom