How to handle PLC failures

[rootboy]

I respectfully disagree with this in general. I have seen PLCs run flawlessly for more than a decade...Of course the design is critical and spares are certainly necessary. We have one running a refrigeration system that has been bulletproof since around 1996. It's a PLC-5 though, not a Micrologix.

I'll second that. We still have a few GE Series Sixes that have been running non-stop since 1989. With these PLCs having a processor fault was extremely rare, and it was nearly impossible to fault it out through a logic error (a divide by zero would only result in "overflow" in the output register and it would keep on trucking).

The watchdog timer on the Sixes was hard-coded to 150ms, and I have yet to see one time out.

Compare this to the time that a software company (who will remain nameless) that was developing an upload/download application for the the Shreveport Truck & Bus plant shut down 80+ PLCs at same time by trying to access a non-existent file in the PLCs.

I guess that they didn't see the sense in testing it out on just one PLC first...

 

[rootboy]

What kind of failures usually happen if any? It will be in a hot climate (AZ/MX/CA border area with summer temps in the low 100s-120) on the south side of a building but they are not usually in production the summer. The enclosure will be cooled by an A/C unit. I'm also going to suggest a canopy over the panel.

I did a job in Mecca, CA (the most ineptly named city in the US), and it got up to 126 one day. Leaving your PLC outside is out of the question. We used Conex's with AC units mounted in them and foam insulation on the inside of the Conex. This kept things cool enough to where we had no problems (the control room doubled as the unofficial breakroom ). Spiders, scorpions, dust, mud, etc, was a different story.

It seems that designing a backup hard wired system would defeat the purpose of using a PLC in the first place. The idea of splitting half the controls to one PLC and the other half to another was also brought up.

Not at all, that is what Ford did (and probably does do to this day) with their controls. Hardwire a PLC Running relay off of a PLC output which the PLC keeps on when it is running. Should it drop out, have a set of N.C. contacts bring in the manual controls and sound an alarm.

What kind of UPS protection would you recommend? The customer states that the utility voltage is really high (+/- 500VAC is the summer normal) and can swing. We'd want something that would be an online type I believe.

Power supply fluctuations were very common in Mecca and never did cause any problems. That said, if we completely lost power we were dead in the water, but that wasn't a PLC issue.

I'm familar with the "keep it simple" mentality of the area as well. We're talking about a bunch of farmers who really don't see the need in anything fancy (and in general I would say that they are correct). You will have to show them the advantages of your system before they will buy into it.

That and the lack of trained service people in the area means that repairs could end up taking weeks if not months. They know this and it will be factored into the equation as well.

Are you anywhere near Mecca? Good people down there.

 

[blackandredwarrior]

I did a job in Mecca, CA (the most ineptly named city in the US), and it got up to 126 one day. Leaving your PLC outside is out of the question. We used Conex's with AC units mounted in them and foam insulation on the inside of the Conex. This kept things cool enough to where we had no problems (the control room doubled as the unofficial breakroom ). Spiders, scorpions, dust, mud, etc, was a different story.

I checked the records and I think about 124-126 was the max ever. I'd personally like to see it inside, but they really don't have anywhere to stick it and all of the existing wiring comes outside to this wall where all of the existing panels are.

Power supply fluctuations were very common in Mecca and never did cause any problems. That said, if we completely lost power we were dead in the water, but that wasn't a PLC issue.

I'm familar with the "keep it simple" mentality of the area as well. We're talking about a bunch of farmers who really don't see the need in anything fancy (and in general I would say that they are correct). You will have to show them the advantages of your system before they will buy into it.

That and the lack of trained service people in the area means that repairs could end up taking weeks if not months. They know this and it will be factored into the equation as well.

Are you anywhere near Mecca? Good people down there.

This is down near Yuma, AZ. I don't know how much the power fluctuates there, but I'm worried about the high constant voltage. The customer runs UPSes on all of their computers and they will throw alarms and get the office staff freaked out.

The kicker was today the boss is now wanting to try to sell them on an HMI. I think they would benefit from it being able to see what is causing an alarm to go off. This would also give them a lot more capacity for recording data and remote access.

To keep the budget down I'm looking at the Red Lion G303 and G306M. Has anyone mounted one completely inside of a panel? We're thinking on the inside of the door on some kind of standoff. I'm thinking of using a Hofmann back panel and having it cut out for the HMI.

 

[rootboy]

I checked the records and I think about 124-126 was the max ever. I'd personally like to see it inside, but they really don't have anywhere to stick it and all of the existing wiring comes outside to this wall where all of the existing panels are.

In that case, use a deep panel and stick an AC unit on the side. Keep any electronic screens out of direct sunlight, it's been known to ruin a LCD display (our low-tech solution to protect a display on a flowmeter was to stick a glove over the unit once we were done reading the display. Hey, it works! )

This is down near Yuma, AZ. I don't know how much the power fluctuates there, but I'm worried about the high constant voltage. The customer runs UPSes on all of their computers and they will throw alarms and get the office staff freaked out.

Better safe than sorry...

The kicker was today the boss is now wanting to try to sell them on an HMI. I think they would benefit from it being able to see what is causing an alarm to go off. This would also give them a lot more capacity for recording data and remote access.

Couple that with a smart meter (i.e. a Wattnode) and they will get usage, brownouts, etc as well.

 

[blackandredwarrior]

I thought that I would add a little update on this project. We're pretty much done. There's a few little things that need to be tidied up, like a 2000 amp main breaker that won't reset, but that really had nothing to do with our work (utility changed the transformer taps. Summertime voltage was +/-520 on a 480 service. Runs 480 in the winter under load.)

We split the controls up into 2 PLCs (MicroLogix 1400). One runs the evaporators (SOVs, starters and return air RTDs ) while the other picks up all the rest (condenser pumps and fans, ammonia recirculator pumps, a coolant recirc pump, plus head pressure, temps (outside and control panel) and humidity (not currently used). The PLCs communicate via ethernet with each other to pass some status information. And we used a Red Lion G306M. The PLCs, HMI and 4-20mA are all backed by a UPS.

We did find a DIN rail mounted 3 position switch to use as an HOA switch.

The A/C units are 6000 BTU IceQube units. There will be a "canopy" built to shade the panel as those units won't even keep up with solar gain during the summer.

I've attached some pictures of the panel after all was said and done. The enclosure interior is 72x72x18. It's a beast and probably could have been smaller. When didn't have a lot of time to really work on the layout before we had to order stuff. The customer wanted room to switch (3) 10HP fans to VFDs. As you can see, the Arizona heat is not kind to Panduit. It's sagging, especially near the top. It was out in the sun for 3-4 weeks before we could get it powered up.

This was a huge learning experience for me and I don't even know who all to thank around here as I read a lot of threads. I know I "borrowed" control for alternating the recirculator pumps (weekly toggle, with failover), analog scaling (4-20mA) and an hour timer snippet.

As I've seen a lot around here, never assume anything. Especially true in a 25-30 year old facility that's been expanded on several times and you hate to open a panel up to see what craziness you'll find inside it. It only reared it's head once but it revealed a big existing problem they didn't know about.

The biggest benefit is having maintenance on your side. They know how the plant runs and how it should run. Sometimes the guys in the office who OK things don't know for sure.

 

[JesperMP]

Wow, that is something I havent seen before, an operator panel INSIDE the control cabinet !
Is that to assist in locating electrical problems inside the control cabinet ?
Or to protect the operator panel against hamfisted operators ? So that the panel will work as an emergency backup ? (If that is the case, how to know that the panel is working if noone is using it for long periods of time ?).

Apologies if you have explained this in one of the previous posts. I havent read all posts in the thread.

 

[blackandredwarrior]

Well mainly three reason for it being on the inside:

1. Heat - This is on the south side of a building in AZ. In the summer it's about 100-120F degrees. Figured it would help out the HMI being completely inside.
2. Readability - You can't read it outside anyways.
3. Hamfisted Ops, etc. - It's inside the panel and only the maintenance supervisor knows how to use it.

It was originally there for sending email alarms and remote viewing. It turned into more of what an HMI is supposed to be. Nearly all of the settings (temps, pressure, timers) can be adjusted there. The built in web server leaves a lot to be desired. It doesn't seem to be working correctly now. I need to call Red Lion. Maybe it's a security setting (it kinda worked before I added security to the settings pages.)

If they want to move it (and I'd support that idea) there's a location inside the building about 10' away with an unused enclosure that would be perfect.

 

[JesperMP]

If the operator panel is critically important, then keep the one inside the panel, and install yet another panel in the other location. You then have a backup.

We very often have the problem with high am bient temperatures. We have a PC-based HMI (or several PC based HMIs) installed in the central control room. It is airconditioned, and if the airconditioning fails it will be fixed pronto (the people onsite normally prioritize this very much ), and... the airconditioned control room is normally made by the end-customer, so no extra cost for us which means we get the order since we can keep the price down.

 

[Lancie1]

I thought that I would add a little update on this project.

Very nice. The next guy doing the next modification will thank you for that extra room in the enclosure. Although looking at the result, the next guy may very likely be you.

 

[blackandredwarrior]

Thank you Lancie. I didn't do the actual panel build, though I have done some panels.