PID loop frustration

This in regards to a previous post a while ago on PID loop runaway. We have a machine that has two different ovens on it. Each oven has a thermocouple in it that goes back to an HE693THM409 card in a GE 90-30 rack with a cpu 350 in it. It is using a basic PID loop for each oven. We have had the machine/program in place for about 8 years now and never had any issues with the temp loops until the past couple months. The program for the machine hasn't been touched in a couple years. The problem started a month or so ago, when looking at the oven trends on the HMI for both it seemed that the one oven spiked at one point and the feedback actually locked up at a value below setpoint therefore the CV increased steadily until the oven safety limit tripped out. At the same point in time the other oven did the exact opposit( feedback locked and CV went down). At that time we checked all thermocouples and installed a new thermocouple card in the rack. The machine now has been running flawlessly 24/5 for the passed month and agian the same thing happened last night. The one oven, the feedback trend shows the value locks in at a value and the CV increases steadily for about the next 10 minutes and then corrects itself while the other oven did the exact opposite during the same time period and duration and then corrected itself. Other than this issue the machine is experiencing no other problems. Not sure at this point why the system is doing this. Please help! Thanks

How about the wiring between the TC and the module? Look for grounds or shorts in any connectors or extension leads.
Do they go through terminals? Have they been wired to different terminals?

The critical thing in your post is "feedback locked up".

That should be detectable in PLC code.

It reminds me of a problem we had years ago with a rubber extruder which basically did the same thing...but the result was out of control pressure which kept blowing out $800 rupture discs... Its SLC had analog inputs from a remote rack with M0/M1 file manipulation handling the block transfers. We added logic to detect when there was an error with the communications to shut it down, and also added logic that would limit how long the pressure value could stay at the same value withuot triggering a shutdown. In our case, the pressure feedback should never have been the same exact value for more than one second. Your application could probably benefit from some similar changes.

Paul

Each thermocouple gets plugged into a thermoucouple female receptacle on the side of a juction box ( two different jboxes). From this receptacle a 20 ga. shielded thermocouple cable runs back to the thermocouple card without splices and the shield is terminated on the card only and floating at the other end.

I guess my question then is if we put code in to detect it and shutdown the process then it still doesn't solve the issue as to how to correct it. We have separate high limit controllers in place with a separate thermocouple, these are completely isolated form the plc and will shutdown the ovens if a problem occurs. Also, we have 6 other machine as well with similar plc programs and we have never had this happen to any of them.

First off, unless the temperatures coming into the plc when you had the fault were the same in both fault cases I don't think there is any particular significance in the fact that one oven heated and one oven cooled. That was just dumb luck with where the temperature was when the feedbacks stopped reporting.

So the plc isn't getting data reads back from the analog input card. This is something relatively new and you already replaced the analog input card. Are there any other used input points on the card other than the temperature inputs? If so, do those freeze too? I wouldn't necessarily dismiss noise real quickly. You may be the lucky discoverer of a design deficiency in the alanog input card. Are any other modules affected? This could point to whether it is a module issue or a plc issue. It could also be the rack. How do you get it to start reading again? Does it start back up on it's own or do you have to cycle power?

Keith

Are there any entries in the PLC's I/O fault table? Specifically, a "Loss of module" entry at the same time the thermocouple signal disappears. I'm wondering if the thermocouple module is losing backplane communications with the CPU and the %AI registers are holding their last reported values.

At that time we checked all thermocouples and installed a new thermocouple card in the rack.

Click to expand...

I don't think there is any particular significance in the fact that one oven heated and one oven cooled. That was just dumb luck...

Click to expand...

Or just maybe during that time when the thermocouple card was replaced, someone (call him Murphy) accidently switched the leads, so that the Oven 1 thermocouple is now going to the Oven 2 temperature control PID, and Oven 2 thermocouple is going to Oven 1 PID....

Edward A. Murphy's Law: "If there are two or more ways of doing something, and one of those ways will lead to a catastrophe, then someone will do it."

If the ovens are more or less identical, and are turned on at the same time, then reversed thermocouples may control fairly well, until there is some upset condition, like maybe the door on one oven is opened, or is leaking a little. Then the reversed controls would cause some really strange stuff. Suppose Oven 1 PID reaches Oven 2's setpoint. So it calls for heat off, while Oven 2 PID is still heating Oven 1. Then you get one heating, one cooling, and no control, so Oven 1 goes into runaway heat, while Oven 2 gets colder and colder...

Funny when you think about it! These things actually happen. See the story of the real Murphy's Law at Edwards Air Force base in California (not Wright Patterson in Ohio as incorrectly reported by Wikipedia) in the 1950's.

http://www.howstuffworks.com/framed...ves/paperair/volume9/v9i5/murphy/murphy0.htmlhttp://people.howstuffworks.com/murphys-law1.htm

1) Could it be that the thermocouple card has a failure mode that is "hold last", rather than "Fail High" or "Fail Low", that would account for the fixed value?

2) Modbus comm failures frequently manifest themselves as "maintained last value", which makes me wonder whether Mr. Bailey's assessment of backplane comm failure is a possibility. It would be worth checking the fault table or error log.

Dan

Thanks for the help guys and it gives me some things to start looking into.I can tell you the following to answer some of the quetsions above basically the first time this haapened we had to actually reboot the plc to get it working again, this past time it corrected automatically on it's own after about 10 minutes. There are no other inputs connected to this card and the machine is experiencing no other problems. When we changed the card we removed no wiring as the card has removeable terminal blocks. We also have verified that the thermocouple wiring is correct. I'll look into the looking at the plc fault table as Steve suggested, I haven't done that yet. If we do find that the comm. is lost between the module and the plc and it is holding the last register values, then what would correct that. Would it be a bad backplane or CPU? Thanks

One other thought:

The effect on your oven that you describe fits EXACTLY for what happens when a PID loop is switched from auto mode to manual mode.

Whatever the PID output is at that exact instant is held manual mode is enabled will produce much the same effect: "gradually heats up" or "gradually cools off", depending on what the PID output is.

You originally said, the "feedback actually locked up at a value below setpoint therefore the CV increased steadily until the oven safety limit tripped out" so can we assume that you have some means of knowing (like historian/recorder) that the signal from the thermocouple is the culprit that flatlines, and that there's no chance that someone, somehow, did a Murphy's Law and put the two loops into manual mode where the PID output flatlined, creating the problem?

Dan

If we do find that the comm. is lost between the module and the plc and it is holding the last register values, then what would correct that.

Click to expand...

The first thing would probably be to change the backplane, but before you do I would suggest opening a case with both GE Fanuc and Horner support. I have seen one other instance where a Horner module didn't play well on a 90-30 backplane. In that case the resolution was to use an older revision level backplane. Some component changes that GE Fanuc had made to the backplane caused problems for Horner. It was not a thermocouple card though.

Another possibility would be to use signal conditioners to convert the thermocouple signals to 4 - 20 mA and replace the Horner thermocouple module with a GE Fanuc analog input module.

Dan,
In our Cimplicity HMI we trend both the CV and PV of each oven. In each instant you see the PV of each oven actually flatline. The first case we had to reboot the PLC to correct. This last time it flattens out for 10 min. and then corrects itself with no intervention.

Steve - Do you have a GE phone # to contact to open a case and also do you need to have a global care contract to do this. We currently don't. IF not I'll research.

Thanks for all the help!!!

GE Fanuc's tech support number is 800 GEFANUC (433-2682).
Horner's is 877 665-5666

GE Fanuc will ask if you have a Global Care number, but I don't think they'll refuse to help if you don't have one. They may just be slower to get back to you. I haven't had to use them for several years so I can't comment on the quality of their support these days. If my past experience is any guide, you may get some finger pointing between GE Fanuc and Horner.