Watchdog Faults on ControlLogix

majabat · Aug 1, 2013

Hello everybody,

We are having random Watchdog Faults (see attachments) on a ControlLogix controller. Each time it stops on a different place of the PLC task (different program and different routines), so we can't find what or where is the problem.

Any suggestions of what may be causing this problem and a way of finding it will be greatly appreciated.

Regards,

Mat

(8{)} ( .) · Aug 1, 2013

Well you've got to look at your code.

My recommendation, if this code is under development, is to place AFI instructions before each JSR in order to try and isolate the problematic one. You can monitor the scan time for each task by selecting the Task--> Properties --> Monitor.

Good Luck,

(8{)}

.)
(Yosi)

jkerekes · Aug 1, 2013

What is your task watchdog set at?

majabat · Aug 1, 2013

The code was developed several years ago, and it controls part of a production line, so it's not in development. But it was modified several times by different people and now nobody knows where the problem is.

Each time the Faut occurs it is reset and the production continues, but it's something that now we want to fix.

The watchdog is set to 2050 ms. It seems that its value has been increased several times to try to avoid this watchdog faults.

We are using a ControlLogix 1756-L62.

Thanks,
Mat

jkerekes · Aug 1, 2013

That watchdog time is getting up there especially if you are occasionally exceeding it. You need to go into that task and look at the individual programs and see where the time is being spent. There is also a task monitoring tool which will show you how much time the processor is spending where.

majabat · Aug 1, 2013

John, I didn't know about the existence of this tool (Logix 5000 Task Monitor).
I'll give it a try.

Thanks!

majabat · Aug 1, 2013

I attach some screen captures from Logix5000 Task Monitor. Maybe together we can extract some useful information.

We see that the CPU is working at more than 90%. ¿could this be a cause for Watchdog failure?

All the programs included in the same Task have a Max Scan that is several times (between 2 and 6) the Last Scan time.

There is not only one Program that is taking longer than expected, so it seems that there is no easy way to detect what program or routine is causing the problem.

Suggestions are welcomed!

Thanks,
Mat

mellis · Aug 1, 2013

The CPU will always be close to 100% when you have a continuous task. That's not the issue. It could be that you have a lot of conditional routines that don't always execute and when too many get triggered on the same scan, you go over the watchdog. Since we don't have any code to look at, it's guesswork.

majabat · Aug 1, 2013

Mellis, it's interesting what you say about the conditional routines. I'll check the code to see if that could be the case.
I'm sorry, but unfortunately I'm not allowed to publish the code.

Thanks,
Mat

daba · Aug 1, 2013

Let us see this for what it is...

Normal processing sees your task execution at typical values, but then all of a sudden your task watchdog gives up, and it's set for over 2 seconds !!

I find it hard to believe it possible, with correctly written code, to even get near to a 2-second scan time.

This is indicative of a fault in the program...

...perhaps a subroutine gets called in an infinite loop because the code doesn't handle the end conditions correctly.

...perhaps there's a JMP backwards in the code that has no termination conditions under certain circumstances.

These type of "code errors" can be tricky to find.

You say the watchdog timeout occurs when the scan is in different parts of the code, but analysis of the history of where it stops can help to find where the processor is getting "hung up".

AustralIan · Aug 1, 2013

Have you got motion task? Have you got a high priority interrupt task?

If it is happening on a random task/program each time, i suggest that one of your higher priority tasks is executing a lot, and not giving time for the lower priority ones.

Look for minor faults like task overlaps.

majabat · Aug 1, 2013

Daba, I totally agree with you, there’s something wrong.
Two seconds is a huge amount of time, but this was the temporary solution the maintenance people found worked for a while.
Now we want to find the root of this Watchdog fault to correct it.
I'll check if there are JMP instructions.

Thanks,
Mat

majabat · Aug 1, 2013

AustralIan said:
Have you got motion task? Have you got a high priority interrupt task?

If it is happening on a random task/program each time, i suggest that one of your higher priority tasks is executing a lot, and not giving time for the lower priority ones.

Look for minor faults like task overlaps.

AustralIan, thanks for your reply.
I've checked the code and it seems that there are no motion tasks.
The only task that it is running is the continuous task, so there are no priorities involved.

I've seen some JMPs backwards inside an Add-On instruction and some conditional routines that I'll check tomorrow.

Thanks everybody for the great feedback,
Mat

daba · Aug 1, 2013

majabat said:
AustralIan, thanks for your reply.
I've checked the code and it seems that there are no motion tasks.
The only task that it is running is the continuous task, so there are no priorities involved.

I've seen some JMPs backwards inside an Add-On instruction and some conditional routines that I'll check tomorrow.

Thanks everybody for the great feedback,
Mat

Be VERY suspicious of JMPs backwards, anywhere !!

Conditional routines are rarely the problem, unless that routine has like 30,000 rungs in it !!

I favour there's a transient condition that causes the JMP backwards to always execute, which will pop the watchdog.

AustralIan · Aug 1, 2013

daba said:
I favour there's a transient condition that causes the JMP backwards to always execute, which will pop the watchdog.

My vote is the JMP

Watchdog Faults on ControlLogix

majabat

Member

(8{)} ( .)

Lifetime Supporting Member

jkerekes

Member

majabat

Member

jkerekes

Member

majabat

Member

majabat

Member

mellis

Member

majabat

Member

daba

Lifetime Supporting Member

AustralIan

Member

majabat

Member

majabat

Member

daba

Lifetime Supporting Member

AustralIan

Member

Similar Topics