PID Periodic Task & redundant system

Hi There
I am having a strange problem, i have put all my pids in to a periodic task and now i am having synchronization of the processor problem. After some time, the primary processor becomes out of sync with srcondary and the switch over takes place. At the same time, previously primary processor OK LED starts flashing. I switched the key positions from remote to program and then run and back to remote. By doing so couple of times, red Flashing LED goes away but green comes in. Once system becomes synchronized again, after running sometime, now primary processor becomes outof synch and does the same thing....This problem starts coming right after i put my PIDs into periodic task, before they were in continous task and was not having this problem at all. I am using Control logix L63 redundant processors, here is the program task paramters:

Continuous task Scan time = 170 ms Approximately
PID Periodic Task scan time = 20 ms Aprox.
PID Periodic task Period = 250 msec
PID Loop update time = 250 msec
1756 Input card RTS(real time sampling) timer = 100 msec (can not reduce it down to 100msec)
1756 Input card RPI time is 100 msec

Anybody have any idea what is doing it? I am on site so please any of your comments will be highly appreciated. Thanks in advance

Have you read the Knowledge base article

ID 25955 "Optimizing a ControlLogix Redundancy System, Programming Guidelines and Case Study."
Basically you do not want any continous task and use BOOL arrays Not Bool tags

Reason for changeover ? Never seen this one guess an hardware failure
Try Connecting to the Failed Processor after a switch over and checking the IO tree for faults (and processor warnings)
Quote from rockwell KB as reasons for switchover
1. Power loss
2. Module hardware failure
3. Module firmware assert
4. Module removal
5. Chassis backplane failure
6. Controller user program major fault
7. CNB going lonely (for example, disconnecting the tap out of a primary CNB)
* 8. ENBT going lonely (when V11 redundancy firmware is released)
9. Switchover command from user program or from SRM configuration dialog in RSLinx.
10. Inserting a module in the primary
11. Bringing up a secondary when the primary has a module with a hardware failure or firmware assert (will switch over after synchronization)

The 1757-SRM modules will tell you what reason they switched over; use RSLinx to bring up the Redundancy Module logs.

The flashing-red LED on the primary controller indicates a program fault. You need to go online with that controller BEFORE you clear the fault and read the fault code from the Controller Properties window.

It's possible that you have created a PID task that causes too many task overlaps. I have not read your other posts in depth but I think I recall that you were attempting a very unusual indirect-addressed PID implementation.

170mSecs for a continious task sounds awful high to me.

bkottaras said:

170mSecs for a continious task sounds awful high to me.

Click to expand...

Yes, but the program is also quite big and involves a lot of calculations.

Anyway Guys, today i went to site and i was told that during night PLC has switched over about 5 times. Last time cross load happend in the morning, since i was not there so i couldn't trap the fault. But since 8 in the morning till 7 pm night, it never occured, i was monitoring the PLC continuosuly, but never happend. So it is pretty strange for me. However i captured some data regarding the scan time of the main task and the periodic task and also a log file screen shot for 1756-SRM module, i am attaching my finding with this post (left hand side task is the periodic task and right hand one is continuous task), may be it is helpful for you guys to determine why it happend only the 6 or 7 times in total. For me i really don't understand all this. If you guys can share thoughts based on your experience and the data attached, it will highly be appreciated. Thanks in advance

Hi Guys I was told that last night it faulted again and crossload happened for about 4 times in space of 1 hour and 30 minutes. Please help me out whats going on in it......i appreciate your help in this regard

plcnovel said:

Hi Guys I was told that last night it faulted again and crossload happened for about 4 times in space of 1 hour and 30 minutes. Please help me out whats going on in it......i appreciate your help in this regard

Click to expand...

Ken Roach said:

The 1757-SRM modules will tell you what reason they switched over; use RSLinx to bring up the Redundancy Module logs.

The flashing-red LED on the primary controller indicates a program fault. You need to go online with that controller BEFORE you clear the fault and read the fault code from the Controller Properties window.

Click to expand...

Ken knows his AB stuff. Please follow the directions.

After you do so, we can help.

CroCop said:

Ken knows his AB stuff. Please follow the directions.

After you do so, we can help.

Click to expand...

I have followed his directions and thats why i attached my findings. I was not there so i couldn't trap the fault. But i will try today if it happens again. So basically , you want me to go online to faulted processor which becomes the secondry disqualified processor? is it right?

plcnovel said:

I have followed his directions and thats why i attached my findings. I was not there so i couldn't trap the fault. But i will try today if it happens again. So basically , you want me to go online to faulted processor which becomes the secondry disqualified processor? is it right?

Click to expand...

Ken Roach said:

The 1757-SRM modules will tell you what reason they switched over; use RSLinx to bring up the Redundancy Module logs.

The flashing-red LED on the primary controller indicates a program fault. You need to go online with that controller BEFORE you clear the fault and read the fault code from the Controller Properties window.

Click to expand...

Go online with the faulted controller. Before you clear the fault, go to the controller properties window & get the fault code. Read the fault code.

That's right; you have to go online with the faulted secondary processor and look at the fault code that is indicated in the Controller Properties.

The 1757-SRM log shows clearly that the primary PLC is undergoing a program fault which causes the pair to switch over, and that that fault is being cleared later (when you toggle between PROG and RUN, likely).

The SRM log can also be exported by selecting events and clicking that Export button. This format is more detailed than a screenshot.

Is it possible for you to contact Rockwell Automation Technical Support ? This isn't a hobby application, you need expert help.

Ken Roach said:

That's right; you have to go online with the faulted secondary processor and look at the fault code that is indicated in the Controller Properties.

The 1757-SRM log shows clearly that the primary PLC is undergoing a program fault which causes the pair to switch over, and that that fault is being cleared later (when you toggle between PROG and RUN, likely).

The SRM log can also be exported by selecting events and clicking that Export button. This format is more detailed than a screenshot.

Is it possible for you to contact Rockwell Automation Technical Support ? This isn't a hobby application, you need expert help.

Click to expand...

OK ken, I WAS ABLE to go online and trap the fault, The fault is

Type 4, Code 20, Array subscript too large........ The thing that i dont get is that, i never had this fault when my pids were in continuous task, but as soon as i put my pids in periodic task, this fault starts to come in......any thoughts on to it.....
Thanks for you guys help

I realize that putting the PID instructions into a periodic task appears to be correlated directly with the controller fault, but the actual proximate cause is an array element that has been addressed beyond the limit of the array.

The Controller Fault window should also show you which Task, Program, and Routine generated the fault.

I am going to go back and review your thread about indirect-addressing PID instructions. I recall intending to post "please don't do this".

Double, triple, quadruple check the limits on your index that points to the PID datatype elements.

Ken Roach said:

I realize that putting the PID instructions into a periodic task appears to be correlated directly with the controller fault, but the actual proximate cause is an array element that has been addressed beyond the limit of the array.

The Controller Fault window should also show you which Task, Program, and Routine generated the fault.

I am going to go back and review your thread about indirect-addressing PID instructions. I recall intending to post "please don't do this".

Click to expand...

The task is the continuous task, program is different then what my pids were in before i put them in a periodic task. and the routine is where i am doing my shutdown stuff. This relation of error i dont understand. All i did was to cut n paste my pid (written in structured text) alongwith associated tags in a periodic task.

Follow the evidence

Don't be too focused on the "all I did...." aspect of this troubleshooting. Maybe something in these shutdown routines is malfunctioning that did not previously, and it has nothing to do with the PID.

This is what the fault information in the secondary controller will tell you. The rest of the "Array subscript too large" error message tells you which task, which routine, even which rung is responsible for the controller fault. There, and only there, is the root of your problem.