ControlLogix COP command affected by CPU Temperature

[maexio]

So this is a weird issue and its really a stretch to guess what is happening.

We have what is known as a lug chain, which is basically a continuous chain with attachments on it. It is driven by a K5700 servo drive with Scaling set 0-100 per attachment 'lug space'.
We have an event task that is being fired on forward only transition to 0 from the k5700.
In the event task we Cop ScannerLC[0] ScannerLCTmp[0] 100
then COP ScannerLCTmp[0] ScannerLC[1] 99
then FLL 0 ScannerLC[0] 1

There are no other mass cop instructions with a length greater than 1 in the plc, or FLLs, or Clrs and ScannerLC[0] is set to Read Only for external access.

What appears to be happening is that the system clears all values for ScannerLC[0] -> ScannerLC[26] inclusive, Randomly.

This is a startup so the system is just getting into spring / summer weather in Missouri, and the panel appears to be getting quite hot (100f output temp).

Last night we powered down the whole system and it started up this morning fine. It ran for about 3 hours this morning without issues, then the exact same issue started happening (no changes to code during this time), and we opened the panel to check the temperature (letting out some of the hot air) and it has stopped happening. Since then it has rained, so the overall temperature is a little lower, and the problem is not happening.

This sounds stupid to suggest temperature could cause a random change to the data, but we haven't seen any other correllations that make sense.

Processor is a 1756-L83E, and the ethernet network has been clean all day (no errors / faults, etc).

Last night we tried turning off produced / consumed tags (in case this was happening), and it still occurred.

The really odd part is that there are 4 lug chains that are cammed together and moving data, but the ScannerLC is the only array this is happening to. It is also the highest priority number of the event tasks, so not sure if that is a coincidence or not.

Let me know if more info is needed..

 

[spaderkung]

Copy with the CPS instead?

 

[maexio]

Copy with the CPS instead?

We did try that last night, and it still occurred exactly the same. Thanks for reminding me.

 

[kamenges]

It isn't odd to think that heat could create some odd, random events to occur in a system. But what you are describing is far from random. You are talking about a very specific data change to a very specific set of memory locations. The only thing that seems to be partially random is when the event occurs. Is there any way you can confirm that multiple unwanted triggers are not occurring?

Aside from that, what do you expect this code to do? From what I can see the code is shifting all array elements up by one location (0->1, 1->2, 2->3, etc) then putting 0 into array element 0. You don't say anything about anything else putting anything into array element 0. I would expect that after 100 triggers the whole array would be full of zeros.

Keith

 

[Ken Roach]

Fascinating !

The odds that this is un-detected "data corruption" at the chip or OS level due to high temperatures is close to zero.

We have an event task that is being fired on forward only transition to 0 from the k5700.

Is there any chance that when it's particularly hot, the encoder or the physical mechanism bounces back and forth in counts ?

I have a machine that startled me by executing its MAOC triggers multiple times in a few milliseconds, because the axis struck an obstacle and went across the threshold from less to greater several times.

How exactly do you detect the encoder rollover; a LES instruction that fires an EVENT ?

Have you set any logic traps to look for event overlaps ?

 

[maexio]

So there is a sizeable udt associated with this array, and things like incrementing lug id, packid, board data (its handling lumber), etc. So it is populating with data constantly, which is how the UDT clearing can be witnessed on the HMI setup we made to try and catch this.

 

[maexio]

Fascinating !

The odds that this is un-detected "data corruption" at the chip or OS level due to high temperatures is close to zero.



Is there any chance that when it's particularly hot, the encoder or the physical mechanism bounces back and forth in counts ?

I have a machine that startled me by executing its MAOC triggers multiple times in a few milliseconds, because the axis struck an obstacle and went across the threshold from less to greater several times.

How exactly do you detect the encoder rollover; a LES instruction that fires an EVENT ?

Have you set any logic traps to look for event overlaps ?

Its possible that the encoder rolls over, but it does about 60 lugs per minute or 60 'triggers' per minute, and the MAW command is only re-enabled by a lim between 50 and 75 centilugs (out of 100 per lug) preventing a rollover condition from happening, and it instantly removes 26 'rows' of the udt array.

The encoder rollover is in a Motion Arm Watch (MAW) that is energized in forward only based on the drive scaling and the encoder is on the output shaft of the motor that is all direct wired to the Kinetix 5700 drive.

I have put logic traps in the event task to see if one is triggering more than the others, but of course i haven't had the glitch again since lunch - and i put the traps in when it happened.

 

[maexio]

COP(ScannerLC[0],LC_ClipBoard[0],75)COP(ScannerScansLC[0],Lc_ClipBoardScans[0],75)COP(LC_ClipBoard[0],ScannerLC[1],74)COP(Lc_ClipBoardScans[0],ScannerScansLC[1],74)FLL(0,ScannerLC[0],1)FLL(0,ScannerScansLC[0],1)OTL(ScannerLC_NewLug);

Is the Data move Rung (my tag names were a little off in my original description)

And the Event task is enabled by :

LIM(50,EM512_LugChain.ActualPosition,75)MAW(EM512_LugChain,MainLugChain.MAW_DataShift,Forward,0);

 

[Ken Roach]

oh, and maybe disconnections of the Kinetix 5700 via CIP Motion ?

The fact that this is happening on a transition to zero suggests that maybe there's a connection failure (overheated Ethernet switch ?) and an initialization value that gives you a zero in the position feedback tag.

Have you trapped the previous value before it rolls over to zero ?

I've seen Rockwell devices give bad initial values on their first data update after initialization before. Maybe that's happening here.

 

[Ken Roach]

there is a sizeable udt associated with this array

I've seen things like this when I wrote/copied/cleared a tag whose elements are structured prior to that Array tag, with an over-size Length.

A Cross-Reference won't find that for you.

 

[maexio]

I recall reading about that, any way to find out which tags are structured adjacent to that array tag?
I thought it was Produced / Consumed though, so thats why we turned that off.

 

[Ken Roach]

ControlLogix structures the UDT in roughly the order you see the elements in the UDT definition.

If you've got a UDT made up of other UDTs the map can get a little complex.

Start by looking at the UDT that contains the Scanner_LC[100] array, and start checking the UDT elements above it for COP or FLL commands.

 

[maexio]

Thanks - Will do!

 

[maexio]

Well, the answer was a stupid problem. Event task interrupt with re-used clipboards. Not sure why i didn't look at that to start with. Focused on the wrong array.

So - i was copying from my Data_Array[0] to Clipboard_Array[0], and then Clipboard_Array[0] to Data_Array[1] in an event task.

Only problem was that other 'higher priority' event tasks would interrupt it in between cop instructions or during, and replace some of the data with other data :

Because i was stupidly using the same clipboard_array in multiple event tasks... something i normally don't do.

Stupid event task overlap issue.

Thanks for everyone's help!