SLC 5/05 Serial Message Glitch?

[krakenfan69]

Hi guys.


I have a system we work on. It's a SLC 5/05 at a battery connected through CH0 to a serial radio to a L61 Control Logix processor at a satellite. The alarms at the Sat are all dropped into an array that was designed to mimic data table addresses from whenever it was upgraded. So they all land in an N40 Array. Those 100 words are read back to the plant into an N47 data table in the SLC that FTView 11 is using for alarming. (Alarm & Events Server) Every few months, every alarm form this site rings in. All of them at the same time. H2S fault and the 4 hour SD for the same bit ring at the same time. Analog values go wild too. When operations logs in the Sat is running fine. A quick ack/reset clears the alarms. I can't find anything writing to those registers at the plant so I am wondering if the message instruction itself may be the culprit. It gets something crossed and reports the entire array as a Zero value which triggers the alarms and callouts. The next read seems to pull valid data as they can always reset right away. Anyone seen anything like this before?


Thanks,

****** Fan #69

 

[alan_505]

Is there code that writes to the N47 when the message errors?

 

[krakenfan69]

Not as I have been able to see. It happened at 1:00 AM and they saw the site was still up so they ack/reset and it cleared. I wasn't informed until much later it was happening at all. So I haven't been able to be online when it was occurring.


****** Fna #69

 

[Ken Roach]

It is unlikely, but not impossible, that the MSG instruction or something in the radios or buffers between the satellite and the battery stations is at fault.

I would examine the ControlLogix user program first, keeping in mind that communications and user program execution are asynchronous in ControlLogix. If this system was upgraded from an SLC-500, the logic probably doesn't account for the differences between Logix and SLC operating system access by external communications.

When that MSG instruction arrives at the ControlLogix serial port and gets executed, it can do so in between rungs, in between instructions in a rung, or even in between elements of an ordinary element-by-element COP instruction.

Look for any logic that clears the INT[x] array that is mapped to Data File 40 prior to copying any updated fault flags or analog values.

When I need a table of data to remain consistent, I use an intermediate array in all of my logic, then one last Copy Synchronous (CPS) instruction. CPS locks the source and destination array tags from external access until the whole copy process is complete.

 

[drbitboy]

,... keeping in mind that communications and user program execution are asynchronous in ControlLogix.... .

+1e6

PLC operations are about time. The question is not why this is happening, it is rather when is this happening, and what is the PLC doing at that moment, to sub-millisecond resolution?

 

[James Mcquade]

do you know what the communications frequency is?
How close to you is a railroad?
the reason i ask is because i have been in plants that are near railroads. the train engineer would key up his radio and the plant radio signal would go nuts.
the plants have since then switched to alternate frequencies and communication methods.
the same can also be said for locations near airports. i had to change our wifi radio frequency to avoid the restricted airport frequency bands.
james

 

[Ken Roach]

If there is interference over the air or on the serial lines that causes corruption of the data payload, then the MSG instruction will complete with an /ER bit, not a /DN bit, and no data will be written to the destination data table.

Again: NO data will be written to the destination data table. Not all zeroes, not some random set of data, but rather no data at all.

It is technically possible for only data in the payload to be corrupted, but the mechanism responsible would either have to not be detected by CRC-16, or will have to corrupt the checksum itself in precisely the correct value. The probability of corruption that causes multiple values to "go crazy" without being detected by the checksum is close to theoretical zero, much less repeating every month or so.

 

[drbitboy]

It gets something crossed and reports the entire array as a Zero value which triggers the alarms and callouts.

Could you add code to the SLC 5/05 that checks for this "all zeros" condition, and when that condition occurs it

1. inhibits the transfer from N40 to N47, for at one or a few MSG cycles?
2. writes a timestamp of the event?

You could do the same thing in the L61, and have it write a timestamp each time it finds that condition?

At least that way you could compare the timestamps and see if the all-zeros are coming from the L61 via the serial link.

I think the railroad thing sounds cool.