Communication Issues- MSG Timeout

I've got a Controllogix system that is doing a MSG Read from an SLC500 PLC and over the past week we have had multiple issues with the MSG timing out(Extended Error code: 16#0000_0204). While the MSG is getting the errors I can ping both IP addresses and they both have under 2ms response times. None of the other MSG instructions in the program are getting errors, just the one PLC. Today when it happened, I noticed the panelview connected to the SLC500 had a message on it that said "16415: arp info overwritten for 0ad3e0e0 by f4:54:33:94:31:c8". The hex address "0ad3e0e0" converts to the IP of the SLC controller that is having the issues and the mac address is a different SLC processor on the other side of the plant. I've checked both processors and the IP configurations are still correct in each of them. I've also done a network scan while this issue was going on to make sure the IP address we are messaging matches the MAC address of the device just incase it was a duplicate IP issue, but all that looks good. The only thing I can see so far that is a little odd to me is the arp message on the panelview.

Any suggestions? I'm not really sure what my next step would be to troubleshoot the issue.

My first reaction was duplicate IP, but you covered that. I would check the counters for each IP address and look for errors, overruns, etc. Maybe a cable that is pinched somewhere and is starting to go bad perhaps?

I just got back from fighting this issue some more. I was at the point where I didn't know how else to troubleshoot so I just started changing parts. We replaced all cables in the cabinet. I also replaced the SLC500 processor. I have not changed anything on the Controllogix side. I can't see anything there being the problem as other MSG's are working fine. This one node is the only one stopping.

mjp123gp said:

I just got back from fighting this issue some more. I was at the point where I didn't know how else to troubleshoot so I just started changing parts. We replaced all cables in the cabinet. I also replaced the SLC500 processor. I have not changed anything on the Controllogix side. I can't see anything there being the problem as other MSG's are working fine. This one node is the only one stopping.

Click to expand...

How many messages do you have active at the same time ? Too many can fill up the SLC buffer, inevitably the failed message never gets into the queue, because an earlier one in the scan grabs a free slot as they appear.

I assume you are not "continuously" firing the MSG instructions off (i.e. XIO MSG.DN on the MSG rungs). You need to fire them off with a timer, or you can "chain" them, as each message completes, trigger the next and so on.

daba said:

How many messages do you have active at the same time ? Too many can fill up the SLC buffer, inevitably the failed message never gets into the queue, because an earlier one in the scan grabs a free slot as they appear.

I assume you are not "continuously" firing the MSG instructions off (i.e. XIO MSG.DN on the MSG rungs). You need to fire them off with a timer, or you can "chain" them, as each message completes, trigger the next and so on.

Click to expand...

In the CLX processor, there is only one MSG read and 2 MSG writes to the SLC. I'll have to dig a little deeper and see if any other controllers are messaging the the SLC. They are on a timer but I don't remember how it was setup. I know the 2 MSG writes are enabled at the same time but I think the read is on a separate timer. Just for troubleshooting purposes, I did disable all messages from the CLX and tried to manually fire the read so it was the only active message and it still failed.

It's just weird that I don't see any slow pings while the MSG is getting an error. I can even get online with the SLC processor while the errors are happening.

Way back I had a similar issue, and cured it by simply deleting the message instruction, its tag, and re-creating both from scratch. It worked for me.....

daba said:

Way back I had a similar issue, and cured it by simply deleting the message instruction, its tag, and re-creating both from scratch. It worked for me.....

Click to expand...

It happened about 5 times yesterday and 4 times Thursday but luckily they haven't called me in yet this weekend over the issue. The next time it happens though I'll try that and see if it helps. I'm out of ideas at this point.

Thanks

daba said:

Way back I had a similar issue, and cured it by simply deleting the message instruction, its tag, and re-creating both from scratch. It worked for me.....

Click to expand...

Do you know what the max number of messages I can read/write with that SLC5/05 is? I just set up some test logic to see when the messages are being triggered and it looks like all 4(2 read, 2 write) are being triggered at the same time quite often. The two read messages are reading a total of 76 elements while the two write are writing a total of 3 elements. Could this be overloading the buffer in the SLC processor?

mjp123gp said:

Do you know what the max number of messages I can read/write with that SLC5/05 is? I just set up some test logic to see when the messages are being triggered and it looks like all 4(2 read, 2 write) are being triggered at the same time quite often. The two read messages are reading a total of 76 elements while the two write are writing a total of 3 elements. Could this be overloading the buffer in the SLC processor?

Click to expand...

By default, an SLC5/05 will only process one message per sweep. To change this setting for the Ethernet port, reset bit S:2/15 and set bit S:33/7.

daba said:

Way back I had a similar issue, and cured it by simply deleting the message instruction, its tag, and re-creating both from scratch. It worked for me.....

Click to expand...

I have had to do that before as well. Logic and SLCs. Seems like if you edit it to many times it get corrupted.

Also if the SLC is a large program you can place a few SVC instructions about every 100 rungs. The SVC tells it to process messaging during the program scan. But be carful it can add a little to the program scan time.

Bit_Bucket_07 said:

By default, an SLC5/05 will only process one message per sweep. To change this setting for the Ethernet port, reset bit S:2/15 and set bit S:33/7.

Click to expand...

I was a little unclear with my comment about the number of read/writes. The SLC itself isn't doing the reading or writting. All the messages are handled on the Controllogix side. Would resetting and setting those bits still apply in this case?