Guardlogix & point i/o safety system comms

Hi Gents,


I am battling for a few days now an issue at one of my customer's site with a Guardlogix system. The System has the CPU's and three 1756-ENBT/A cards f/w 6.6 each.
One of the 1756-ENBT's is plugged into port 1 of a STRATIX 5700 managed switch. The system has three point I/O racks with 1734-AENT/A adapters fw/3.12 plugged in into ports 2,3 & 4 respectively. The switch has all its ports set to smart-ports and set as "AUTOMATION DEVICE".
Occasionally the system crashes on a comms fault. When I say comms I mean faults 516 & 515 come up either on the AENT's or in one or more of the cards attached to the AENT's. Cards like 1734-IB8S/A or 1734-OB8S/A.The fault moves around as in some day AENT adapter 1 will fault other days AENT adapter 2 will fault. The system recovers in about 11 seconds and everything resumes but it trips the whole system in the process. This is bad as the system is linked to another system which is running a coal mine.
The cabling is good as we changed the CAT5 to new CAT6 cables routing the cables basically on the ground temporarily until we find the problem. The longest CAT6 run is probably 12m. Everything is in the same room physically.
RPI rates increased and the timeout multipliers increased for the cards.
We have monitored traffic between port 4 and port 1 and what I captured there using wireshark is absolutely puzzling.

When the system is working ok the protocol used between the 1756-ENBT & 1734-AENT is ENIP. When the system trips on a comms fault wireshark shows that the comms protocol used is CIP SAFETY ??? for a few transactions then it becomes ENIP again for a few packets then CIP SAFETY again and the whole thing goes like this?

As far as I understood the kb article the protocol should be 573950 it should always be CIP SAFETY ? as the task itself is a safety task ?

Rockwell support so far failed to explain why I have this change in protocol from ENIP to CIP SAFETY and whether is the actual cause of the random faults or the faults come first and the system switches to CIP SAFETY (perhaps to ensure that the safety IO comms is ok?)

Could you please help with some ideas how could we troubleshoot this ?and most importantly does someone know why I have this ENIP to CIP SAFETY protocol change?

Where is this CIP SAFETY/EIP setting? Is this in the stratix?

Shot in the dark: We had a lot of random issues with a recent safety PLC network with a Stratix when all of the devices were set to Auto Negotiate. We changed them all to fixed 100MB Full Duplex.

Be aware there are some issues with certain 1734-AENT's and I/O connection faults see KB588404.

I don't know whether I would completely trust Wireshark's dissectors for differentiating ENIP and CIP safety. I would assume during normal operations there would be both protocols, but when i have a look here in the lab I see wireshark calls all the traffic ENIP. It could be that when an error occurs Wireshark then identifies those packets as CIP safety.

You're using autonegotiation for everything right? Check the AENT ethernet web page diagnostics to see if anything is amiss.

The CIP SAFETY comes up on the Wireshark log.Everything is set to Autonegotiate.I suppose we can try to set everything to fixed.

Timbert said:

Be aware there are some issues with certain 1734-AENT's and I/O connection faults see KB588404.

I don't know whether I would completely trust Wireshark's dissectors for differentiating ENIP and CIP safety. I would assume during normal operations there would be both protocols, but when i have a look here in the lab I see wireshark calls all the traffic ENIP. It could be that when an error occurs Wireshark then identifies those packets as CIP safety.

You're using autonegotiation for everything right? Check the AENT ethernet web page diagnostics to see if anything is amiss.

Click to expand...

Yes I knew about the AENT series B issues. Unfortunately all AENT's are series A in this system and most of them if not all have been already replaced.Regarding Wireshark all I can say is that all day long Wireshark shows the communication protocol as being ENIP. With no exception. Then when the fault occurs the protocol chnages to CIP SAFETY. In the lab if you have access to one could you see CIP SAFETY if you let's say deliberately create a fault? Like power down the AENT ? Or pull one I/O card out while running ? If you could see CIP SAFETY after you create the fault it means that the protocol is chnaging automatically to CIP SAFETY as some sort of a safety measure but it's the effect...Not the cause of the faults in my case.

A quick test. I pulled out an OB8S module, the POINT bus status light went flashing red as expected, Wireshark still reports ENIP as only traffic.

Pulling the Ethernet cable from between the controller and AENT results in no traffic at all.

When I plug the cable in I see autonegotiation take place and we're right back to ENIP without a hiccup.

One thing I did note is that when the module re-establishes communication the sequence number reported by Wireshark resets to 0 and begins incrementing again. I'm assuming this happens to you as well.

BTW my Wireshark is version 1.8.4

Timbert said:

A quick test. I pulled out an OB8S module, the POINT bus status light went flashing red as expected, Wireshark still reports ENIP as only traffic.

Pulling the Ethernet cable from between the controller and AENT results in no traffic at all.

When I plug the cable in I see autonegotiation take place and we're right back to ENIP without a hiccup.

One thing I did note is that when the module re-establishes communication the sequence number reported by Wireshark resets to 0 and begins incrementing again. I'm assuming this happens to you as well.

BTW my Wireshark is version 1.8.4

Click to expand...

Hi. Thanks for looking into this. Unfortunately I don't have access to the hardware. But going by what you said you could not trigger a CIP SAFETY protocol change from the ENIP protocol. Unfortunately Rockwell could not answer this question either just yet.