Ethernet IP troubleshooting

I need some help with troubleshooting Ethernet IP network. I am at the start up of the machine with 15 Ethernet IP nodes. PLC processor is Control Logix 1756-L61.Network card is 1756-ENBT/A revision 6.4. Besides there are five 1794-AENT with remote IO, four Powerflex 40 drives, three Powerflex 70 drives, and two PanelView 6 Plus 1000. Everything is connected to the 16 port unmanaged Hirschmann network switch RS20. Nothing else is connected to the system exept my laptop.
The problem I am having that I am loosing communication to some or all drives. It happens at random once in 30 min or hour, regardless of machine is running or idling. Communication fault is resetable. After communication fault is reset machine is ready for operations.
I can look at the each module in IO tree, but I am not sure how to interprete various numbers.
I can access each module through the web browser. However I do not know how interprete these numbers either.
RPI for the drives are set at 50ms, and for the remote IO adapters at 20ms.
Please help with any suggestions.

Thank you,

PaulB

paulB said:

I need some help with troubleshooting Ethernet IP network. I am at the start up of the machine with 15 Ethernet IP nodes. PLC processor is Control Logix 1756-L61.Network card is 1756-ENBT/A revision 6.4. Besides there are five 1794-AENT with remote IO, four Powerflex 40 drives, three Powerflex 70 drives, and two PanelView 6 Plus 1000. Everything is connected to the 16 port unmanaged Hirschmann network switch RS20. Nothing else is connected to the system exept my laptop.
The problem I am having that I am loosing communication to some or all drives. It happens at random once in 30 min or hour, regardless of machine is running or idling. Communication fault is resetable. After communication fault is reset machine is ready for operations.
I can look at the each module in IO tree, but I am not sure how to interprete various numbers.
I can access each module through the web browser. However I do not know how interprete these numbers either.
RPI for the drives are set at 50ms, and for the remote IO adapters at 20ms.
Please help with any suggestions.

Thank you,

PaulB

Click to expand...

Does not help your immediate problem, but I always split out the ethernet into two phsyical networks, one for I/O and the other for HMI/SCADA and programming.That would require a 2nd ENBT card and separate ethernet switch.

You want to test your ethernet cables with an ethernet tester to check the cables and connections.

Also if you use your laptop and web browser to point to the IP address of the ENBT card, there are diagnostic pages that will show you the utilization % of the ENBT card.

What firmware in the processor? Your entire network is most likely multicast on a L61 so the unmanaged switch is likely 90% of you issues.Use the ethernet IP capaicty tools to see whee you stand on connections http://www.rockwellautomation.com/solutions/integratedarchitecture/resources3.html

I've checked ENBT card processor utilization. It is 34.8%
I have run ethernet capacity tool. It is saying that
CIP connection 52 of 128 used
TCP connection 14 of 64 used
I/O packet per sec 1530 of 5000 used
and 76CIP, 50TCP, 3470 IO and 575 HMI available for communications.

paulB said:

I need some help with troubleshooting Ethernet IP network. I am at the start up of the machine with 15 Ethernet IP nodes. PLC processor is Control Logix 1756-L61.Network card is 1756-ENBT/A revision 6.4. Besides there are five 1794-AENT with remote IO, four Powerflex 40 drives, three Powerflex 70 drives, and two PanelView 6 Plus 1000. Everything is connected to the 16 port unmanaged Hirschmann network switch RS20. Nothing else is connected to the system exept my laptop.
The problem I am having that I am loosing communication to some or all drives. It happens at random once in 30 min or hour, regardless of machine is running or idling. Communication fault is resetable. After communication fault is reset machine is ready for operations.
I can look at the each module in IO tree, but I am not sure how to interprete various numbers.
I can access each module through the web browser. However I do not know how interprete these numbers either.
RPI for the drives are set at 50ms, and for the remote IO adapters at 20ms.
Please help with any suggestions.

Thank you,

PaulB

Click to expand...

Play around with the R.P.I. settings. Make them 3x what they are now, and see if the problem changes.

If possible, move to point-to-point connections. IGMP filtering won't help here since your using an unmanaged switch.

jdbrandt said:

Play around with the R.P.I. settings. Make them 3x what they are now, and see if the problem changes.

Click to expand...

This will make the OP's problems of comms loss worse not better. You go up on the RPI and the packets go up also. Like putting more cars on the highway when it's already rush hour.

paulB said:

I've checked ENBT card processor utilization. It is 34.8%
I have run ethernet capacity tool. It is saying that
CIP connection 52 of 128 used
TCP connection 14 of 64 used
I/O packet per sec 1530 of 5000 used
and 76CIP, 50TCP, 3470 IO and 575 HMI available for communications.

Click to expand...

Still need that firmware version. If you are using a version that supporrts unicast then you may be able to solve this problem and use the switch you have. I would bet money that you have a multicast network and with that many nodes on a multicast network that unmanaged switch is not going to cut it.

Any chance this is down to auto-negotiation issues ?

I responded to another thread where comms was flaky, and a resolution was found - see post #17

HTH

The Plc Kid said:

This will make the OP's problems of comms loss worse not better. You go up on the RPI and the packets go up also. Like putting more cars on the highway when it's already rush hour.

Click to expand...

I think Jeffrey was suggesting 3 times slower, not faster.

paulB said:

I've checked ENBT card processor utilization. It is 34.8%
I have run ethernet capacity tool. It is saying that
CIP connection 52 of 128 used
TCP connection 14 of 64 used
I/O packet per sec 1530 of 5000 used
and 76CIP, 50TCP, 3470 IO and 575 HMI available for communications.

Click to expand...

Don't see any issues here...

daba said:

Any chance this is down to auto-negotiation issues ?

I responded to another thread where comms was flaky, and a resolution was found - see post #17

HTH

Click to expand...

Very likely.

daba said:

Don't see any issues here...

Click to expand...

I think the firmare may be something less than 18 where unicast was a choice or below 17 Not even a option at all and the network is just getting flooded by multicast traffic.

The embedded Web pages can help a lot, because they can tell you if there are errors on the Ethernet ports.

Do you know if you're having comms errors with the 1794-AENT modules as well ? These might not be as apparent because they would reset themselves, not latch in a fault like the drives do.

My first stop would be the embedded Web pages to see if there are obvious physical media errors showing up in the port/media counters.

The next step will be to look at the I/O connection webpage for the 1756-ENBT to see if the "Lost Packets" value for any of the I/O connections is going up.

The whole multicast vs. unicast thing might affect the PowerFlex 40's more than other devices, as they have the least powerful Ethernet chipset and might be having trouble filtering out the multicast data being sent to other devices.

Since this is a single-CPU system, and it sounds like it's modern firmware, you should set anything that supports it to Unicast.

What's the exact fault code you see on the drives ? On a PowerFlex 70, for example, error code 85 is different from error code 75, though they both are related to the 20-COMM-E module.

The Plc Kid said:

I think the firmare may be something less than 18 where unicast was a choice or below 17 Not even a option at all and the network is just getting flooded by multicast traffic.

Click to expand...

Whether the network has a multicast flood or not, the ENBT is handling it with ease - 35% utilisation is what it says...