Intermittent Power Flex Ethernet Com Fault

[SunKing]

System has been in place for 4+ years with no problem...There are 2 (nearly identical machines) that each have 6 Powerflex VFDs (5)( PF700 & 1 PF40) all communicating over an Ethernet Network... Problem started about 5- 6 months ago...When the fault occurs, it is for Port5 Adapter error.... The fault can occur a couple times within a few minutes, or it can be days in between (typically no longer than a week without a fault)...When the fault does occur usually multiple drives in each machine are affected (but not always all of them) and usually faults occur in drives of both of the machines at once...When the machines are disconnected from the Plant network there are no faults. (big clue I know!)..No other equipment on the plant network has these type of issues...The plant network consist of about (60) other AB devices including PLCs, HMIs, & VFDs and (of course) Data collection...for some reason only these (2) machines are affected...

So it seems apparent that something is causing the drives to fault when connected to the plant network, but I am not sure what the cause could be or even how to begin measuring or isolating the problem...Any help is appreciated .

 

[Firejo]

How are the drives connected to the plant network I.E. via a hub, switch or managed switch? The preferred method would be a managed switch preferably one with IMGP snooping.

 

[Jeev]

4+ years sounds like it's before AB's transition to Unicast support on Eth/IP connections. As above, I would have a look at what's going on with traffic and network hardware. I've had SEW MoviDrives go down in the past thanks to Multicast traffic, and more precisely; a certain piece of asset managing software that would flood the network periodically. On that site, we had ControlLogix PLC's, so we physically segregated machine and site networks with a second ENBT card.

Multicast, just another drawback of open connections between Automation and Administration/Site networks.

 

[Ken Roach]

Do you know if the fault code on the PowerFlex 700 drives is a 75 or an 85 ?

The 5 is DPI port 5, but the 7 indicates a problem on the network side and the 8 indicates a problem on the DPI ribbon cable side (or a total failure of the module). The PowerFlex fault history should show you.

The 20-COMM-E may have some error statistics that would show noise or network traffic storms. The connected ControlLogix interface will, too. These are usually accessed via the onboard Web servers.

How do the drives recover ? Do the Ethernet connections just come back after a few seconds, or do you have to power cycle or something else ?

Are you familiar with Wireshark ? It's a free, open-source network analyzer that is a great way to get your hands dirty with network analysis. Download it from wireshark.org and do a quick capture of Ethernet traffic while connected to a spare port in one of the cabinets with these drives. That's going to give you your first look at how much multicast and broadcast data might be flying around your network.

Edit: Thanks, by the way, for launching this as a new thread. Easier to keep track of this particular set of facts and devices and history.

 

[SunKing]

Thanks everyone for the quick response and good ideas....
Firejo - Machines were originally installed (and ran fine for years) with unmanaged switches...When this problem arose, the engineer working on this issue before me changed them to managed switches, but this didn't seem to help (although I don't know if they had IMGP snooping or not)...They are currently running off of Cisco switches supplied by IT (not sure managed or not) but still same issue.

Ken - I will find out for sure what the fault code is, and the recovery method (this issue has not personally happened to me yet, I was just tasked with figuring it out) - Was not familiar with Wire Shark, I will try it out on next visit, (currently my understanding of Ethernet networks doesn't go much beyond assigning IP addresses, guess it is time to learn more huh?) ...Thanks for all the good advice

 

[Jeev]

If you have IT, they should be able to VLAN your machine traffic off the network and use NAT for access to devices from site (PLC and HMI). Also IGMP Snooping does not always work to limit multicast traffic. It can still set up relationships that allow that traffic out into the site network, and vice versa.