Ethernet I/P troubleshooting with Wireshark

RonJohn

Lifetime Supporting Member
RonJohn
Join Date
Jul 2013
Location
NE Ohio
Posts
535
I could use some help getting educated in using Wireshark to troubleshoot an Ethernet IP issue between a Logix processor and Kollmorgen AKD drive. There's a Stratix 6000 managed switch between the two so per Rockwell, we set up port mirroring on the Stratix and I was able to get a meaningful capture and send it off to tech support.

Now the drive folks want me to eliminate the managed switch and attempt the same test with an unmanaged switch. When I do so, I don't see any data in my Wireshark capture involving the drive. My unmanaged switch is a plain-jane switch, no built in port mirroring.

I don't fully understand port mirroring and why its needed to see PLC to drive comms going through this switch. I would think that in a simple network containing the PLC, the drive and my PC that I should see everything going to/from the switch with Wireshark. Any explanation would be greatly appreciated.

Thanks in advance.
 
I've seen this behavior before as well. I think it has to do with the PLC opening a direct connection with the Drive, and not broadcasting to the whole network. I would be interested in hearing a more in-depth explanation myself.
 
I don't fully understand port mirroring and why its needed to see PLC to drive comms going through this switch. I would think that in a simple network containing the PLC, the drive and my PC that I should see everything going to/from the switch with Wireshark. Any explanation would be greatly appreciated.
Click to expand...
The un-managed switch memorizes what is on each of it's ports, and only forwards the messages that are destined to something on the port. Therefore, when the switch figures out which port the drive is on, and which port the PLC is on, it will cease sending anything out the port your PC is on.
 
Sounds like your drive guys are just grasping at straws.

Unfortunately with out an ethernet hub or a managed switch like you were using, wireshark is useless, since unmanaged switch traffic is directed to the port of the assigned mac/ip address.

You could use this hub in place of your switch, and then you could use wireshark. It's a fools errand that they're asking you to do, but at least that should satisfy them.

http://www.amazon.com/Netgear-EN104TP-4-Port-Ethernet-Uplink/dp/B00000J4M9
 
Thanks for the inrush of interest, folks! Basically, here's the highlights:
- I'm currently bench testing this setup prior to connecting to the actual process equipment, so I have a bit of latitude to play around.
- I was able to connect the drive to the PLC with no issues. I'm using the drive mfg. specification of unicast connection over EI/P, RPI = 20ms.
The problems start when a connection is temporarily lost, i.e. disconnected Ethernet cable. RSLogix comes up with a 16#0204 error indicating a connection timeout error.
- When the cable is plugged back in, the connection doesn't get reestablished and 16#0204 stays put, along with the yellow triangle. Note: I can ping the drive just fine at this point.
- The only way I've discovered to resolve the issue is to cycle 24VDC to the drive. I've already informed all parties involved that this is NOT a workable solution!
- Rockwell is already pointing the finger at the drive folks. The drive folks are pointing at the Stratix since their in house testing used an unmanaged switch. I now know that the issue occurs with an unmanaged switch also.

From another post, I saw a link to a switch with built in port mirroring. That may help down the road, but not today.
 
Gomez_ said:
I've seen this behavior before as well. I think it has to do with the PLC opening a direct connection with the Drive, and not broadcasting to the whole network. I would be interested in hearing a more in-depth explanation myself.
Click to expand...

So, Gomez, are you saying that checking the Unicast Connection over Ethernet/IP selection in the module properties can cause this issue to present itself?
 
Basically local network communication is done by MAC addresses. Devices have an ARP Table which is a mapping of IP to MAC addresses. Switches contain a MAC Table which is a mapping of MAC addresses to (Physical) Ports.

When a device wishes to send a packet to an IP address it checks it's ARP Table to see if it already knows the MAC address. If there is no listing in the ARP Table then the device will send out an ARP Request, which basically says "I am IP Address so an so and my MAC Address is blah blah blah, Who is IP Address such and such and what's your MAC Address?" The Switch will receive the ARP Request and since this type of request is a broadcast request the Switch will send the request to all it's Ports. At this point the device with the IP Address such and such will say that's me and will send an ARP Reply stating I am IP Address such and such and my MAC address is yada yada yada. The Switch sends the ARP Reply back to the originating device and the originating device updates it's APR Table.

When the Switch receives a packet it looks at the header and checks the Source and Destination MAC Addresses. The Switch will update it's MAC Table with the Source MAC Address and look for the Destination MAC Address in the MAC Table. If the Destination MAC Address is in the MAC Table the Switch will send the packet to the proper port, if the Destination MAC Address is not in the MAC Table the Switch can send a Flood to all the ports to try and determine which port the MAC Address is at.

What should happen when cables are unplugged?
  • If you unplug the receiving device and plug it back into the same port, everything should pick right back up.
  • If you unplug the receiving device and plug it back into a different port, one of two things should happen 1)the entry in the MAC Table will time out (maybe 5 minutes) and the Switch will then try to determine where the device is at or 2)the originating device will determine there is a problem and start sending ARP Requests.
  • If you unplug the originating device and plug it back into the same port, everything should pick right back up.
  • If you unplug the originating device and plug it back into a different port, everything should pick right back up because the Switch will read the packet header and update it's MAC Table with the new Port.

As others have pointed out, get a HUB and use Wireshark to see what's happening when communication is lost and then restored.
 
Be careful not to conflate "this issue" with the Kollmorgen drive failing to reconnect to the ControlLogix with "this issue" of how ordinary Ethernet switches work and how Wireshark needs a mirror port to monitor traffic.

Because you're using a Unicast connection, it's using very ordinary IP addressing and not attempting to perform multicast with a multicast group IP address. That means that any switch, managed or unmanaged, is only going to send data between the drive and the PLC; it won't go to any other port unless you specifically set up a 'mirror' port.

I do like that DualComm switch because it's as close to unmanaged as you can get while still getting the mirror port. You can eliminate any questions about whether the various management features like VLANs or QOS or port roles or (yadda, yadda) are involved.

When I'm doing heavy-duty Ethernet analysis, I break out my main battle tank and connect my Ethernet CommProbe and fire up Frontline Test Equipment's NetDecoder.
 
Thanks Tark for the detailed explanation. That clears some stuff up for me and confirms some other stuff.

Ken, your thoughts are appreciated as always. I have to admit, I had to look up "conflate" to verify the definition ;) good stuff. That Ethernet ComProbe looks to be quite the toy. We'll have to check that out.

The issue occurs regardless of managed or unmanaged switch on my setup with a 1756-L71 & 1756-EN2T/D. The Wireshark trace on the managed switch showed that the controller will close the connection if the drive responds with a session = 0, so I suspect the same issue occurs on the unmanaged switch. Kollmorgen can't duplicate the issue on their end with a 1769-L32E and they feel that the EN2T is not giving the drive enough time to respond. I'm going try using a CompactLogix PLC (L18ER) that we have on hand to see if I can narrow this down.

I'm beginning to suspect the EN2T/D. Rockwell tech support asked me if Kollmorgen tested their drive with this new module as "a lot has changed" with this new version. Frankly, I don't know if it's on the 3rd party to verify their product with every conceivable A-B Ethernet device or for A-B to test their new EN2T on every Kollmorgen drive!:confused: All I know is I'm stuck somewhere in the middle, beta testing so some poor soul can avoid this same grief in the future.
 
Last edited:
Just wanted to give an update: Kollmorgen was able to duplicate the issue when using a CompactLogix PLC with V21+ firmware. This coincides with my own findings using an 1769-L18ER (V21) and the aforementioned 1756-EN2T/D (V10.1).

They're working on a firmware upgrade for their drive. Depending on their time frame, I may end up stepping the L18ER back to V20 and call it a day.
 

Similar Topics

D
Troubleshooting possible Ethernet/IP problems
I have recently added a ControlLogix 1756-L72 5570 to our industrial network and brought a number of tags into our existing ignition project...
Replies
2
Views
1,982
ArtWalls
A
J
DLR Ethernet troubleshooting help
I've been trying to troubleshoot a DLR network for the past month with some forward progress, however I've exhausted all of the things to try that...
Replies
6
Views
4,803
Plastic
P
P
Ethernet IP troubleshooting
I need some help with troubleshooting Ethernet IP network. I am at the start up of the machine with 15 Ethernet IP nodes. PLC processor is Control...
2 3
Replies
34
Views
15,267
rguimond
R
C
Control Logix Ethernet Troubleshooting
I am currently using a Control Logix Processor (L55) with an ethernet bridge (1756-ENBT/A) and accessing it with a half dozen HMI PC's, each with...
Replies
11
Views
4,013
Operaghost
Operaghost
rPraveenkum
Troubleshooting an EtherNet/IP system
Troubleshooting an EtherNet/IP system
Replies
0
Views
1,739
rPraveenkum
rPraveenkum
Back
Top Bottom