Panel view and Micrologix disconnect when uplink connected

[GarlandTech]

Good afternoon PLCS.net!

I have a very odd situation that is happening, and hoping someone has some insight.

We have just laid out a new 10G fiber network, using managed switches into new buildings for a waste water treatment plant. From the switches in each building, there are cable runs to each cabinet, where there is a Rockwell dumb switch, and the Panelview and PLC.

When the uplink to the managed switch is connected to the Rockwell dumb switch in the cabinet, the Panelview and ML stop communicating. The devices can still be pinged from across the network (layer 2 subnet).

Testing from the dumb switch with a laptop, we also can ping everything across the network, as well as the two devices in the cabinet. The only issue is that the Panelview and ML don't talk. Disconnect the uplink to the managed switch, and we have full connectivity again within the cabinet.

Has anyone experienced this issue? Or does anyone know what protocol the Panelview uses to talk to the PLC itself? I ask because I know in a saw mill, we have to enable IGMP to allow the PLC's to connect correctly to some VFD's that we have, but those are directly connected to a managed switch and there is no dumb/unmanaged switch in the mix. I did enable IGMP on the upstream switch, but no change.

I'm about to head to the site this week and use a packet capture to see what is happening when the connection is made, but hoping there is something easy that someone has dealt with that could fix it.

Thank you!
Sean Garland - Network engineer with enough PLC knowledge to keep the machines running

 

[Ken Roach]

Welcome to the PLCTalk forum !

The most likely explanation is a duplicate IP address. I'm sure you've taken care to get the whole network configured correctly and don't think you've overlooked any rogue devices, but it's still the most likely explanation.

MicroLogix and PanelView will use TCP Port 44818 to communicate. No UDP, no multicast, no broadcast, just ordinary TCP/IP.

ICMP PING is a blunt instrument. Use at least TCPING, or Powershell's tnc command directed at Port 44818.

 

[GarlandTech]

Good evening Ken,

We checked pretty heavily today that there wasn't any duplicate IP's as the vendor thought that might be it. Disconnected from the uplink, on the rest of the network, no duplicate found. And we have 2 cabinets, with same issue, so something is awkward.

Thank you for the info about communication and the other test tool for pinging against the port itself. We'll give that a shot tomorrow, and if I have to head out there I will with packet capture, but that is a huge help.

Thanks you - to be continued....
Sean

 

[GarlandTech]

Update and resolved!

Went onsite to troubleshoot once and for all the issues there with the network.

*** FANFARE ***
The issue? The engineers, had setup polling against the ML processors from a much higher end control logix processor that could only pull data when the backhaul was plugged in. As soon as the backhaul was plugged in, the PLC in another cabinet started hitting the ML processors as hard as you can do with a ControlLogix processor, effectively doing a denial of service attack, and taking them offline.

Engineer backed off the requests and turned off a number of tags, and low and behold everything was okay.

Moral of the story - if you enable some stuff, and things start breaking, back them off before pointing at the 10G fiber network with enterprise switching as the problem. Multiple engineers, including Rockwell support couldn't figure it out. I'll bet nobody mentioned that they were pulling "real time" from them from another processor.

Anyway - thank you for your help, and issue is resolved and only took about 4.5 hours of pulling connections, and disabling gear to figure it out, and 5 minutes to fix on the programming side.....

Sean