PLC Network communication problems

Hello,

I have been having some issues with my plc's that are on thier own vlan.

I have a server with 4 network interfaces that sits on my business network and my plc network. The server gathers historical data from the plcs. However, after my power outage of about 5 hours last week, my switch (Catalyst 2960) had lost it's previous setting because the last admin didn't save it to the startup config. Therefore when the batteries on the switch ran out, so did the vlan config. So my issue is this...

After putting my vlan back together on one of the switches, I can no longer ping one of the PLC interfaces. Ping everything just fine from all the other machines on this network. All the other machines on the PLC network can ping the server on the interface it has on that network. One of my PLC just can be reached by the server. After a restart, I'll be able to reach the PLC I couldn't before but then I cannot reach one of the other ones. This problem seems to be hopping around the vlan and seems to involve my server's ability to contact any node on the vlan at any given time after a restart or an outage.

I have mirrored a port to look at what's going on here and I see a whole lot of multicast traffic from my Allen Bradley PLC interfaces to addresses like 239.192.3, .2, and .4 . Is this some kind of diagnostic or debug protocol? Wireshark lists it as ENIP. However it accounts for the better part of half the traffic on the vlan. There is no scheme like that on my network so it can't really be going anywhere. ARIN lists the domains for this as specialty reserved items that are nothing. They are listed for IANA special use. I don't really think this is causing a packet storm on my network but I could be wrong.

Any ideas would be appreciated. I know I might be excluding some other information here. Please let me know as I am really stumped by this.

Thanks,

Jeff

You're in good company: that traffic is the first thing that anyone troubleshooting a ControlLogix I/O network encounters.

What you're looking at is multicast production of data by Allen-Bradley ControlLogix controllers and I/O devices. This is totally normal traffic for those I/O networks.

Usually this traffic is constrained to ports that are part of an IGMP multicast group by switches that are configured to perform IGMP Snooping. The switches observe the devices subscribing to the IGMP multicast group and keep the traffic limited to ports that have devices in that group. There also has to be an IGMP Querier out there on the network as well; sometimes it's a manages switch, sometimes a router.

The Catalyst 2960 was probably configured for IGMP Snooping and IGMP Querying prior to its configuration loss.

There are quite a few technical notes in the Rockwell Automation Knowledgebase and elsewhere about how to configure switches for IGMP Snooping. The user manuals for the Rockwell Stratix 8000 switches might help you too: these are essentially Cisco IE3000's so they are very functionally similar to Catalyst 2900 series from an OS and configuration standpoint.

A network diagram complete with IP addy's and desired routing paths will help us help you solve this.

Can you post something?

Thanks for the reply Ken. I will check to see if its on but the Cat2960 is supposed to default to IGMP snooping as turned on. Still I'm sure this traffic was there before I got here. What I'm not understanding (and I know there's plenty I don't understand about PLCs) is why I can plug my laptop into the same vlan port with a static ip address and reach everything consistently just fine. I am going to try one of the different interfaces on my server tomorrow morning. This problem I'm hoping doesn't go into next week without being resolved. That's when the machine behind the PLC I can't reach has to come back up. Luckily it's down for repairs right now. However, I should still be able to ping the plc from the server since it is still online and I can reach it from everywhere else on the vlan including my laptops wired adapter. I'll put something together that I can post on here and hopefully it will help for a diagram. I could post a wireshark pcap too if that will help