L61 IPv4mcast

Hi,

I have 2 PLC's, (1) L61 and (1) L55 that broadcasting a butt load of MCast packets. WireShark shows about 87 million per day.


We have 25+ PLC's on plant. Mostly 1756-L61, One L55, several 1769-L32E and several SLC500

The other 1756-L61 do not Mcast packets.
All L61’s were installed at same time when mill was upgraded.
All L61's are flashed to 15.x (been in service about 5 years.)
Fiber and cat5. Katron Tech, KTI managed switches but not managed (plugged in out of the box)


Only been at this plant a short time, they told me they have always had a super slow network. They didn’t know what the problem was so about 2 years ago, they hired a firm to separate the HMI displays from the Process Network because the network was so slow. Did not cure the problem. (I’m assuming no one sniffed packets or checked band width usage before coming to the decision to separate networks)

Worst offender is one L61 - flashed 15.1 It has 9 remote racks over ControlNet. Rack Optimized. Mix of CNB/A, IA16I, OW16I, IF6I, OF6CI in remote racks. ENBT/A, CNB/A, IR6I & OF6I in main rack

Destination is an address outside our network
Over 1000 packets per second.
239.192.2.99
239.192.2.101
239.192.2.100
239.192.2.102
239.192.2.98

The L55 - flashed 15.5 only has ENBT/A and DNB in main rack. 1 remote DeviceNet Beckoff and 1 remote DeviceNet panelview


Destination is an address outside our network.
239.192.3.192

I talked to Rockwell Tech support, he said it will not be an easy or simple fix.

Tech said we need to be at Rev 18+ (Flashing a processor and loading program will not be an issue. I have spare L61’s) He emailed me Tech Note #66324 Unicast I/O Support in Logix.


As far as I found all installed I/O cards are compatible but there is no mention about the CNB cards.


Not really understanding why Rockwell Tech said it would be a complex procedure. If it is simply flashing to 19.x and reloading program. Not an issue but there must be "The Rest of The Story" as Paul Harvey would have said.


Has anyone else run into an Mcast issue and how did you resolve it.


Attached is a 3 Centisecond time slice screen grab



Greatly appreciate your guys input.

Best Regards
Allen

.

Screen grab of Mcast packet data.

.

EtherNet/IP class 1 io communications. make sure you have a proper igmpv2 configuration on your managed switches, per the requirements. Tune down your rpi settings as well.

What you see is most likely coming from the produced tags that is consumed by another controller (or controllers). This is the only way that local ENBT will produce multicast.

So you need to investigate other controllers that have this processor in I/O configuration.
Once found, you need to make sure that these program have ENBT 192.168.0.12 listed with Comm Format "NONE".
Then you need to find all consumer tags in these processors and verify RPIs.

Since V15 does not support unicast, to use produced/consumed tags, you must manage your "managed" switches to handle multicast, enabling IGMP snooping and possibly IGMP querier.

Based on your screenshot, you have two devices sending multicast:
192.168.0.12 and 192.168.0.22
What are these devices?

It would be helpful if instead of picture you post complete unfiltered traffic capture (1-2 seconds), then I can tell you what devices actually consuming multicast.
These programs actually need to be reviewed.

Contr_Conn said:

Based on your screenshot, you have two devices sending multicast:
192.168.0.12 and 192.168.0.22
What are these devices?

Click to expand...

Hi, Greatly appreciate your help.

The .12 and .22 are sending the most packets. They are the ENBT/A cards in the Racks.

There are several other devices but they request about 1 sec intervals.

When they set this network up, not sure why they decided to use the 192.168.0.x network but it is what it is.

I attached an Excel sheet I put together of the 192.168.0.12 rack that is mcasting the most. Basically recreated the tree from RsLinx.

Also attached a zip of WireShark PCAPNG file format packet capture. Researching how to decipher the packet.

Hope this info is helpful, greatly appreciate your help.

Unfortunately, you did not enable port mirror when you did capture.
As a result, you can't see any unicast TCP traffic that will be the key to identify what devices consume data from this PLC. These devices would acknowledge multicast packets (via unicast) and we can identify them.

I can only say that this is definitely Produced/Consumed traffic.

So start with basics - configure managed switches:
- first for port mirror to mirror .12 traffic to the PC. and do capture.
- then set to .22 and do it again.

Once done, disable mirror and configure switches for IGMP snooping. If correctly set, then your PC will not see any multicast any more (with port mirror disabled).

Contr_Conn said:

Unfortunately, you did not enable port mirror when you did capture.

- snipped part of quote-

Once done, disable mirror and configure switches for IGMP snooping. If correctly set, then your PC will not see any multicast any more (with port mirror disabled).

Click to expand...

I will scope it out.

If I understand what you mentioned in other messages above.
The Mcasts are caused by other PLC's on the ENET requesting data/ tags that were set up in the ENET tree? kinda in a nutshell?

If what I gathered reading about Mcast/ Unicast, what you mentioned, and talking to RS Tech there are several options.

1) Install a switch to block/ direct the Mcast packets. This does not fix the root problem. The PLC would still be broadcasting, just get blocked/ redirected by the switch. Consuming processor time/ resources and still creating network traffic (to the switch).

2) Flash processors to Ver 18.x plus. use Unicast

If the Mcasts are caused by other PLC's requesting tags, for a quick fix, could a guy message the data to the requesting PLC?

Nothing is critical time based so 2 second message. redo tags in PLC to point to message file. This would keep all produced and consumed tags internal to the PLC that was requesting the data?

Could that work?

I am on the last day of my 3 week rotation. Working in Alaska, fly back to Montana for 2 weeks on Monday.

Will do some more research when I am at home.
When I get back to Alaska, I also plan on setting up a Cacti Server to monitor network traffic.

If I understand what you mentioned in other messages above.
The Mcasts are caused by other PLC's on the ENET requesting data/ tags that were set up in the ENET tree? kinda in a nutshell?

Click to expand...

Yes, another PLCs have this Processor in the I/O tree for produced/consumed communications.

1) Install a switch to block/ direct the Mcast packets. This does not fix the root problem. The PLC would still be broadcasting, just get blocked/ redirected by the switch. Consuming processor time/ resources and still creating network traffic (to the switch).

Click to expand...

Your root problem is that you did not configure switch correctly and did not manage RPIs.
You already have managed switch, it just needs to be set for IGMP Snooping
There is nothing wrong with multicast traffic, but by default it propagates to all devices slowing them down. IGMP snooping isolates this traffic, similar to what switches do with unicast.

2) Flash processors to Ver 18.x plus. use Unicast

Click to expand...

No need to do it, you have to correct RPIs, I/O tree comm formats and set switch to IGMP snooping.
And you can't flash L55 to ver 18.

If the Mcasts are caused by other PLC's requesting tags, for a quick fix, could a guy message the data to the requesting PLC?

Nothing is critical time based so 2 second message. redo tags in PLC to point to message file. This would keep all produced and consumed tags internal to the PLC that was requesting the data?

Click to expand...

I would not use this approach, Poduce/Consumed tags are the most efficient way. If you need data on 2 second interval, then why not to set consumer program RPIs to 2000ms?
Today you have it at 5ms and 10ms based on the traffic capture.

You locked your mind down on the eliminating multicast and this is a wrong approach.
Multicast is Ok if set and managed correctly.

Instead, you have to fix the root cause:
- wrong RPIs.
- no IGMP snooping.

Contr_Conn said:

Yes, another PLCs have this Processor in the I/O tree for produced/consumed communications.

Your root problem is that you did not configure switch correctly and did not manage RPIs.
You already have managed switch, it just needs to be set for IGMP Snooping
There is nothing wrong with multicast traffic, but by default it propagates to all devices slowing them down. IGMP snooping isolates this traffic, similar to what switches do with unicast.


No need to do it, you have to correct RPIs, I/O tree comm formats and set switch to IGMP snooping.
And you can't flash L55 to ver 18.

-snipped_
Instead, you have to fix the root cause:
- wrong RPIs.
- no IGMP snooping.

Click to expand...

Thank you, getting better understanding of what is going on.

I will research and figure out how to config the switch. When I.T. set up the system, they just installed out of the box. The company that programmed the system did not set them up either.

Most of the RPI's are set at 20 to 25 ms.

they just installed out of the box

Click to expand...

Why waste money for managed switch if it's not configured? Makes no sense.

Most of the RPI's are set at 20 to 25 ms.

Click to expand...

No, they are not.
Based on the traffic capture they are 5 to 10ms

Again, you are looking at wrong RPI, you need to see RPIs set in other PLC programs for consumer tags for this connection.

Contr_Conn said:

Why waste money for managed switch if it's not configured? Makes no sense.

No, they are not.
Based on the traffic capture they are 5 to 10ms

Again, you are looking at wrong RPI, you need to see RPIs set in other PLC programs for consumer tags for this connection.

Click to expand...

LOL, the amount of money wasted when they built this plant about 5 years ago would blow your mind. lol I could tell you stories but all you would say is 'No Way' be hard to tell you one story and not break into 10 other stories that related to it. Have to buy you a beer sometime and tell you about it. Heck, it going to take a lot of beer to tell you everything

Yea, I was looking at card RPI not the consumed tag RPI.

5 tags are set to 5 ms (at least one tag in each processor)
5 tags are set to 500 ms (all 5 in one processor)

I downloaded Logix 5000 Produced and Consumed Tags for some reading material while sitting in airport.

The destination Ip was really bugging me as I can not ping the 239.192.2.xx network from the process network. 192.168.0.xx

I pinged those ip's on the HMI Network which is 192.168.1.xx
pinging 239.192.2.xx, I get a ping back listed as 3 processors on the 192.168.0.12, .14 and .22

Not sure how they are doing that, more research needed. I'm going to assume if I ran Wire Shark on that network, be mcasing there too.

Need to get the switch configured correctly.

I greatly appreciate your help, learn something new everyday.

239.192.x.x is automatically generated multicast address.
It's based on the device IP address.
There is a calculator in the knowledgebase.
All multicast addresses originated in 192.168.0.12 are generated correctly.
If you don't need 5ms, why not to change it to a higher number?

I also see 10ms multicast (239.192.2.98). If you don't have any tags at 10 ms, then most likely I/O config "Comm Format" was set to Rack Optimization in consumer programs. It must be NONE.

Contr_Conn said:

239.192.x.x is automatically generated multicast address.
It's based on the device IP address.
There is a calculator in the knowledgebase.
All multicast addresses originated in 192.168.0.12 are generated correctly.
If you don't need 5ms, why not to change it to a higher number?

I also see 10ms multicast (239.192.2.98). If you don't have any tags at 10 ms, then most likely I/O config "Comm Format" was set to Rack Optimization in consumer programs. It must be NONE.

Click to expand...

The tags are mostly tank levels that change very slowly, guessing why they set some to 500 ms but no reason they set some to 5ms. Not going to make any changes til I get back. Been like this since installed 5 years ago.

Packing up shop and getting ready to fly out.

I might take a managed switch home with me to play with.

I greatly appreciate help. Thank You.

.

Can you post programs for "Consumer" PLCs?