Arp traffic over Ethernet radio

In my opinion, there should be a router or firewall between the master radio and the rest of the automation or enterprise network, for exactly this kind of reason.

Once you've got an enterprise network that includes ordinary Windows workstations, or printers, or WiFi access points, or anything with Universal Plug and Play, the amount of multicast and broadcast traffic can be substantial. If you happen to have multicast I/O, it can get huge.

I once got thrown off a job shortly after showing the customer that their printer discovery traffic from Downtown was showing up over the radios at their remote pump stations.

There are certainly commercial routers and firewalls that do this sort of thing, as well as sturdy open-source stuff like PFSense.

You can also segregate the broadcast domain through VLANs. The problem is at some point you will have to route the traffic if devices on separate VLANs need to talk. You can also place a NAT router on the outbound side of the radios which will also stop multicast traffic and only allow addressed routable traffic.

Ken Roach said:

In my opinion, there should be a router or firewall between the master radio and the rest of the automation or enterprise network, for exactly this kind of reason.

Once you've got an enterprise network that includes ordinary Windows workstations, or printers, or WiFi access points, or anything with Universal Plug and Play, the amount of multicast and broadcast traffic can be substantial. If you happen to have multicast I/O, it can get huge.

I once got thrown off a job shortly after showing the customer that their printer discovery traffic from Downtown was showing up over the radios at their remote pump stations.

There are certainly commercial routers and firewalls that do this sort of thing, as well as sturdy open-source stuff like PFSense.

Click to expand...

Ken is spot on. I sold 900MHz FHSS radios for 20 years and we had an Ethernet version for about 10+ years. That technology has it's pro's and con's with the biggest pro being that it will go miles (with LOS) and the biggest con being that it is very slow as far as Ethernet is concerned. It's slow because of the time it takes for the radio to hop from one channel to another and with that you get a very small bandwidth I.E. very low throughput. That's were you are running into trouble, it might be related to some degree to the speed (or lack of) but more than likely it's the very narrow bandwidth that is causing the problem and is the main reason why you want to isolate the network traffic from the radio traffic. A router or managed switch is a good way to do this however if feasible, a completely separate network would be ideal. Long and short is the only network traffic you want to be accessing the RF network master (or the slaves if they are connected to an extended network) is network traffic with a destination that is on the other side of the RF network. Even then you need to make sure that the "expectations" are appropriate I.E. the sending/receiving devices are not looking for response times of 2ms (most Produce/Consume won't work very well if at all because of this).

Another approach depending on the throughput needs of the network are FHSS radios that aren't FHSS radios. Xetawave is an example of that. They are software defined radios and the guy who started the company was one of the founders of FreeWave and probably the smartest RF Engineer on the planet. If you're not locked into MDS or for future projects you may want to look at these guys (https://www.xetawave.com/). I've not worked with their radios before but I know Johnathan (the founder) and I'm very confident that what they list as the capabilities are real.

Ken Roach said:

Can you quantify "chatty" ?

Do you have a tap/mirror at the master side of this system that you are using the monitor the traffic ?

I just hooked up a tap and Wireshark, and monitored my 1756-ENBT using the filter:

Of course ARPs are broadcast packets and every device on the network sends them out periodically. If you've got a router on the network that's providing DHCP addresses to anything, it's probably the most common source of ARP packets.

Click to expand...

Yes had wireshark running on the far end, would see 8-10 ARP requests between each poll containing data I wanted. and the source of the arp was the EN2TR module dedicated to this radio system.

I will have better network segmentation at some point in the not to distant future, but to do that I need to do a massive IP change at the same time (network is flat at the moment) so that will take coordination and this radio stuff I can do now. When network refurb is done this traffic gets its own vlan but if the source of the ARP is the PLC comm module, that's not gonna change much. I can add a firewall or whatever it takes and I'm aware that is a solution, but I'm just mostly curious why the EN2TR module is sourcing these at all. I wouldn't think it should.

The problem with the ARP quantity is they take over a cycle of "store and forward" required for the radio to reach the valleys from the hilltops and this duplication of data just chokes the throughput.

This are not 900mhz, these are 200mhz, licensed freq and the RF coverage is ideal for the local terrain, but of course that comes at a cost of bandwidth, which was expected and is fine.

I'm pretty stuck with the MDS radios. It's sad to say these well reviewed models have been a royal pain because the Store and Forward tech would not work and we had to prove to GE that was the case. Because of that we have been responsible for several recent firmware updates from 6 to 7x. But unless I can prove they just wont work, I can't return them. And I can't believe they wont, Me or GE just have something set wrong, ARP should not be broadcast on this type of low bandwidth communication by default.

As I said the serial comm channel works great, Quiet until called upon and responds like lightning... not so much the ethernet portion.

So the battle trudges forward.

Maxkling said:

Also network storms are pretty much looping and multiplying ARP requests, I know it would probably cause more havoc than you are describing, but I’ve never seen how this would react with a throughput restriction of a radio in between.

Click to expand...

I created one of those loops a while back accidentally when I was using the microwave link as a backhaul. That was memorable to put it mildly.

Maxkling said:

You can also segregate the broadcast domain through VLANs. The problem is at some point you will have to route the traffic if devices on separate VLANs need to talk. You can also place a NAT router on the outbound side of the radios which will also stop multicast traffic and only allow addressed routable traffic.

Click to expand...

There is no need for this chain of equipment to talk across vlans. One module runs the radio network on it's own vlan and acts as a data collector, another EN2T module in that rack is on a different segment and plant SCADA system will post/get it's data through that interface. The PLC is common, but not the comm modules.

Wireshark will tell the tale.

There is all sorts of wild stuff you learn when you pop open the hood of your network and run Wireshark. It's easy for even experienced folks to go chasing rabbits.


> ARP requests between each poll containing data I wanted.

Great; can you associate a time in between those ? How frequent are the polls ?

If you can post that data (or share it privately) that would be interesting. In Wireshark you can just filter on (arp) and save the displayed packets.


What I've got to test is a 1756-ENBT running on a small network, testing a Baluff EtherNet/IP to IOLink block. But because I've connected everything to my local network, I see traffic from a bunch of other printers and phones and various embedded devices.

The 1756-ENBT sends out a "gratuitous ARP" about every 122.45 seconds. Gratuitous ARPs are basically network-change probes so that devices that have MAC tables can update them. The target MAC ID will be ff:ff:ff:ff:ff:ff and they should be labeled "ARP Probe" in Wireshark.

Similarly, my I/O block on the network sends out an ARP probe every 93 to 145 seconds.

I've got an RSLogix 5000 session talking to the ControlLogix too, and there are ARP packets exchanged between my computer and the ControlLogix periodically as well (anything from 5 to 300 seconds).

And other things on the network contribute ARP packets too. Phones, tablets, the printer, a Raspberry Pi, the Chromecast in the ship's main salon, the ESP32 monitoring the blackwater tank.

If all the ARP traffic you're seeing out in the field on the radio link is for devices that are on the radio network, then you likely don't have a way to reduce them unless that feature exists in the radios.

But if they're coming from devices that are not on the radio network, then you do.

There's always the possibility that something mis-configured or malfunctioning is causing more ARPs than necessary to come from the 1756-EN2T.

I forgot to mention that I have tested the 5069-SERIAL module including installing the firmware update that added DF1. It is a full suite of DF1 protocols and it works just fine.

I don't have any experience with the radios you're using, but we make extensive use of Schneider Trio Q series radios in the UHF band. When operating in Layer 2 mode, you have an option to filter ARP and/or broadcast traffic. If the GE radios don't have that, it might need to be a feature request?

I very rarely use Layer 2 though as the Trio radios fully support Layer 3 routing. I set up my remotes with a /27 subnet which gives me 30 addresses per radio, more than enough for a remote pump station with a couple drives and a RTU and HMI. But, you can also do larger subnets if you need.

The networking guy that did the backhaul routers and firewalls taught me a lot. We only allow DNP3, DF1 and SSH out over the radio network, the router that the master radio plugs into is a firewall as well.

These work so well that a less "radio aware" contractor went online with a remote site PLC for half a day from head office over the radio network. I only found out a few days later when he complained that it was a bit slow. I'd never tried it before because I knew how much traffic RSLinx generates when you go online with a PLC. He got a telling off, but I was kinda impressed that it worked. Its all a hell of an improvement from the 1200 baud analog radio modems we had before.

IT dept is installing a Palo Alto firewall (for other reasons) but I am going to take advantage of that when it arrives. However...

We have also had a consistent SAF problem with this radio since purchase and we thought it was something we were doing wrong... turns we could duplicate it for GE and it isn't something we did wrong. If they can't deliver a fix in short order then I'll be changing radios and moving on to a different set of problems. This has been going on for far too long.

I'll grab a capture again when this settles out.

Firejo said:

I forgot to mention that I have tested the 5069-SERIAL module including installing the firmware update that added DF1. It is a full suite of DF1 protocols and it works just fine.

Click to expand...

Well that's good news. wasn't sure they were actually going to do it.

If you do end up changing radios, Racom RIPEX radios and the Trio Q series are both products I've seen working well in the field (neighboring water authority uses Ripex). More than happy to help with configuring Trio radios if you ended up going down that path. Certainly learnt a few lessons on our roll out that resulted in a fair bit of driving around to fix config files until i figured out how to use the SSH shell!

Thanks Saffa

Had a conference call yesterday with GE management, 3 GE engineers plus my own guy. Excuses were given and promises were made. Looks like we will have made it through 2 revisions, 6.x, 7.x and now we will be 8.x . Our problem stems only from the forwarding that has to occur to get over the terrain in this area. On the sites that are LOS we don't have this issue. And I find that not every radio has a forwarding capability.

If this meeting doesn't bear fruit I'll be looking at all options. Not familiar with RIPEX but they will be looked at.


Still can't grasp the ARP issue though. ARP is necessary for switches, and I guess the EN2T modules consider themselves switches. Seems that blocking ARP at the radio or with a firewall ahead of the radio will only "frustrate" the EN2T module as it will keep asking. Perhaps Ethernet over low bandwidth is a bad idea all together. Or maybe it will be fine once the radios work like they should.

I'm ready for a beer.

TheWaterboy said:

Still can't grasp the ARP issue though. ARP is necessary for switches, and I guess the EN2T modules consider themselves switches. Seems that blocking ARP at the radio or with a firewall ahead of the radio will only "frustrate" the EN2T module as it will keep asking. Perhaps Ethernet over low bandwidth is a bad idea all together. Or maybe it will be fine once the radios work like they should.

Click to expand...

create a new subnet for each end of the wireless link, put a router on each end of the wireless link, put the EN2T modules in the new subnets, no ARP traffic over the wireless link.


I see EN2T modules in wireshark pumping out one ARP per second when they are trying to establish a connection.

I understand that would work, but there are a lot of sites that would then need routers and they have their own housekeeping traffic so I am only wondering why the radio doesn't handle the arp cache itself so it doesn't have to broadcast it over the air.

These radios can be routers but support told me that it wasn't necessary to go that far just for this reason.