ControlLogix Wireless I/O Comms Losses. RPI Maxed

[anthr_plcnewbie]

Hi All,
I am having lots of issues with wireless I/O and losses.
Here is what the architecture is:
Central - ControlLogix L62 processor.
Remote 1 & Remote 2 are hardwired to the L62 processor.
S1, S2, S3, S4, S5, S6, S7 - Wireless Ethernet/IP through Prosoft RLX2-IHG radio.

S1 - 1747-AENTR
S2 - 1747-AENTR
S3 - 1747-AENTR
S4 - 1747-AENTR
S5 - 1747-AENTR
S6 - 1769-AENTR
S7 - 1769-AENTR

I get 'comms not responding' with the yellow triangle on my I/O modules at the wireless satellites randomly. My RPIs on EACH module at the wireless satellites are maxed i.e. 750ms and yet I see comm losses. I am unsure what other steps I can take.

The radios at satellites are configured to 'Automatically choose best method to get to Master' even if it means to reach the master using hops.

 

[anthr_plcnewbie]

EDIT:
The signal strength on the radios installed at each tower is pretty good. They are in the range of -40dBm to -50dBm. There are a lot of retries though. Couple of satellites have more than 60% retries on them.

 

[JeremyM]

What does the total network look like? What kind of access points are the radios attached to? Is the whole system on its own subnet or is there some routing going on between devices? Is there other traffic on the wireless network? Was this previously working and now intermittent?

 

[anthr_plcnewbie]

Hi,
The whole network has its own subnet. The network has wired (modbus devices, HMI server) and wireless devices (remote satellites already mentioned). It has not been working solidly. It has broken few times. I was monitoring the module errors in the past 12 hours and I haven't seen one comm loss. However, last night it was terrible and two satellites were dropping like crazy. Don't really know for what reason. It is intermittent. I am using GSV instruction to capture the module faults/errors and run a counter to see how many times they drop out.

The radios are Prosoft RLX2-IHG installed as master for the main controllogix and then repeaters at each remote site, 7 remotes in total. The signal quality of the radios seem to hover around -40dBm to -65dBm, which I believe is quite good.

 

[Geoff White]

Have you checked to see if any rogue access points (operator phones on Hotspot) have been set up on the same channel?

Has someone brought a shoddy microwave oven into the lunch room?

 

[harryting]

I might be mistaken but I'm pretty sure Ken said that wireless just isn't going to work for IO. You can, however, do messaging on wireless and get consistent result.

 

[anthr_plcnewbie]

No microwaves, no hotspots. Browsing using the wifi analyzer, the channel is empty with only the radios on it. I've been monitoring and the signal has been rock solid in the past 48 hours. I have had 0 losses.
I changed a config on 3 radios to 'automatically choose best way to reach master'. Initially, I thought all of them were setup this way but they weren't. These three weren't setup the way I had hoped. I hope this fixed the issue.
I cannot do messaging because they are remote I/Os and not controllers at satellites.

 

[harryting]

Understand your setup. Just saying it's not going to work consistently according to what I read. If you do a search this very issues comes up quite a bit. Here's one:

http://www.plctalk.net/qanda/showthread.php?t=89574

There are a few wireless expert on this forum, Firejo is one of them. They can give you more in-depth suggestions.

 

[anthr_plcnewbie]

Thanks for the link. I do understand that it will drop packets, it is wireless, it is not reliable.
I was handed the site as-is with originally 5 satellites, and I added in two more and upgraded central. So I basically inherited and ran with it.
The process is not demanding and thus I reduced the RPI to max it can be. With this I was hoping fewer, if any, comm losses. Although, it might not be feasible at this point but my recommendation is going with fiber for future.

 

[Firejo]

Understanding that typically wireless and produce/consume don’t play nice together, one thing you can try is staggering the RPI settings. As it is now, all of the produced tags are getting consumed at roughly the same time and there might be too much data trying to get through at once. If you take the tags with the least amount of data (discrete tags if you have them) and shorten the RPI settings to say 500ms, then 525, 550, etc… then not all of the data will be coming back at the same time freeing up bandwidth. Look at it this way, if you have a two lane highway and 500 cars that need to go from point A to point B, if all 500 cars got on the highway at the same time you’d have a traffic jam. However if you let groups of only 25 cars (per group) on every 30 seconds traffic would be smoother.
This is just a “shot in the dark” suggestion but I’ve been testing some new high speed modems and when I stagger the RPI settings in the test equipment I’m using (I’m testing Produce/Consume using ControlLogix) I get far fewer connection resets compared to when I have all the RPI’s at the same rate.
One other thing to consider trying is fixing the radios RF speed and fixing the path back to the Access Point (master). In both cases, the radios are using the bit error rate to determine health of the connection and if the bit error rate gets to high the radios re-negotiate the connection path and rate. This process will cause a gap in the communications which will cause dropped connections. If the radios are set with a fixed RF speed and a defined path to the AP then they (typically) won’t drop the connection. Normally I recommend leaving the RF speed at “auto” but from time to time I do come across applications where they run better fixed. Try the slowest speed first and if it gets better try increasing the speed until it starts getting worse a gain (find the sweet spot).
Keep in mind that if the application cannot tolerate any dropped connections then wireless isn’t going to work. No matter what you do you will lose packets with wireless and statistically that will eventually lead to a dropped connection.

 

[Ken Roach]

I recommend taking a step back and making sure that the radios are only handling the traffic they're supposed to handle.

Do you have a managed switch at the master CPU station that you can set up for port mirroring, so you can intercept data with Wireshark ?

Examine all of your I/O connections to be certain they're set up for Unicast, as well.

 

[Ken Roach]

However, last night it was terrible and two satellites were dropping like crazy.

The Delta Aquariid meteor shower was at its peak on July 28.

I'm not saying that because meteors cause radio dropouts, but because I was sitting on the command bridge of my boat at 48 degrees North, sipping Green Spot Irish whiskey on ice, watching meteors streak across the sky on a very hot and clear night.

Sometimes systems that have radio repeaters work fine under normal conditions, but when conditions are better than normal, the sites that normally need a repeater to hear one another can now get the primary signal through.

That's not as big a deal with clever radios that seek one another out and establish paths and do packet switching. But boy, it's havoc on simple radios working with simple protocols.

 

[Firejo]

I recommend taking a step back and making sure that the radios are only handling the traffic they're supposed to handle.

Do you have a managed switch at the master CPU station that you can set up for port mirroring, so you can intercept data with Wireshark ?

Examine all of your I/O connections to be certain they're set up for Unicast, as well.

Ken makes a great point that I completely missed. A lot of applications involving wireless Ethernet don’t have good managed switches in place between the radios and “other stuff”. It’s always good practice to use a managed switch with wireless unless the only thing connected to the radio is the PLC (or I/O rack).

 

[anthr_plcnewbie]

I recommend taking a step back and making sure that the radios are only handling the traffic they're supposed to handle.

Do you have a managed switch at the master CPU station that you can set up for port mirroring, so you can intercept data with Wireshark ?

Examine all of your I/O connections to be certain they're set up for Unicast, as well.

Thanks Ken.
The network is definitely a mess and there are no managed switches installed. There are about 13 modbus/tcp devices hardwired to switches all over the place, a HMI server (ignition) sitting in completely different building with a 5GHz radio link to the client side (where the PLC sits), and with the prosoft radio (for remote I/O) installed to the switch. We even had couple of loops in our network which took a hell of a time to figure out.
The I/Os are setup for unicast connections.

Understanding that typically wireless and produce/consume don’t play nice together, one thing you can try is staggering the RPI settings. As it is now, all of the produced tags are getting consumed at roughly the same time and there might be too much data trying to get through at once. If you take the tags with the least amount of data (discrete tags if you have them) and shorten the RPI settings to say 500ms, then 525, 550, etc… then not all of the data will be coming back at the same time freeing up bandwidth. Look at it this way, if you have a two lane highway and 500 cars that need to go from point A to point B, if all 500 cars got on the highway at the same time you’d have a traffic jam. However if you let groups of only 25 cars (per group) on every 30 seconds traffic would be smoother.
This is just a “shot in the dark” suggestion but I’ve been testing some new high speed modems and when I stagger the RPI settings in the test equipment I’m using (I’m testing Produce/Consume using ControlLogix) I get far fewer connection resets compared to when I have all the RPI’s at the same rate.
One other thing to consider trying is fixing the radios RF speed and fixing the path back to the Access Point (master). In both cases, the radios are using the bit error rate to determine health of the connection and if the bit error rate gets to high the radios re-negotiate the connection path and rate. This process will cause a gap in the communications which will cause dropped connections. If the radios are set with a fixed RF speed and a defined path to the AP then they (typically) won’t drop the connection. Normally I recommend leaving the RF speed at “auto” but from time to time I do come across applications where they run better fixed. Try the slowest speed first and if it gets better try increasing the speed until it starts getting worse a gain (find the sweet spot).
Keep in mind that if the application cannot tolerate any dropped connections then wireless isn’t going to work. No matter what you do you will lose packets with wireless and statistically that will eventually lead to a dropped connection.

Thanks very much for your insight. I will test this theory out with different RPI settings. Fortunately, the process is not very demanding and hence I have room to play with the settings a little bit.
Could you please provide an example on what you mean by 'RF speed'? Is it the Tx/Rx rate that you are speaking of? I'll try to check if it is configurable in the prosoft radios.