Intermittent Radio Problems with Calamp Vipers

Hello everyone. I am trying to help solve a radio/comms issue involving several remote stations, using licensed spectrum (153 MHz) Calamp Viper/Dataradio modules and AB PLCs.

We have a somewhat odd setup, in that we have a repeater station between one of our remote locations and the main office, due to layout of the terrain.

The main office needs to monitor values at each remote pump station (5-10 min max interval), and occasionally update PLC values at the remote locations (weekly or less).

We need to receive fairly frequent/consistent updates at the main office (on the order of 5-10 minute intervals max).

Here's a diagram of our setup: https://snag.gy/As0pCJ.jpg


The problem:

Most of the time everything works great and data is updated at the main office within a few seconds (i.e. less than 60).

However, approximately once or twice daily, we see comm failures for the Repeater and TC stations. These comm fails occur within minutes of each other, and can last for 1-2 hours. Then the Repeater and TC failures resolve within minutes of each other, and everything continues to work fine for several hours/days, with fast updates on the order of a few seconds.

During these failures, the DC station never loses comms, and its values continue to update at the main office.

We also see somewhat more frequent comm fails between the Repeater and Main Office which don't correspond to failures at TC. However, these failures are short and usually resolve within 5-10 minutes.

During the comm failures, I can still Ping the Repeater and TC radios from the main office.

We're planning to do a radio test this week to determine if it's a Radio Path issue or a networking issue.

Any insight and suggestions for diagnostics would be greatly appreciated.

First, describe the devices at each location.

Is there a "master PLC" at the main office, or is that the SCADA master computer managing the network ?

Are these CompactLogix, SLC, MicroLogix, or other controllers ?

Is this a TCP/IP network, or DF1, or some other protocol ? You mentioned PING, which suggests it's TCP/IP. But I think the CalAmp also has a serial comms port, so I'd like to understand for sure what you're doing with them.

How, exactly, do you detect and log communication failures ?

I have worked with the 450Mhz Vipers a little bit. I have had some odd issues too, but when they are all working well and I don't change any settings, I don't have problems.

Do your radios have a webpage that shows the RSSI?

What PLCs (if any) are you using?

I had one issue caused (I think) by an old series ML1100 sending out gratuitous ARP requests that were not being filtered. It only occasionally caused errors but then I learned how to filter out that traffic with a setting in the radio (I don't recall the exact term).

I also had one station that gave me fits for several weeks and seemed to disrupt the whole network. I simply replaced that radio and solved the problem. I don't know what if anything is wrong with the one I took out, but it did have older firmware in it.

I am also curious to know whether you might have better success if you were to poll more frequently so that the Ethernet connections would be maintained and not have to be re-established after a long period of not being used.

And there is always the possibility of interference.

At those distances, especially with line of sight, you could use unlicensed 900Mhz radios. We use Phoenix TWE line for all new projects where possible since they have proven to be very solid performers, include a repeater feature, and allow us to add I/O to the radio module and negate the need for a remote PLC in some instances.

Hello and thank you both for your replies!

I was trying to keep my original question short to avoid TLDR issues. Sorry I left out important bits!

PLCs:

Main office: SLC5/05 (Ethernet and Serial to radio)
WG station: SLC5/05 (Ethernet to radio)
TC station: SLC5/05 (Ethernet to radio)
Repeater: Micrologix (Ethernet to radio)
DC station: Micrologix (serial port to radio, DF1)

The PLCs initiate data transfers using the MSG instruction. From what I can tell, the Main Office is supposed to update watchdog timer values in the other PLCs, which are then in turn triggered to read and write data to the Main Office PLC. However, I'm not sure this is implemented properly, and may be allowing more than one PLC to attempt a read/write operation at the same time. (I did not write the PLC programs, so I'm trying to follow the somewhat poorly commented logic ).

It appears the polling happens at about 10-15 second intervals.

Comm failures are detected by watchdog values in the Main Office PLC failing to change for more than a set time. These values are updated by the remote PLCs after apparently being requested by the Main Office PLC.

I am told by the radio expert at Calamp that we should not be using Bridge mode for this setup, and should switch to Router mode and utilize TCP Proxy, to avoid unwanted multicast/broadcast traffic across the radios. This makes sense from a networking standpoint. The original designers did not want (or did not know how ) to configure Router mode.

He also stated that by using Router mode, we would not need a PLC at the Repeater location, as the radio could handle this easily, eliminating what I consider to be unnecessary complexity.

The radio config pages show RSSI, but I am not clear on how to interpret the values.

I would be curious: how could one radio disrupt an entire network? Is it by flooding the airwaves with transmissions?

FYI I have also looked up all FCC licensed stations on this spectrum in the area (and nationwide licenses that apply) and it doesn't look too crowded, so hopefully interference isn't the issue. I don't have a spectrum analyzer unfortunately, but they may allow me to purchase one if I can eliminate the other possible causes.

Sadly it's probably not an option to switch to different technology at this point, unless we absolutely can't make these work reliably. They had invested quite a lot in this setup before I was asked to help, and are going to be very hesitant to change without really good reasons.

Thanks again for your advice and suggestions!

The use of both Ethernet and serial is a little unusual.

Is there an Ethernet switch between the Main Office SLC-5/05 and the radio ? I presume there is, since there is likely a SCADA system attached as well.

If that's a managed switch, set it up with a "mirror" port that will let you capture and analyze the Ethernet traffic that's going out onto the radios.

That's my very first step whenever I'm working on a system like this. If the radio is haphazardly connected to an enterprise network, there's no telling how much data is going across the radio channel, or what sort of connections some other software might be making to your controllers in the field.

Being inexperienced, I was not aware it was uncommon to run both ports simultaneously. While I didn't design or implement this existing setup, it may be necessary to present arguments for modifying it. Do you see specific problems with the utilization of both serial and Ethernet communications through the same radio?

Yes, there is an Ethernet switch (unfortunately it's unmanaged), and along with the SCADA there are at least 5 other AB PLCs operating within the same subnet and communicating with the main PLC and SCADA.

Could I evaluate this traffic issue with Wireshark? For example, would high volume multicast/broadcast traffic within the Main Office network be an indicator that the radios are attempting to push all this traffic? Other indicators?

What about implementing a managed switch and enabling IGMP snooping?

The Calamp radio tech stated that AB PLCs generate frequent multi/broadcast packets which can clog low-bandwidth links, and that the only solution is to enable Router mode and TCP proxy on the radios to filter such traffic.

You may have hit upon the problem with your use of the word "haphazard".


Thanks again!

I would have the main office PLC initiate all the messaging. This would ensure that the traffic is controlled. You can still exchange heartbeat values in both directions with all the slaves and take appropriate action if one PLC detects that the other PLC is no longer active and running.

I believe it was the act of enabling TCP proxy that helped me with the rogue Micrologix sending ARP requests.

Mine (not mine, but a network I inherited and expanded) are definitely in router mode. This along with some other settings keeps unnecessary traffic off the network.

I am polling 7 Vipers with reads and writes of about 100 words each way to five of them, 16 words each way to two of them, and making it through all of the messages in 15 to 30 seconds. I did add a couple of pauses, one at the beginning of the message cycle and one about halfway through. The pauses are about 5 seconds each. Having some dead time on the air waves leaves a little room for me to get online with a remote radio to view or edit settings. This is another reason to have one master in control of all the traffic. When I needed more time to do things to a remote device, I increased my dead time to 15 seconds simply by changing a timer preset.

Another thing I remember now that bit me early on. We were brand new to these radios and wanted to test a new one to make sure we knew the right settings and prove our antenna location. I put the network into "auto scan" mode to pick up the new radio for our test. When we were done and removed that radio from the network, all the others had already learned about it and a couple of them decided to "hop" (repeat) through it. I had put the radios back into manual scan mode and thought I had them reconfigured without it in their neighbor tables, but apparently the Vipers have a memory like my own....kinda fuzzy and comes and goes... It was a little strange that it was several hours later when the messaging started to bomb out.

I had to put the radios (global settings) back into auto scan mode multiple times to get them to finally forget about that new missing radio. Then the next day, one of them remembered it again. Rather than calling for tech support (it was after hours by then), I hunkered down and learned the hard way how to get things working right again.

Also, the RSSI numbers I looked for was anything above about -80. At one point I had a fouled up antenna and a radio would occasionally error on a message with an RSSI of -96. When I got a new better antenna and pointed in the right direction (yagi) I think all my RSSI values are now above -78.

Thanks Okie. Am I understanding you correctly: are you able to access the remote PLCs via RSLogix through the radios? I didn't know they could push enough bandwidth to allow going Online with RSLogix. (Ours are set at 16kbit speed.)

An aside: today we are having rain and heavy cloud cover, and it seems this always improves the radio function (i.e. few/no failures), especially for the TC station, which is basically behind a hill from the Main Office.

Could this be pointing to signal path issues?

It sounds like I need to check the RSSI value on each radio to determine signal strength.

ryangriggs said:

Thanks Okie. Am I understanding you correctly: are you able to access the remote PLCs via RSLogix through the radios? I didn't know they could push enough bandwidth to allow going Online with RSLogix. (Ours are set at 16kbit speed.)

Click to expand...

Yes and no. It is intermittent whether I can connect to a remote PLC. That is one thing I really like about the Phoenix TWE line. It is pretty easy for me to go online and even perform online editing even through a single repeater to a remote PLC connected to the ethernet port using the TWE radios. It is not nearly as fast as being directly connected, but I have even performed a complete upload over the radio of a small program running a pump station over the air through a repeater and it only took about 3 minutes.

With the Vipers, it was much slower, but of course at half the frequency, the amount of data that can be packed onto those waves is even less. And your radios are much lower in frequency which may make remote access of the radio web pages and PLCs downright impossible.

An aside: today we are having rain and heavy cloud cover, and it seems this always improves the radio function (i.e. few/no failures), especially for the TC station, which is basically behind a hill from the Main Office.

Could this be pointing to signal path issues?

It sounds like I need to check the RSSI value on each radio to determine signal strength.

Click to expand...

If your antennae are not line of sight, you are likely to have intermittent problems, however we have some UHF radios that "talk" through (or around) hills and trees and buildings quite well. Depending on distance, the 900 MHz radios can sometimes work well too.

We use Google Earth (because it is free) to create a map of the radios and draw lines between antenna points to establish how high in the air our antennae need to be mounted to try to get line of sight. Much of Oklahoma is pretty flat, but we do have some hills in the eastern parts of the state where we have some UHF radios and line of sight isn't possible.

I have been told (and seen in action) that lower frequencies tend to work over hills and in tough terrain much better than higher ones.

I would definitely look at the RSSI values. I am not sure if you can see them all from one radio unless you're in router mode. But that is how I do it. I log into the "master" radio and look at the neighbor table. The RSSI values will update periodically and I can clear them and ensure I am looking at fresh numbers when they repopulate.

You are using a different radio, so I can't guarantee the same features exist. I should also point out that I am self-taught and relatively new to the radio telemetry world (1.5 years) so what I am telling you is not from an expert point of view.

PS: I told you wrong earlier, we have 7 radios total, one we call master and 6 slave PLCs. We have multiple Phoenix Radio networks at this site and ended up putting one of them we'd planned for a Viper on a Phoenix radio where we had line of sight to a Phoenix Master and there was a 100' tall ridge blocking airwave access to the office Viper.

Here's a screenshot of the Main Office Viper RSSI page at the site where I was tasked with adding two radios, relocating three of them and learning how to make it all work.

Thanks OkiePC, that's very helpful.

I was just poking around in some of the radios' settings and noticed that 1) our Main Office and TC station radios do not list any Neighbors in the Neighbors table. They are also configured to Manual discovery.

On the Main Office radio, it shows a RSSI value on the Status page associated with the MAC address of the Repeater station.

On the TC radio, it shows a RSSI value on the status page associated with the MAC address of the Main Office radio.

To me, it sure looks like the neighbors configuration is not correct. I'm wondering if switching to Router mode and ensuring all the neighbors are configured properly may resolve my issue...

It's really hard to tell if changes improve the problem, since the loss of connection only happens occasionally.

We have using Viper radios for quite some time. I always use router mode and set them up manually (disable Neighbor Discovery). I know its easier to use auto discovery but it messes things up with the rest of the sites. We use the scheme on the attachment to make things easier when setting up the radios.

With that said, I have had the same problem you guys are describing. I have one site that quits communicating for minutes or hours' then it comes back on its own and works fine then down again. This particular site is set up as a polling master. It communicates with the remote sites perfectly. The only place it gets lost is the HMI. I have several polling masters in this particular system which is not a problem with Viper radios because they play nice with each other. I have used simultaneous serial and ethernet many times so that really isnt unusual because only one poll is active at a time no matter how many sites you have.

I have gone over all of the settings on Radios and PLCs (Siemens S7 1200) and nothing is sticking out as being the cause of the communication errors. Every site on the system is setup exactly the same way. I gotta figure out what is going on; no choice in the matter. Almost ready to call in CalAmp guys.

Hi bkinard. I think we have narrowed it down to signal path issues. If you look at our initial diagram, "TC Station" is behind a hill, and the signal quality back to the main office is very poor. It appears that the "Repeater" station is trying to communicate through the TC station instead of directly back to the main office, and thus causing us to lose both these stations.


Is there a way to force a radio to *only* talk to a specific peer? I realize the messages are broadcast, but can they be configured to ignore all traffic which does not originate from a specific MAC? Note that we're stuck using Bridged mode at this point, since some of our equipment has fixed IP addresses that we can't change without buying the programming software and reprogramming everything.

Good luck with your setup!

One thing I saw is that your RSSI levels appear to be fine. I had a VHF customer who had a tornado rip through his system and the system continued to work with the antennas on the ground as far as 10 miles away with a hill in between. In my experience, anything from -78 or higher is a good path. I generally shoot for mid 70s or better.

One thing you may want to check is voltage. Unplug the RF cable from the radio and check for AC or DC voltage between the antenna connector to ground. There should be nothing there. If there is any voltage at all, you will have intermittent problems. Ground the radio chassis. Thats a good habit anyway.

As for getting the radio to only talk to a specific peer, I dont use bridged mode so I cant remember what parts of the Viper software is active and inactive. If the gateway is active, then you may be able to set up that up.

What are your IP addresses at each site? You may still be able to use router mode. The only thing that makes the Viper a radio is an antenna connector and a frequency; from there on out, its a router.

You can contact me directly if you need to.
Brian Kinard
[email protected]
803-924-7435