trying to get SCADAPack to interface with existing serial radio system

Hey guys,

I'm working on a project where the client has an existing water/wastewater system that is running in NI Lookout 6.0 with unlicensed, spread spectrum serial radios for comms. It's a very simple system, basically some wells for water and a few lift stations for wastewater. The idea was that we'd replicate their existing HMI in InTouch, and use a SCADAPack PLC to control everything and get rid of Lookout. Basically, I have a new control panel and a PC workstation, and we simply unplug the radios from the existing PC running Lookout, and plug it into our new SCADAPack and do testing.

For the past few days I've been working on getting comms to work with the existing system; the radios are Digi XTends and there's a single radio at the district office that polls all the other sites; no store and forward or anything fancy, the existing Lookout program polls each site separately.

What's weird is that with the existing Lookout system, each site communicates very well; there's a comms status page that shows what's going on, and within five seconds of plugging the radio into the existing system they're all polling rock solid. But when I plug the existing radio into the SCADAPack, a few sites respond pretty well but the rest are absolutely god awful.

In order to make things as easy to test as possible, I've written a very simple program that can poll the sites manually via a push button in the HMI (here's a link to it). The program is extraordinarily simple, it's just a single MSTR block that is manually energized via a pushbutton in the HMI with a counter for successes and failures. All I'm doing to change from site to site is change the Modbus address (the various remote sites are addressed 1-10, the SCADAPack is address 11). So what's mind-numbingly frustrating is that a few sites will successfully poll all day long, but others just fail over and over. And like I said, when I plug the radio back into the existing system, within five seconds all sites are reporting good comms.

At this point I don't know what else to try, I've triple checked all the SCADAPack settings and compared it to the existing Lookout settings and nothing looks wrong. And if something was wrong in the SCADAPack settings, why would a few sites report back just fine and others not? I've checked the radio wiring and the only conductors there are Rx, Tx, and Gnd. No CTS, RTS, nothing.

Not sure how to try to figure this one out, but I figured someone here might have a suggestion for what to try.

I should have mentioned that all the remote sites are the exactly same Automation Direct RTUs that pack and unpack discrete and analog signals the exact same way, using the exact same addressing. Thus, if I can communicate with one, it would seem that I should be able to communicate with all of them but that’s obviously not happening.

That does seem rather odd. I have had radios before that had issues around starting to transmit before they had a full packet in their receive buffer. Was only an issue with one site with a large payload but since yours are all identical that doesn't seem likely.

I would use a modbus simulator on my laptop and try reading from the remote sites to see if it's a SCADAPack issue or a radio issue. And do the same on the remote end (set up modbus simulator with registers that mimic what you're currently reading).

Once you know which end is the issue we can check a few other things.

Are none of the radios doing any reapeater type functions? You're quite fortunate to get all sites from one base on 900Mhz! I've used rebranded versions of the xtends and their repeater function was a bit laggy.

Saffa said:

I would use a modbus simulator on my laptop and try reading from the remote sites to see if it's a SCADAPack issue or a radio issue. And do the same on the remote end (set up modbus simulator with registers that mimic what you're currently reading).

Click to expand...

That's a great idea. Is there a particular modbus simulator you'd recommend?

Once you know which end is the issue we can check a few other things.

Click to expand...

Yeah that's the frustrating part of this, trying to figure out how to drill down to the root cause. It's fairly bizarre, I talked to my SCADAPack sales rep and sent him my code and he saw nothing wrong with the methodology or settings so it feels like there's something odd going on that's beyond simple configuration. I have learned about the Diagnostics abilities of the SCADAPack and have turned those all on, will see what they say when I'm back on site.

Are none of the radios doing any reapeater type functions? You're quite fortunate to get all sites from one base on 900Mhz! I've used rebranded versions of the xtends and their repeater function was a bit laggy.

Click to expand...

You know I can't answer that for certain, the sites are somewhat far apart so there might be some sort of repeater functionality that I'm not aware of. I actually haven't physically visited every remote site so checking each radio may be worth the time to get a better picture of the radio network; there may be something going on that is causing this oddity that I'm not aware of.

Thanks for the response!

QModMaster is my Modbus simulator of choice for Modbus masters. I don't have any particular recommendations for slave simulators - the ones that pop up on the 1st page of Google seem legit enough.

At first, your issues seem signal-related but you say you aren't changing the radio network. It does seem quite bizarre.

scrambled said:

QModMaster is my Modbus simulator of choice for Modbus masters. I don't have any particular recommendations for slave simulators - the ones that pop up on the 1st page of Google seem legit enough.

Click to expand...

Thanks for the suggestion, I'll definitely give that a shot.

At first, your issues seem signal-related but you say you aren't changing the radio network. It does seem quite bizarre.

Click to expand...

Yeah exactly. What's even more strange is that the sites that I can communicate with are some of the farthest out, and the site that is literally next door is one that we've never had any successful communication with. But as soon as I plug it back into the old system they're all just fine.

I'm hoping that between the Diagnostics registers I've set up in the SCADAPack, and with a modbus simulator, there will be a smoking gun that will help figure this out.

Thanks!

Instead of comparing the serial port settings in the software and the SCADAPack compare the settings in the SCADAPack and the radio modem. The SCADAPack may have a faster turnaround speed (from receive to transmit) than the PC did and the radio modem may not be able to handle a rapid turnaround. Also check if there is a response delay’s in the serial port on the SCADAPack and if there is try increasing it. I’d start with 500mS and work backwards from there.

Firejo said:

Instead of comparing the serial port settings in the software and the SCADAPack compare the settings in the SCADAPack and the radio modem.

Click to expand...

That's a good idea. Earlier this week I attempted to connect into the radio with some config software but it wouldn't work as I expected, and I was a bit hesitant to keep pushing it since I didn't want to break anything and then need to fix existing as well as get my new hardware connected. They do have spare radios now that I think about it, maybe I could connect to one of those to get a better feel for the software.

The SCADAPack may have a faster turnaround speed (from receive to transmit) than the PC did and the radio modem may not be able to handle a rapid turnaround.

Click to expand...

That's an interesting idea. Can you think of a way to see if something like that is occurring? I could ask the radio guys if they've ever had that issue.

Also check if there is a response delay’s in the serial port on the SCADAPack and if there is try increasing it. I’d start with 500mS and work backwards from there.

Click to expand...

I've looked around for settings like that but I haven't come across anything. I'll keep looking though.

Thanks for the suggestions!

Quick update: I've done a fair bit of reading and research on the SCADAPack and Digi XTend serial connections, and I'm thinking that my issue could be that I'm trying to use a Tx/Rx/SG connection without any flow control. Both the SCADAPack and XTend manuals show that for DTE to DCE (or SCADAPAck to XTend) connections, all the lines are in use to control flow. Not entirely sure why the existing system can get away with only Tx/Rx/SG, but FireJo mentioned that the turnaround speed may be a lot faster in the SCADAPack and I'm thinking that could be it. The Lookout's serial connection actually goes through a USB to serial converter, so that may be slowing things down sufficiently to keep conflicts from happening. And, I have no idea what the Lookout driver is doing as far as turnaround speed so there's that as well.

Monday I'm going to try to connect the SCADAPack and XTend with a full serial complement and see if that does anything to help. Maybe the sites that I can communicate with are going through a repeater, and the delay helps with keeping collisions from happening?

Will report back Monday. Thanks for all the help guys.

As Firejo mentioned, I would look at the RTS ON / RTS OFF delays in the SCADAPack. On a wired network you can usually get away with a 5mS RTS ON delay (probably the default setting), the problem with a radio network is that the radio can't switch from receive mode to transmit mode that fast. Having too short of an RTS ON delay on a radio network will cause the PLC to start transmitting before the radio can switch to transmit mode, thereby causing a loss of data in the transmission.

To give you a starting point on your settings, I generally use an RTS ON delay of 100-150mS and an RTS OFF delay of 20-50mS.

On the AutomationDirect PLCs the RTS and CTS aren't being used because they already have the RTS ON/OFF delays set correctly. It's a Master/Slave network, so only one PLC should be transmitting at any time.

Tark said:

As Firejo mentioned, I would look at the RTS ON / RTS OFF delays in the SCADAPack

Click to expand...

Yeah, I spent most of Friday and Saturday looking all over the place and for serial Modbus, I can't find anywhere that would allow adjustment of those delays. It seems like there are a LOT of adjustments like that for DNP, but not for Modbus.

But yeah, I think that very well could be the issue, and I'm gonna head down today and try a full serial wiring with RTS and CTS and see if that helps.

Thanks!

You briefly mentioned Modbus, if this is Modbus RTU then you’ve got a different issue that is related to timing but not the same way as I first thought.
Most serial communications protocols have an “end of message” character in the data packet so that the receiving device knows when the message is ending. Modbus RTU (not to be confused with ASCII) doesn’t use that end of message character but instead uses a gap in the data stream. So when the receiving device sees a gap of more than 3.5 bytes (so 4 then) it treats it as the end of message and stops paying attention. The problem is that spread spectrum radios are going to introduce a gap in the data stream which will be treated as the end of message. The old licensed radios typically don’t have this issue which would be why the old radios worked and the new ones don’t. If this is the issue you’ll need to see if you can use a different protocol or different radios that have a Modbus RTU mode. You can try playing with the timing but it probably won’t work (very well anyway).

Firejo said:

You briefly mentioned Modbus, if this is Modbus RTU then you’ve got a different issue that is related to timing but not the same way as I first thought.

Click to expand...

Ah, interesting. Yes, this is Modbus RTU.

Most serial communications protocols have an “end of message” character in the data packet so that the receiving device knows when the message is ending. Modbus RTU (not to be confused with ASCII) doesn’t use that end of message character but instead uses a gap in the data stream. So when the receiving device sees a gap of more than 3.5 bytes (so 4 then) it treats it as the end of message and stops paying attention.

Click to expand...

That was something I had come across in my readings about Modbus recently, thanks for explaining it.

The problem is that spread spectrum radios are going to introduce a gap in the data stream which will be treated as the end of message.

Click to expand...

Very interesting.

The old licensed radios typically don’t have this issue which would be why the old radios worked and the new ones don’t.

Click to expand...

Not sure what you mean here, if you're describing a hypothetical system or my actual system. In case you mean the my actual system, I want to make sure it's known that I haven't replaced any of the radios, they're all existing in place. They're all Digi XTend-PKG, the local "base" and all the remotes and we haven't made any changes to them. There has never been licensed radios in the project, new or existing.

If this is the issue you’ll need to see if you can use a different protocol or different radios that have a Modbus RTU mode.

Click to expand...

I believe these radios are compatible with Modbus RTU (at least according to this Knowledge Base article http://knowledge.digi.com/articles/Knowledge_Base_Article/Modbus-Protocol) but it looks like they somewhat generally describe some caveats.

Unfortunately when I contacted Digi to get some help with trying to interface with the radios via their configuration software, they stonewalled me and said that I would need an annual "support package" that costs $3700 so I'm not sure how to try to verify the radio parameters; there are weird issues with their software and it's not working as they describe in their user manual.

You can try playing with the timing but it probably won’t work (very well anyway).

Click to expand...

I'm wondering if that's what the former integrator did; maybe he got it to work with a slower system (PC driver, the RS232 to USB converter, etc) and now that we're going straight to a new SCADAPack the radios are now no longer able to deal with it.

I do know that with qModMaster, I was able to query all the sites just fine with my laptop, so it *has* to be something with the SCADAPack and how it is interacting with the radios.

Thanks!

So my reference to licensed radios was because I was confusing your thread with a different one (oops).
I read the tech note that Digi has and I can see what they are trying to do. They have you run the RF speed between the radios faster so that the radios have time to get all of the data across before the serial port needs to send it. The problem is that is assuming that the connection between the radios is solid and that there isn’t a lot of retransmission taking place. However (unfortunately) it’s not hard to interfere with a Digi X-tend radio (formerly Microhard) and if you put them into a noisy environment they fall apart pretty fast. I’m not saying that this is what you are seeing but one of the first things you’d lose is the ability to send Modbus RTU. Having said that, are you running at 19.2Kbaud? Apparently that won’t work because the radio can’t run twice as fast as the ports so maybe the fix is to slow the baud rate down. That would give the radios more time to get all the needed data through but it won’t deal with lost packets (which you will get in noisy environments).

Firejo said:

So my reference to licensed radios was because I was confusing your thread with a different one (oops).

Click to expand...

Oh good, I'm glad I didn't make my previous posts confusing, haha.

I read the tech note that Digi has and I can see what they are trying to do. They have you run the RF speed between the radios faster so that the radios have time to get all of the data across before the serial port needs to send it.

Click to expand...

Ah ok, thanks for explaining that.

Having said that, are you running at 19.2Kbaud?

Click to expand...

No, the existing Lookout PC is running at 9600 baud, and while the Digi config software is acting strangely, I have been able to verify that the existing radio is set to 9600 baud as well, along with 8 data bits, no parity, and 1 stop bit.

Apparently that won’t work because the radio can’t run twice as fast as the ports so maybe the fix is to slow the baud rate down. That would give the radios more time to get all the needed data through but it won’t deal with lost packets (which you will get in noisy environments).

Click to expand...

I could definitely try slowing things down, the only bummer is that means I'll have to visit all the remote sites and do a lot of testing which we didn't budget for. But if it's the best way to move forward, I can do that (but that being said, it seemed like you were suggesting that if we were at 19.2k, not 9.6k).