Small-ish Waste Water plant comm problems

This system is a 1769-L32E PLC/SCADA, with three 1794-AENT flex bus remote I/O stations, over Ethernet. This system has been in place for 10 (or so) years. Earlier this year (March), we had problems with blower(s) motors just dropping out, for a few seconds (<5 secs), then re-starting again. I thought I found the problem, with a Ethernet switch. I replaced the switch, and either replaced patch cables or re-terminated the ends to get all new/clean connections for the Ethernet cables. But that did not fix the problem. I 'fixed' the problem by setting the 1794-OA16/A module outputs to remain in their last state, upon comm loss.

Now another 1794-AENT and Flex I/O station (part of this same system) is showing the same output dropout problems. I would sure like to hear about troubleshooting methods, possible areas/devices to check out, and past experiences with this problem, to help me find the solution. I would like to add programming to monitor the comms status, but I'm not sure what is the best way to do that. Some advice on comms monitoring would be great too.

Thank You!

What about the SCADA application running on the same subnet as the remote I/O?

HMI(s) or Distributed?

Any new additions or development/editing?

Older controllers will eventually have a hard time controlling I/O on networks shared with broadcast type communications applications such as HMI/SCADA especially when the supervisory systems are further developed and/or updated.

dmargineau said:

What about the SCADA application running on the same subnet as the remote I/O?

HMI(s) or Distributed?


Any new additions or development/editing?


Older controllers will eventually have a hard time controlling I/O on networks shared with broadcast type communications applications such as HMI/SCADA especially when the supervisory systems are further developed and/or updated.

Click to expand...

HMI OR Distributed
** Not known. Surely something to check on.

Any new additions or development/editing?
** Nothing new, that I am aware of, or was informed of.

Older controllers will eventually have a hard time controlling I/O on networks shared with broadcast type communications applications such as HMI/SCADA especially when the supervisory systems are further developed and/or updated.
** Again, something I will check on. I 'think' this is all on the same sub-net. I will find out.

Just something to keep in the back of your mind.

The older the device (and I mean anything electronic) it's tolerance to "Heat" and "Cold" will shrink!

I have experienced similar issues with non-segregated (I/O vs. SCADA) systems which had been further developed or updated with newer HMI terminals.

Monitoring remote I/O status is useful for minimizing 'damages' when connections are 'lost', however, you have an issue that needs to be solved monitoring implemented or not.

The Ethernet media is obviously the first culprit; switches and RJ45 connectors do go bad; Cat.5/6 cable fails too.

With that being said, since you are experiencing a progressive connectivity issue, SCADA could be a culprit and even the L32E CPU might just show its age, especially when significantly 'loaded' for ten or more years.

Type the IP of the PLC into a web browser and go through the diagnostic info.


Attached is based on an up-time of: 87 days, 12h:22m:54s

If the application is tied to a larger corporate network (or even a small corporate network) I'd be looking at what changed in March with regards to the overall network. While changing the subnet mask to isolate the IP addresses could help and should be done regardless, in the grand scheme of things network traffic is network traffic and it gets more complicated and busy every day. Ideally you want your SCADA network completely separate from the corporate network but that's not always feasible so the next best thing would be a good set of managed switches or at least one good managed switch between the corporate and SCADA networks.
I guess where I'm going with this is it could be nothing more than increased traffic on a network that might have already been near capacity. In my past life I dealt with wireless Ethernet (both WiFi and much slower FHSS radios) and one of the common calls would be pretty much just what you are describing, a system that's been running for some time starting to fail without any hardware failures. It was not uncommon that the cause of that call was increased traffic on a system that wasn't managing it very well. Network traffic of today is very different than network traffic from ten years ago. Just think of what you can do today via Ethernet compared to what you could (or couldn't) do back then and how much bandwidth requirements have changed.