1769-L36ERM Communication Issues

[bradal]

Hello,

We have been having persistent communication failures on our network and recently upgraded to the 36ERM to support additional CIP connections.

Currently I am logging a comm failure with a GSV for my first remote I/O module status, logging when it isn't a value of 4 which means running. When I detect a comm failure, the status will change to a 3 and comms will be down for an extended period of time. Communications will rectify themselves 30-90 seconds later but by that time my remote I/O has failed and a number of other issues are triggered.

I have not been able to determine 100% if the issue is network hardware, server connection software polling data or another PLC configuration issue. We replace all control panel ethernet switches except our main Meraki router and switch with no change in the problem. Troubleshot cables, replaced a few to make sure, even pull new runs to the Remote I/O with shielded Cat6.

What I am needing to know, is there some specific GSV values I can get from the controller that will tell me the number if CIP connections are active at the moment of failure or possibly any other items that you may suggest logging.

Any other suggestions/questions welcome. We have 5 racks of remote I/O with a wide range of discrete and analogs.

 

[dmroeder]

I would open the PLC's web page in your favorite web browser and start looking at statistics. Specifically:

• Diagnostics > Application Connections
• Diagnostics > Ethernet Statistics

Screenshots would be great. I'd be looking for missed/rejected packets, FSC/Alignment errors, etc. The only acceptable values for these statics is zero.

 

[bradal]

Screens

Do these help, the one screen for connections is too long. Currently no missed Rx packets.

 

[dmroeder]

Very helpful. What stands out to me is your 3rd pic, you have a lot of rejected and missed packets. That is not good. The first thing I would start doing is looking at baud rate configuration for your devices. I was expecting to see missed/rejected packets on image 2, but that all looks pretty clean, unless I'm not seeing every device.

Do you have a device/network layout, showing what these devices are and how they are all connected?

 

[bradal]

Would those stats you mention be affected by a device being turned off for a period of time. We have two WEG VFD's that are powered down for repairs.

 

[dmroeder]

Would those stats you mention be affected by a device being turned off for a period of time. We have two WEG VFD's that are powered down for repairs.

Hmm, that's a good question. I'll try to test this on a machine later today.

 

[bradal]

See link below, file too big to attach. A lot of devices but if you look at first page, center of page, CP-01, and all devices on the 10.10.57.x VLAN.

http://www.filedropper.com/baytownnetworkdiagramarchitecturerev2b-shareable_1

 

[bradal]

Am I correct in assuming that the PLC is limited in CIP connections (64) and on the diagnostics page, the maximum total observed would be of concern?? Or are there different classes of CIP connections, some counting against total limit and some not?

Currently it states max total observed of 74.....

 

[dmroeder]

The 1769-L36ERM can handle 48 nodes, which means 48 devices in the I/O tree.

Older controllers went by connection count, which made it hard to predict capacity because some devices can make several connections. Modern controllers now go by node count.

 

[bradal]

Just as a quick note. The reason I am concerned about connection count is that we have developed our own in house operational management software. On this software we have also integrated a "SCADA" portion that communicates though an INGEAR driver. Creating multiple connections. Could this be a failure source?? Too many connections?

 

[dmroeder]

What device is 10.10.60.10 (I haven't looked at your network layout yet)? I'm guessing INGEAR, it appears to have the most connections and are class 3. Are you currently only have a comm failure with one particular device?

My experience is that mis-configured baud rates usually cause the most problems, especially so if only one device is doing it. I have found that leaving devices at autobaud is generally gets the best result. While I can't yet rule out INGEAR, I doubt that it is the the problem, it will have the lowest priority and your CPU utilization isn't high.

Pay attention to the device that is giving your trouble, how it is connected to the PLC (multiple switches) and it's baud rate settings. One thing we've ran into is multiple VFD's connected to a switch, all set to auto negotiate except for one, the one causes trouble with the rest.

 

[bradal]

It seems that the failure is absolute across the board, as in, nothing communicates with the PLC. It may actually be more devices even outside the 57 VLAN, but I can log all device modules statuses (all remote I/O, analog cards, VFD's), and they are fail at the same time. This is where my concern for connections is coming from. What conditions could cause the ethernet port to reject any/all devices. I have not been able to ping a separate devices when this happens (never been connected when it happens) to see if it is just PLC traffic or all traffic that collapses. We are adding some logging/sniffing to see what we can find.

10.10.60.10 is the main server, yes, INGEAR is running there.

 

[bradal]

What about this? This is a separate PLC on the network that we hand some information back and forth. This is it's web interface diagnostics.

 

[dmroeder]

Ooh, those missed packets aren't good. It is possible that this is causing an issue with the PLC, eventually dropping it's connection, which would mean that all devices would stop communicating to the PLC.

I won't say baud rate is the only cause of missed/rejected packets, but it is by far the most common in my experience, that is why I keep bringing it up. I would double check this on both PLC's. Again, autobaud is almost always the correct setting.

 

[bradal]

As part of troubleshooting, a while ago I started setting devices away from auto negotiated and to 100mbps and full duplex. At one point they were all auto.

How would PLC react to being set 100/Full forced, if VFD etc. was auto? Does anything even really talk in half duplex unless set that way?