1734-AENT Loss of Communication

I have a network with 17 1734-AENTs. One AENT losses communication regularly.

The symptoms are

AENT
Module Status - Green
Network Activity - Rapidly flashing green
Network Status - Flashing red
Point Bus Status - Flashing red

IO
Module Status - Green
Network Status - Flashing Red

I have exhausted all my normal fixes without correcting the problem.

Replaced the ethernet cable to the first switch.
Set the RPI to 250ms (it is a VERY slow system)
Cable is run in a dedicated conduit and not near any AC.
Replaced the module.
Swapped a "good" module that has never failed with the "bad" one that fails. Now the "bad" one works fine in its new location and the "good" one is failing.
I do not believe it is an IP conflict because whenever the node goes down I cannot ping it.

Cycling power on the AENT always fixes the problem.

Does anyone have any other ideas on things to check before I connect with Wireshark?

Also, are there any recommended videos or blogs that help explain what to look for in Wireshark when trouble-shooting an AB network? I am a novice when it comes to Wireshark.

Power surges might be causing it?
Can you replace the power supply that is supplying the affected module?

I know when you add a new adapter in Logix it will by default set the scan time to 20ms. That's a bit aggressive if your network is not high speed and has un-managed switches. I always slow it to 50 to 100ms. Unless there is something high speed going on. If that's the case then we setup QOS to give that unit high priority.

Have you checked the PLC Ethernet module web page statistics?
What is the baud rate set for in your AENT? It should be Auto Negotiate, unless you have a very good reason to set it to something else (like DLR).

How many modules are in the chassis? What types?

Network Status - Flashing red
Point Bus Status - Flashing red

Click to expand...

Those two together suggest that the I/O connections to the POINT modules from the controller have timed out. You can probably simulate that condition by simply unplugging the PLC from the network.

I do not believe it is an IP conflict because whenever the node goes down I cannot ping it.

Click to expand...

If it were an IP conflict (real or falsely detected) then the 1734-AENTR's Network Status LED would be solid red.

But if it were a short-term I/O timeout caused by noise (or loss of connectivity to the ControlLogix) then the IP stack should still be running: you should be able to PING, and HTTP, and browse with RSLinx.

Can you describe the physical path between the ControlLogix and the affected 1734-AENT ? Is it the only malfunctioning device connected to the switch it uses to connect to the rest of the network ?

When you are diagnosing with PING after a failure, what switch are you connected to and what's the physical route to the 1734-AENT ?

What make/model of switches do you use ?

Cycling power on the AENT always fixes the problem.

Click to expand...

Do you cycle power to the 1734-AENT itself only, or to the entire enclosure it's installed in ? I'm probing whether the switch it is connected to also gets power-cycled.

Your description of the Network Activity light staying flashing suggests that the physical connection is OK, but the inability to PING (or TCPING, or HTTP, or RSLinx, I assume) suggests that access to that Ethernet port from the rest of the network is blocked or unavailable.

All, sorry for the slow reply. I had to wait for another failure to test some things.

tdoa - Thanks for the suggestion on the power supply. I did change it but the problem did not go away.

dmroeder - Everyone is set to auto-negotiate. I need to look at the statistics pages. I have in the past but not in detail.

JeremyM - There are only two modules. One IB4 in the first slot and one OB4 in the second slot.

Ken Roach - You are correct. I can simulate the Network and Point Bus status lights by unplugging the cable.

Thanks for the "solid red" tip on the IP conflict.

Once this failure occurs I cannot ping, access the webpage, or see the AENT in RSLink. I tried over the network and connected directly to the device. In both cases I get nothing. It's like the AENT itself shut down the port.

When I cycle power I am cycling power to the AENT only. There is not a local switch in the panel, just the one connection for the AENT.

As for the network I cannot provide any details at this time. The system is in multiple buildings so I was made to use the corporate network (NEVER AGAIN!). I will see if I can get some details from IT.

I can say that this is the only AENT connected to the network in that building.



My next step will be to connect a switch with a mirrored port and run wireshark. The hope is I can capture the events around the fault.

>It's like the AENT itself shut down the port.

Click to expand...

The fact that you can't PING or otherwise get a reaction from the 1734-AENT while literally directly connected to it certainly supports that conclusion. But the activity on the Network Status LED suggests that it's receiving and transmitting. I'm not sure what layer of physical/transport/application protocol traffic that LED actually indicates.

It's not impossible that the IP stack in the 1734-AENT is crashing. It's a headless Windows CE box, which is pretty darn stable.

I agree that the next step is to try to capture the failure as it happens, and Wireshark connected to a mirrored port is the most common tool.

>I was made to use the corporate network [...] this is the only AENT connected to the network in that building.

Click to expand...

I worked on a system a few years ago that had a "DHCP enforcer" utility deployed. It would find static IP address devices and somehow compel them to enable DHCP and renew. I never quite figured out how it worked, but once the IT department guy said "oh, whoops, that's not supposed to be on your subnet" the problem went away.

A change to the IP address would be consistent with the Network Status LED indicating traffic, but being unable to PING/HTTP/RSLinx connect to the device.

What IP address assignment method are you using: DHCP, Static IP, or thumbwheel (192.168.1.xxx) ?

A Wireshark capture has the best chance of detecting that sort of problem. A direct connection during the "shutdown" mode would see DHCP requests or ARP packets. I like to use a USB/Ethernet dongle and remove all of the networks services from it so my computer doesn't try to transmit or participate in anything, so the device is purely a promiscuous packet capture device.

Similar problem found cause

Hi. I know it is an old thread, but feel I want to share the solution to a similar problem with the 1734-AENT. I use a V-lan on our corporate network for all automation equipment. I had a problem where two 1734-AENT's where failing and the only way to sort it was to power cycle them. I found that replacing them with 1734-AENTR modules worked but still wanted to find the cause. I left one single port in the machine configured on the PLC. One of the guys in the IT department made a comment of them having a similar problem with T&A fingerprint scanners. Ended up they disabled POE on the ports of those devices sorted the problem. So we tried the same with the port where these two 1734-AENT's where on. Problem solved. Hope it helps someone else with the same issue.

It's always IT faults. Good to know. Thank u!