Rockwell 1734-AENT Point IO module keeps faulting

Hi there,

Long time lurker posting for the first time.

TLDR: Trying to configure a small Point IO drop into an existing system, the point IO keeps faulting after a random amount of time.

Relevant Details

Network Layout: See attachment - Drive Comms.jpg (sorry for the low resolution)
Logix Code: See attachment - PM1_gpr_30_APR_2015.zip

This is a drive system for a small paper machine, all the hardware was installed before my time with the company and I only added a small subroutine to the software and the PointIO modules. The drives used are old Parker 590+ connected to a Link system. The CompactLogix PLC serves as a bridge of some sort to map between the newer Parker TS8000 HMI's and Wago IO modules, and the Link system. My additions to the system are a single PointIO adapter and one 4 channel analog module.


Processor: CompactLogix 1769-L32E Rev. 19.11
Switch: Siemens Scalance X-108 Unmanaged (that is what the CompactLogix and PointIO are connected to)
Remote IO:
1 of 1734-AENT/B - Revision 4.5
1 of 1734-IE4C - Revision 3.4


Preface.

I was tasked with putting a couple of analog signals (pressure readings) on to our historian system. To do so I decided to use an existing PLC (Compact Logix L32E), and add an remote IO drop (1734 modules) for the two pressure readings. Simple enough right? Just buy all the kit, configure the software, and wire everything up.

Ahahaha, no. It took me half a day and a few calls to Rockwell tech support to figure out that a) I needed to download an AOP for the PointIO adapter, and b) the bloody chassis size needed to be set properly!

Anyway, I did eventually get it 'working' and I could read the two pressure tags on the Historian software. All was well and good until I came back the next day.

Problem.

I came back the next day and noticed that the pressure readings were not updating. Loading up RSLogix showed that there was an I/O fault (green light blinking) and some yellow triangles on the Point IO modules. Power cycling the Point IO module got everything back up and running, fault cleared.

Checked it again the next day, same issue. This was a week ago and I am still having the same issues. I am unable to see most of the properties of the AENT module when it is faulted (just comes up with a time out error in Logix), and the only fault code that I can see is 16#0204.

Lights that are flashing on the modules:

* 1734-AENT - Network Status light flashing red
- Point Bus Status light flashing red

* 1734-IE4C - Network Status flashing green



What I have tried so far.

Based on a few conversations with Rockwell support I have tried the following:

* Check CPU Utilization - Utilization at ~100% Woah, that's probably not good. Turned out the attached Parker HMI's (see the diagram) were trying to poll too fast. Added a slight delay (100ms) to each of the HMI's and the utilization dropped to ~50%. Still the same issue with I/O dropping out.

* Try Pinging I/O when faulted - Cannot ping the 1734-AENT module. Also can't right click on any of the modules on RSLinx (it just freezes up).

* Checked the Ethernet cable - Using a cable tester, looks fine.

* Check Switch - Not really sure what to check on an unmanaged switch.

* Checked the Ethernet counters on the web browser - No errors logged.

* Checked driver diagnostics on RSLinx - Noticed that the Point I/O module had a lot of [10060] Connection timed out errors. Also noticed "Can't set port type to 288: no Harmony port for this IP address" show up at the end.

Rockwell has recommended I upgrade the firmware on the CPU and possibly on the PointIO modules as well to see if that fixes the issue. I will try this on Thursday when the machine is down.



Question.

Can anyone see any glaring issues with the software/hardware configuration?

What else should I check/try before upgrading the firmware on the CPU?


My own experience.

I am a graduate electrical engineer (well 2 years out of school now) with a focus in power systems but have recently been working more and more on industrial controls and programming (not something I really studied at school). I have not had any formal training in regards to any of the current platforms but have built up some basic hands on experience on the Allen Bradley PLC's (SLC and 5000 series), Parker SSD VFD's and HMI's, and a very old Alspa DCS system.


Thanks for reading, and apologies if this has already been asked 1000 times.

I used bootp, pretty sure I followed the instruction correctly, and the module was not powered down during the process.

I could try resetting it (888 on the dial, power down right?) and assigning the IP again if you think it's worth a shot.

Did you ever get this solved? I had trouble recently with modules on a 1734-AENT faulting. Everything worked, but after a random amount of time, one of the modules would fault.

Turned out to be a problem with the port configuration. If anything goes wrong with the speed and duplex negotiation on an Ethernet port, the default is to drop back to half-duplex communications. I had one end of the line (can't remember if it was managed switch or 1734-AENT) set specifically to 100 Mbps/full duplex and the other to auto-negotiate. In the end, it dropped to half-duplex and caused random communications faults.

Setting both ends to auto-negotiate cleared everything up. You've probably already got this working, but if not it's worth a check.

There is NEVER a reason do disable negotiation. Period.
http://etherealmind.com/ethernet-autonegotiation-works-why-how-standard-should-be-set/

The only times I disable auto-negotiation is when I'm dealing with a device that doesn't support it, like an old PLC-5E or an old Ethernet/Fiber converter or TCP/IP radio.

That prevents the link negotiation algorithm from periodically dropping the link for diagnostic purposes.

While dropping the link for diagnostic reasons is great for most general purpose IP networks, it's often bad for automation networks that rely on low numbers of retries for connection diagnostics.

With a Logix-family controller and a POINT adapter, I leave auto-negotiation enabled. This is especially true when using unmanaged switches, since they can't be configured for anything but their default functions.

We get plenty of these sorts of posts where the user has a hard-to-diagnose Ethernet problem and admits they used an unmanaged switch because they didn't see the value in being able to diagnose Ethernet problems at the switch level.

Most Ethernet cable testers aren't more useful than the link LED on the port. We have a very sophisticated spectrum tester that we use to certify every Ethernet cable we deploy in the field. It's expensive, it's cumbersome, it's hard to use, and it's worth every penny.

rdrast said:

There is NEVER a reason do disable negotiation. Period.
http://etherealmind.com/ethernet-autonegotiation-works-why-how-standard-should-be-set/

Click to expand...

Interesting.
I have read somewhere in RockwellLand that if you use fixed speed, that both ends need to be set that way (i.e PLC and switch).
And, I was on a very large site where that was the rule...fixed speed...and, if the switch didn't get set right, the PLC ethernet port would report errors until the switch was set to fixed speed.
No "slackers" set this network up...its a national WAN with multiple factories all together.
I figured they must have had a good reason.

jdbrandt said:

Interesting.
I have read somewhere in RockwellLand that if you use fixed speed, that both ends need to be set that way (i.e PLC and switch).
And, I was on a very large site where that was the rule...fixed speed...and, if the switch didn't get set right, the PLC ethernet port would report errors until the switch was set to fixed speed.
No "slackers" set this network up...its a national WAN with multiple factories all together.
I figured they must have had a good reason.

Click to expand...

They didn't. They had a corporate culture that knows better than anyone else. The Auto-negotiation protocols work as they are supposed to. If you have some legacy piece of trash that doesn't support AN, then set that port and that port only (or preferably, throw it out and replace it).

I guess that national WAN with multiple factories will never go to gigabit Ethernet. Bet they have a bunch of GB switches though, that don't work.

Adam13 said:

Did you ever get this solved? I had trouble recently with modules on a 1734-AENT faulting. Everything worked, but after a random amount of time, one of the modules would fault.

Turned out to be a problem with the port configuration. If anything goes wrong with the speed and duplex negotiation on an Ethernet port, the default is to drop back to half-duplex communications. I had one end of the line (can't remember if it was managed switch or 1734-AENT) set specifically to 100 Mbps/full duplex and the other to auto-negotiate. In the end, it dropped to half-duplex and caused random communications faults.

Setting both ends to auto-negotiate cleared everything up. You've probably already got this working, but if not it's worth a check.

Click to expand...

Unfortunately no, the problem still persists.

Since my last post I have tried:

* Updating the controller firmware. Went from v19 to v20 based on a suggestion from Rockwell tech support. Still having the same issue.

* Replacing the 1734-AENT and 1734-IE4C modules. Still having the same issue.


I have yet to try updating the firmware on the 1734-AENT (currently on 4.5) which tech support suggested.

Both the 1769-L32E and the 1734-AENT are set to auto-negotiate on the port settings. Is there a way to check if the auto-negotiate has failed for some reason? Is there also a way to check/track if the module is dropping to half duplex?

The switches used are all unmanaged Siemens X-108's. Not sure if it's worth replacing this as everything else on the switch seems to be communicating fine (no Ethernet errors on when I browse the L32E on my browser).

You don't have someone plugging something in with a duplicate IP, do you?

How many switches are between the L32E and the AENT? Can you try eliminating some, even temporarily? Run a wire on the ground if need be.

What terminations are on the Ethernet cables? Are there just 8P8C plugs crimped onto the cables? Or are you using decent patch cables, and punch down blocks?

Are your Ethernet cables 600V rated?

Can you determine if the device drops out in response to some action on the line? (noise?)

What is the distance overall of your network?

Do you have DHCP/BOOTP disabled everywhere for sure?

Are there any external connections to the machine level network? Like upstream to a corporate network?

Are there other CPU's on the network, perhaps multicasting?

Switch: Siemens Scalance X-108 Unmanaged (that is what the CompactLogix and PointIO are connected to)

Click to expand...

These do not support IGMP snooping. And that is a must-have feature in an Ethernet/IP based system.

You don't have someone plugging something in with a duplicate IP, do you?

Click to expand...

As far as I know there isn't. When I unplug the AENT module (assigned to address 192.168.0.50) I cannot ping the relevant address. Is there another way to check for IP duplicates?

How many switches are between the L32E and the AENT? Can you try eliminating some, even temporarily? Run a wire on the ground if need be.

Click to expand...

Only 1 switch (a scalance x108) between the L32E and AENT.

What terminations are on the Ethernet cables? Are there just 8P8C plugs crimped onto the cables? Or are you using decent patch cables, and punch down blocks?

Click to expand...

The cables are terminated with a weidmuller plug: http://catalog.weidmueller.com/catalog/Start.do?localeId=en_DE&ObjectID=1963600000

Don't really know the quality of the actual cable used.

Are your Ethernet cables 600V rated?

Click to expand...

Honestly don't know.

Can you determine if the device drops out in response to some action on the line? (noise?)

Click to expand...

Just from historical trends, it doesn't seem to drop out due to any specific action on the system. Seems to be somewhat random.

What is the distance overall of your network?

Click to expand...

The run from the AENT back to the switch (which is right next to the processor) is ~20 meters.

Do you have DHCP/BOOTP disabled everywhere for sure?

Click to expand...

Positive it is disabled on the AENT module. Fairly certain it is disabled for the processor but will double check when the machine is down.

Are there any external connections to the machine level network? Like upstream to a corporate network?

Click to expand...

There is a single connection for an OPC link up to a machine on the corporate network (for our historian).

Are there other CPU's on the network, perhaps multicasting?

Click to expand...

No other AB CPU's on the network. There is an old Parker LINK system though which has it's own processor. Don't really know enough about the system to interrogate it. My extremely low-res image I have uploaded kind of gives you an overview of what is attached to the network.

Thanks for the reply by the way.

JesperMP said:

These do not support IGMP snooping. And that is a must-have feature in an Ethernet/IP based system.

Click to expand...

All I really know about IGMP is from the last 5 minutes reading wikipedia. Would a lack of IGMP snooping functionality in the switch lead to issues with connection drop out? If so, why?

Ninja Pirate said:

All I really know about IGMP is from the last 5 minutes reading wikipedia. Would a lack of IGMP snooping functionality in the switch lead to issues with connection drop out? If so, why?

Click to expand...

Yes.
Ethernet/IP uses multicast amongst many things. And that may saturate a simple unmanaged switch.
Check the system requirements for Ethernet/IP.
I am surprised that the experts you have been in contact with hasnt pounced on the unmanaged switch.

JesperMP said:

Yes.
Ethernet/IP uses multicast amongst many things. And that may saturate a simple unmanaged switch.
Check the system requirements for Ethernet/IP.
I am surprised that the experts you have been in contact with hasnt pounced on the unmanaged switch.

Click to expand...

Thanks for the info. I did just do a bit of reading on Ethernet/IP and yep, everything seems to point to needing at least one switch with IGMP snooping.

Are there any unmanaged rail mount switches that you know of off hand that can do IGMP?

JesperMP said:

Yes.
Ethernet/IP uses multicast amongst many things. And that may saturate a simple unmanaged switch.
Check the system requirements for Ethernet/IP.
I am surprised that the experts you have been in contact with hasnt pounced on the unmanaged switch.

Click to expand...

Depends on if he is using Multicast or not. Newer versions of Logix use Unicast by default so a managed switch is not a must have but I still prefer them for the diagnostics.

OP indicated he is using major rev 19 which is unicast by default if my memory serves correct.