1734-EN2TR Connection Conundrum

I made a post about this issue a few months back but it has been persistent and a solution is elusive. I could really use some insight from those more experienced.

I'm having connection issues with a Controllogix 1756-L72, with only a 1756-EN2TR in it's chassis. The primary symptom is that the HMI (Ignition based) will update values for about 5 seconds, then the tags go stale and the controls freeze up. After a few more seconds, the tags begin to update again. This behavior cycles constantly and has occurred since the initial install.

The connection path looks like this:

1756-EN2TR >>> 9300-ENA (NAT device) >>> ~230 Ft heavily shielded Cat6 >>> Unmanaged N-Tron 1080X swtich >>> Server

What I have tried:
1. Checked the port diagnostics, definitely showing errors. If I swap to the other port on the EN2TR, the errors begin to accumulate there.


2. Research into FCS errors suggests an auto negotiation failure or EM interference, or even bad cabling. I rebuilt the cable ends. I ran two entirely new cables (Not in the wire rack to rule out EM).

3. I bypassed the unmanaged switch and connected directly to a different controllogix with a EN2TR. I explicitly set both sides of the connection to 100 full duplex. I tried all possible manual combinations. I tried another unmanaged switch too. (Other devices are successfully using this switch)

4. I bypassed the 9300-ENA NAT device by addressing the EN2TR on our SCADA IP range.

5. Thinking perhaps it was signal degradation, I tried putting a switch in the middle of the cable.

6. Spent a few hours with ignition support. Everything is properly configured.

7. Checked continuity of the chassis ground.

8. I ran a wireshark capture. There is definitely some level of packet loss. Here is an example, I can post the entire capture if that would be helpful:



All of this seems to suggest a faulty EN2TR to me. I do not have a spare to try swapping it out right now. They are pretty expensive, so it would be hard to justify a large expenditure unless I can exactly pinpoint how this is failing. So how can I audit the operation of the EN2TR to better understand what is causing these errors to occur? I'm currently checking out the AOI profile to make sure I didn't mess that up during the installation. Are there any other mistakes that a novice like myself might have made in the setup?

Any help or suggestions would be greatly appreciated.

3. I bypassed the unmanaged switch and connected directly to a *different* controllogix with a EN2TR.

Click to expand...

You connected the ignition HMI server to a different PLC and experienced the same problem? This would indicate the problem is with the server?


Connect a computer in place of the EN2TR and run a bandwidth test to the server (large file transfer or whatever) and if you can saturate the ethernet link with out a lot of re-transmissions then you know the physical layer (cabling, switches, etc) is functional.



The FCS errors as you have described them suggest that the packet loss is at the ethernet media level.



While I wouldn't expect media level errors to result, have you tried changing the system overhead time slice on the processor configuration?


Does the ethernet cable have metal ends thereby connecting the ground at the server to the EN2TR? I have run in to problems with accidentally grounding systems through ethernet cables with metal connectors.

These ones are fun!

The first place I look in these situations is the EN2T's page. I don't have one handy, so I cannot direct you _exactly_ where to go, but I look at either the Application connections or the bridged connections. For each device listed on one of those pages, you will see Missed packets. Zero is the only acceptable value. Next you can look at the Diagnostic overview. You will see some of the same values you have seen in the properties tab of the module. As mentioned, the FSC alignment errors are a problem that you need to resolve.

On the diagnostic overview, pay attention to CPU usage, maximum number of connections, missed/rejected packets and I think there is a packets per second field. CPU usage should be below 80%.

So I have 4 PLCs connected to ignition, all except this one are working fine and share the unmanaged switch.

I have 2 controllogix PLCs, each with it's own EN2TR. In order to manually set auto-negotiation speeds, I plugged the cable coming from the faulty EN2TR to the extra port on the good EN2TR. I was able to see port diagnostics on both sides. The faulty EN2TR generated errors, while the good EN2TR did not.


I have tried changing the overhead timeslice, forgot to mention. I went from 20% to 50% and didn't see a change. Ignition support also thought I was overloading the PLC comms, but the diagnostics show that they are barely taxed.

The cables connecting devices within panels are prebuilt with plastic ends. The long cable between the panels was plastic, then I rebuilt the ends with metal connectors. The other 'test' cables that I ran outside of the wire tray were plastic.

I will try your suggestion of a high bandwidth transfer. Any preferred tools for performing this?

Some screens of the diagnostics so you can see what I'm working with:

Diag:



Bridged Conns:




App Conns:




So I guess I don't see missed packets. If they are getting retransmitted, do they count as missed?

If you have two Controllogix racks each with its own EN2TR, then can you just swap the two cards and see if the problem moves?

Sorry for the rapid fire screen shot posts. I'm perusing the module configuration and have some suspicions.







So the product revision is 10.010 yet the module revision is 10.10? It also lists 'Connection: None', and 'Configured: No'. Are these indicative of an improperly configured EN2TR?

SD_Scott said:

If you have two Controllogix racks each with its own EN2TR, then can you just swap the two cards and see if the problem moves?

Click to expand...

I will do this if I've exhausted any other suggestions first. The controllogix with the working EN2TR cannot be shut down without disruption of production. (I don't think they support RUP) The faulty one controls a tank farm that can go offline with little impact.

But that would definitely be the most straightforward way to test it. We're rapidly approaching worst case scenario

My analysis is that you have a defective 1756-EN2TR.

In my opinion, the important clues are that you're seeing only FCS Errors and MAC Receive errors.

If this were a cabling or EM interference problem, you would see a variety of other Ethernet diagnostic counters incrementing. If it were a PHY problem then it would not move from Port 1 to Port 2.

While the FCS and MAC Rx error counters have similar values, that's probably a coincidence. They are actually mutually exclusive; an error that's counted by the FCS Error counter will not be counted by the MAC Rx error counter.

I noticed that you hard-set the duplex and speed to 100 Mb/S Full Duplex.

It is a very common misunderstanding that "Auto-Negotiate" means "Auto-Detect", because detection of a bit rate and duplex is part of the negotiation. An Auto-Negotiate link takes two to tango.

I only recommend manually setting the data rate and duplex if you have hard-set devices on either end that do not support auto-negotiation, like some fiber optic interfaces.

If you have an unmanaged switch, you should NOT hard-set the data rate and duplex.

Here's why: some (maybe most) unmanaged switches perform auto-negotiate when the link is first established, AND periodically, even if it's running fine at 100 Mb/s and Full duplex with no errors. They tend to do that when the device at the other end does not respond to negotiation signals and they have to fall back to manually checking 100/10/full/half by themselves.

On ordinary office networks this is unnoticeable; maybe a few frames out of place on your YouTube stream. On an automation network it can break stuff.

I don't think that's what is happening here, because generally these interruptions are long enough to break an I/O connection (and your uptimes are in the many-days-long range) and because they don't result in FCS and MAC Rx errors.

But it's a principle worth noting.

Thank you very much for the input guys.

I have the duplex and speed set to auto now. I had only changed them during that iteration of testing, but changed them back immediately after I determined it didn't help.

Is it silly to try to re-seat the en2tr in the chassis? Blow on the cartridge if you will...

Ken Roach said:

I noticed that you hard-set the duplex and speed to 100 Mb/S Full Duplex.

It is a very common misunderstanding that "Auto-Negotiate" means "Auto-Detect", because detection of a bit rate and duplex is part of the negotiation. An Auto-Negotiate link takes two to tango.

I only recommend manually setting the data rate and duplex if you have hard-set devices on either end that do not support auto-negotiation, like some fiber optic interfaces.

If you have an unmanaged switch, you should NOT hard-set the data rate and duplex.

Here's why: some (maybe most) unmanaged switches perform auto-negotiate when the link is first established, AND periodically, even if it's running fine at 100 Mb/s and Full duplex with no errors. They tend to do that when the device at the other end does not respond to negotiation signals and they have to fall back to manually checking 100/10/full/half by themselves.

On ordinary office networks this is unnoticeable; maybe a few frames out of place on your YouTube stream. On an automation network it can break stuff.

I don't think that's what is happening here, because generally these interruptions are long enough to break an I/O connection (and your uptimes are in the many-days-long range) and because they don't result in FCS and MAC Rx errors.

But it's a principle worth noting.

Click to expand...

Ken is 100% right...the speed should always be set to Auto, not fixed. The internet is ripe with uber-Geek discussions about where all this came from, and the AB technotes are full of recommendations to set all ends, everywhere, to Auto.