Ethernet/IP connection fault

16#0203
Losing connection from time to time.
Any ideas?
I've got IGMP managed switches on a dedicated network.
Yet I seem to get a time out once in a while........

I have never seen a system where the ControlLogix traffic actually overwhelmed the network capacity, and we can postulate that the network and devices on the network will work steadily when network conditions are stable.

I start with a raft of questions:

What is the originating device ? (1756-ENBT, -ENET, 1788-ENBT, SoftLogix 5800 ?)

What are the target devices ? (1756-ENBT, 1794-AENT, PV Std ?)

How many packets/second are being served by the originating device ?

How often does this problem occur ? Have you written "trap" logic to find out ?

How many unscheduled devices are also using the network ?

Any evidence of noise in the originator or target port statistics ?

Ken Roach said:

I have never seen a system where the ControlLogix traffic actually overwhelmed the network capacity, and we can postulate that the network and devices on the network will work steadily when network conditions are stable.

I start with a raft of questions:

What is the originating device ? (1756-ENBT, -ENET, 1788-ENBT, SoftLogix 5800 ?)

What are the target devices ? (1756-ENBT, 1794-AENT, PV Std ?)

How many packets/second are being served by the originating device ?

How often does this problem occur ? Have you written "trap" logic to find out ?

How many unscheduled devices are also using the network ?

Any evidence of noise in the originator or target port statistics ?

Click to expand...

1756-ENBT Source

1756-GENERIC Target

1756-ENBT handling around 500 packets/second

I've got trap logic, and it seems once every 2-3 hours will do it.

No unscheduled devices on the network.

I would be hesitant to say noise, as I have no error packets at all when monitoring the switches.

You can guess my next question; what's the Generic module ? What is the connection's RPI value ?

If you have an available hub to install between the 1756-ENBT and the network or a way to set up a mirroring port on one of the switches, there might be a way to do some quick and dirty IP traffic monitoring to catch the incident in action.

I've got a hub.
The Generic Module is an Acromag 983EN-6012. RPI @ 50mS.
Quick question, should there be any Broadcast Traffic with Ethernet/IP?

EtherNet/IP I/O connection data is moved around via UDP Multicast.

You'll see a target IP address that isn't the same as the IP address of the target device; instead, it's a Multicast Group Member address like 239.192.1.x


I'm not familiar with the Acromag device.

Does your trap logic measure how long the timeout condition lasts before the connection is re-established ?

Ken Roach said:

EtherNet/IP I/O connection data is moved around via UDP Multicast.

You'll see a target IP address that isn't the same as the IP address of the target device; instead, it's a Multicast Group Member address like 239.192.1.x


I'm not familiar with the Acromag device.

Does your trap logic measure how long the timeout condition lasts before the connection is re-established ?

Click to expand...

The Acromag is a digital input/output 12 point card. Nothing fancy.

The Trap Logic indicates a failure time of roughly 3.764 seconds before reestablishing a connection of it's own volition.

Well, it's time to break out the ol' protocol analyzers.

Download Ethereal (www.ethereal.com) and WinPCAP and install them.

I have a brand new PC so we'll race to do the installs. Then we'll start having some fun with ControlLogix, SerialKeys, and TEthereal.

Cool.

I'm downloading now, it'll be tomorrow till I can get to work & check it all out on site.

If I would ever set up my VPN.........

It's about to get all networky in here

Attached to this post is a ZIP file with two Ethereal capture files in it. They contain traffic captured using a 10 Mb/s hub between a 1756-ENBT and a 1794-AENT with one Input and one Output module. The connection has a 50 millisecond RPI.

Normal_IO_Traffic.cap is an example of normal traffic.

Of note in this capture is frames 12 and 13. The -AENT did not respond to the outgoing packet from the -ENBT within the RPI time, so the -ENBT retransmitted the packet, but with a sequence number incremented by 1. This is normal behavior. I might have a flaky cable, the hub might have dropped a packet (it's ancient) or maybe the -AENT simply didn't respond.

Note that the sequence numbers between subsequent input/output packets are offset in this capture. I think that indicates that packets have been retransmitted by one of the devices in the connection.

Connection_Failure.cap contains the same connection traffic, but I've unplugged the 1794-AENT for three seconds and plugged it back in.

You'll see the timeout in frames 17-20. The -ENBT retries 4 times before it declares an I/O error. Fourteen seconds later (the -AENT takes a while to recover when it is unplugged), you can see the -AENT come back online and re-join the IGMP Group and start running I/O again.

Note also how the sequence numbers restart at zero when the connection is re-established.

Ethereal is a free and very powerful program but one thing it lacks is triggers that you can configure to trap events on the network.

What I have used to make Ethereal and my ControlLogix work together is to run a serial cable from the ControlLogix to the PC running Ethereal, and use a Windows feature called "SerialKeys" to allow the ControlLogix to run and stop a Windows program.

It's been a while since I worked on this and my notes are incomplete, but I should be able to recall enough of the ins and outs to give an example here.

We tried to get into this a few months ago when one of our Australian members was having trouble with FLEX I/O adapters (it turned out to be a grounding problem) dropping off his EtherNet/IP network. Time to take up the issue again !

Oooooh.
I like the SerialKeys idea........
Quite a bit.

So with this I don't need a port mirror, I just put the hub (must be a hub so it rebroadcasts, correct?) in with the switch port that I've got.

Cool.

Okay, after a lot of fiddling around with Serialkeys and the ASCII Write (AWT) instruction in ControlLogix, I simplfied things a little.

I set up SerialKeys on my computer to work on COM1. In Windows XP, you find SerialKeys by navigating Start-> Settings-> Control Panel-> Accessibility Options-> General and check "use SerialKeys". I run it at 300 baud just because I have not had good luck running it faster.

Next I add logic to the ControlLogix program to detect a connection failure, then wait ten seconds and send the following 17-character string out the serial port of the ControlLogix with the ASCII Write command (AWT):

$1b,combine,ctrl,c.

I start up TETHEREAL, which is the command-line version of Ethereal, by opening a command prompt and navigating to the Ethereal folder

C:\Program Files\Ethereal

and starting TETHEREAL with the following arguments:

tethereal -b filesize:64 -b files:3 -i 3 -q -w connfail.cap

This starts Ethereal in quiet mode (-q) with a ring buffer (-b) consisting of 3 files of 64 kB each, on Interface 3, with capture file name basis "connfail.cap".

I know it's Interface 3 on my computer because I've previously run "tethereal -D" to show the interface numbers of my Ethernet cards.

TEthereal starts running, and I leave focus on the command prompt window. When there is a connection failure, the ControlLogix sends a "Control-C" keystroke to the command prompt window, which stops TEthereal. Now I can use the graphical interface version of Ethereal to analyze the data in the capture file.

Things to remember that already caused me to scratch my head:

Set up your ControlLogix serial port for 300 baud and User Mode.

Remember to disable SerialKeys when you're done; it will take over the COM1 port and you won't be able to use anything else on that port.

The current release of Ethereal (0.99.0) has a bug in TEthereeal that makes the ring buffer save not work correctly. Use an older version of TEthereal (I will post the earlier one I am using).

I will eventually want to develop an example in which the controller used SerialKeys to start and stop TEthereal and manage its filenames, but that's a relatively big project. If you know a little about programming ASCII with ControlLogix this should be a simple little troubleshooting tool.

All this work does teach us a lot about EtherNet/IP and diagnostic tools but only gets us one diagnostic step closer to figuring out why this Generic Module connection fails.

Tethereal 0.10.14 without the ring-buffer file bug that is in the current release.

Download Link: TEthereal.zip