Ethernet I/O errors

[Contr_Conn]

I would keep it here.

I explained how to do capture at RA forum before, but looks like this is a better forum to to it:

90% captures I saw are not taken correctly simply bacause normal switch will isolate all TCP traffic between 2 devices and PC will not see it.
Few ways to get capture:
prefered - Mirror ENBT port to a computer port so PC will see all ENBT traffic.
another way is to use a hub between ENBT and switch and connect PC to this hub. In some cases this will mask the real problem and/or create a new one.

One more way is to use ethernet TAP like this
http://www.blackbox.com/Catalog/Detail.aspx?cid=151,154,991&mid=4220
But I never had one...

 

[TimWilborne]

Ok thanks. Many of our switches at work are capable of port mirroring. I will try to learn how to use it tommorrow

But not to be picky, but we already know that we are loosing 4 consecutive packets between the ENBT and the Flex module(maybe, see below). What additional information can we learn from the monitoring?

Gerryh is saying that the connection between the Ethernet adaptors (ANET) is not actually failing. The connection between the ENBT and the I/O flex modules is failing. I am thinking that if it was a problem with Ethernet hardware there would be faults to the ANET module also. Seems more like configuration or a wiring problem on the I/O modules.

I may be repeating some things from the old forum. I found a cached version for the thread but the later post were missing

 

[CroCop]

The Ethereal will give you very specific statistics of the motion of packets on your network. It is a highly valueable tool, and can help you isolate your problem to a particular device, and slow down the packet transfer to allow you to monitor it.

 

[TimWilborne]

Ethereal is a very interesting program. Although I don't know how to interpret the information that you get from it yet it definitely gives you some detailed information on what traffic is coming across the port.

Plus now I understand why everyone suggest you have port mirroring on your switches

 

[Contr_Conn]

But not to be picky, but we already know that we are loosing 4 consecutive packets between the ENBT and the Flex module(maybe, see below). What additional information can we learn from the monitoring?

It is a big difference if we loosing 4 packets once a week or we constantly loosing one-two packets and detect this only if 4-in-a- row lost.
I am suspecting that we have a constant packet loss possibly because of grounding issue with AENT. The fact that only AENT failing makes me to believe that Gerry has this issue.

 

[TimWilborne]

Ok. And now I understand more of how the Ethereal works. It gives you the detailed packet monitoring you would need to see if it is a once a week or a constant packet loss. I'm starting to get it

 

[Gerry]

Update

OK - I've been back to site and have some more info. I believe the attached I/O connection stats confirm Contr_Conn's suspicions. The numbers in the "Lost" column do indicate consistent packet loss. I presume the changes in the "Up-time" column between captures indicate connection loss.

Site tried lifting one of the flex groups din rail off the gear plate - resulted in increased failures. When re-attached to the gear plate, settled down again.

A few weeks ago, at a local "Automation Expo", I spoke to a US engineering manager, Fred (long German surname that I can't remember), about the problem and he immediately suggested grounding and asked whether the din rail was aluminum or steel. Turns out they're ali and are now going to be replaced with plated steel and bonded (possibly by tomorrow).

I did some Ethereal captures - they're rather bulky - nearly 3MB for a 1-minute capture (too big to attach even when zipped). They don't mean much to me, but I did notice occasional "keep alive" packets between the ENBT and flex.

Thanks to all for the help - I will keep you posted.

 

[Contr_Conn]

Gerry,

Fred is correct, steel DIN rail must be used, and it must have proper direct grounding.
But if nothing helps, I would try to isolate from DIN rail AENT only and leave all other module connected to the grounded rail.
If lifting AENT only from grounded rail helps, then call AU techsupport for future steps.

Also - make sure negative leg of 24VDC PS is grounded and AENT has own power source.

If shielded (STP) cables used, then only one RJ45 connector must be shielded (switch side) to prevent ground loops.

You can email your Ethereal captures to me (PM for email address) - I will take a look.
Ken had FTP link to post it, but I can't find it now.

 

[Gerry]

I've uploaded a zipped copy of one of the Ethereal captures to the download area on this site in the 'Misc' folder.

 

[Contr_Conn]

Unfortunately this cature has no usable information.
I don't see any multicast traffic at all.
I don't know what went wrong - looks like port mirror was not set correctly assuming managed switch used.

 

[Contr_Conn]

Ok, few things I can definately see on ENBT statistics:

I highlighted missing packets - see full pdf above.

This is a good indication of the bad connection.
This AENT .65 lost 31200 packets within 13 days or 100 packets per hour. I think this is your problem.

Without multicast traffic capture it is really hard to filter missing packets.
Also it would be good to see counter packets per second if your ENBT - I see on ehtereal capture ENBT responds to .65
I see ENBT sent 36000 packets in 60 seconds - this is 600 per second.
With RPI 20 ms we are getting 50 incoming packets per connection per second. 600/50 =12 connections open. Does it match the actual number?

 

[Contr_Conn]

Now what can we get from incomplete Ethereal - some good data.
First - capture is only 60 seconds long, but per our estimate we should see some packet loss.

I filtered packets for .65 as it has highest packet loss.
Under normal condition you will see 4 responses (4 connections to AENT) every 20 seconds.

Second column time between packets. Add 4 in a row together and it should be 20 ms.
And under any conditions we should not see interval greater than 20ms.

Now look at this fragment:

As you you see we have late response as a result of missing multicast.

We need to see a multicast to prove this.

 

[Contr_Conn]

Under normal condition you will see 4 responses (4 connections to AENT) every 20 seconds.

Should say 20 milliseconds, sorry

 

[Gerry]

I don't see any multicast traffic at all.
I don't know what went wrong - looks like port mirror was not set correctly assuming managed switch used.

I didn't notice any options on the set-up screen for the port mirror.

But conversation statistics list the multicast addresses.

 

[cjh]

I didn't notice any options on the set-up screen for the port mirror.

You have to setup port mirroring on your managed switch. If you are not using a managed ethernet switch, you will not be able to see a conversation between 2 other devices.
I believe that you are seeing the multicast addresses because the switch will forward the multi cast to all ports but you will not be able to see the conversation between the 2 other devices.