Ethernet I/O errors

This is a continuation of a thread that I started a few weeks ago at the now defunct RSI forum. It was a 3-way conversation there between Ken Roach, Andy123(ContrConn?), and myself. A brief summary follows. Any ethernet experts hanging out here - feel free to jump in.

The problem is with a CLX system installed in 2003 that uses ethernet for I/O. The attached IOCfg.zip file contains a .jpg file showing the I/O tree. As you can see, the I/O is divided between two ENBT's. The attached .pdf files are relevant to the ENBT in slot 9.

Approximately 14 months after installation, comms failed to one of the flex I/O groups causing a small fire. Subsequently, some monitoring programming was installed using GSV instructions to monitor "entry status" of every module in the system. This programming is detecting multiple, frequent errors on all of the flex I/O groups - predominantly on the groups under the ENBT in slot 9. When a fault occurs, the "entry status" for each of the I/O modules goes bad at the same time but the adapter status remains good. The duration of the fault is approx. 8 seconds. The fault code from each of the modules is hex 0203 indicating a lost connection. In spite of the errors, the line continues to run. There are rare occasions, however, when the error does not clear.

There was some questioning of how often the GSV's should be executed - i.e. am I creating the errors?

It was recommended to upgrade the ENBT firmware to 3.9 - has been done - no change.

All ethernet connections are to a Hirschmann 'Mice' switch.

Just before we were cut off at RSI, there was talk of analysing the system with 'ethereal'. I have that program and have even fired it up a few times - but I don't really know what I'm looking at there.

How many GSVs are you executing. I have heard the argument that they should not be executed continuously but I have a program that executes 47 continuously. It was originally done because I didn't know any better but I have never changed it and do not have any communications issues. The only thing different about my GSVs is that I get the FaultCode instead of the EntryStatus and look for it not to be equal to zero. I think the FaultCode gives you a little more detail as for why it is not communicating

Gerry and anyone who wants to join this discussion:

Let me bring up what I already said (and now is gone):

Error 16#0203 that you captured means that ALL the following happened:
1. communication was originally OK
2. a scheduled packet was lost
3. at least 3 additional retry packet lost (4xRPI)
4. module generated 0203, closed connection and opened it again.


If item 3 (retry) sucsessfully delivered at least one packet out of 3, then you will never see 0203, but data was lost.
The fact that connection recovers by itself makes me to believe that you do have some packet lost that are not captured by 0203 error.

Ethereal capture will show us this scenario.

Also I would like to see screen shots of webpages from ver 3.9 - shots above are from old version.
We need all 4 diagnostic pages.

Screenshots from remote modules (CLASS 1 CIP) will help too.

Also I would like to have a I/O configuration of your project in ACD format (no logic) to see RPIs.

And the last question: are you loosing 1794-AENTs connections only? If yes, it may be a different reason.

Trying to remember, the RPI of all of your Flex modules was 20ms. Is that correct?

Gerry

Please check:
- Make sure DIN rail for 1794AENT is grounded
- Make sure Negative Leg on 24VDC power source to AENT is grounded.
- Add ferrite beads to the 24V power and Ethernet cable.

If nothig helps, then the next step will be to isolate 1794-AENT from the DIN rail.
There is a grounding clip on back of AENT: try to slide whole assembly left about 1 inch (25mm) so ground clip is off the rail or just put some other temporatry isolation. If this helps then let us know or call AU techsupport for details.

Trying to remember, the RPI of all of your Flex modules was 20ms. Is that correct?

Click to expand...

that is correct.

And the last question: are you loosing 1794-AENTs connections only? If yes, it may be a different reason.

Click to expand...

Yes. There are no recorded instances of failure to the remote 1756 chassis. And the AENT adapters don't fault either.

The fact that connection recovers by itself makes me to believe that you do have some packet lost that are not captured by 0203 error.

Ethereal capture will show us this scenario.

Also I would like to see screen shots of webpages from ver 3.9 - shots above are from old version.
We need all 4 diagnostic pages.

Click to expand...

I did look at the 3.9 web pages and noticed a field indicating lost packets - neglected to take a snapshot:doh:

I am a contract programmer and often on site, but not always. I will pass along the info re grounding.

To use Ethereal, I presume I'll need to set up a port mirror - is that correct?

To use Ethereal, I presume I'll need to set up a port mirror - is that correct?

Click to expand...

This is correct. If switch has no mirror option, then you need to add a hub between ENBT and switch.

Also please take a look ay my last post - ground clip may be the key!

I was going to start another thread, but I thought it may be helpful for this post.

Contr_Conn - What is Ethereal exactly?

TWControls said:

I was going to start another thread, but I thought it may be helpful for this post.

Contr_Conn - What is Ethereal exactly?

Click to expand...

Not Contr_Conn, but
http://www.ethereal.com/
An ethernet sniffing tool.

I've been there, that is why I addressed my question to Contr_Conn. I think I am more confused than Gerryh on the proper use of it

Another question Gerryh. Can you be more specific on which 1794 modules are failing or is erratically all modules?

As CroCop pointed Ethereal is one of ethernet traffic capture programs sometimes called "network sniffer"
Why Ethereal? - simply because it is really good and it is free.

How sniffer can help Gerry?

As we suspect packet loss, I would do the following:
- Capture traffic at ENBT, save it.
- Apply filter to see only traffic associated with one of 1794-AENTs ans save it again under different name

Next:

- I will filter to see only multicast from AENT.
We are expecting to see multicast packet from AENT every 20 ms.
I would set it to see time between packes and sort them High to Low by time.
If packet lost from AENT present, you will see time between packes 40ms, 60ms and higher
Connection with no problems will show all at 20ms.

Repeat with ENBT packets to AENT.

Also by applying some other filters we can see corrupted packets, retries etc.
If Gerry posted his capture, I can comment on it...

Ok. So from this you can also tell which direction the packets are being lossed also?

Ok. So from this you can also tell which direction the packets are being lossed also?

Click to expand...

Yes.
Technically in some cases when we suspect media or switches capture must be done on both ends (at ENBT and at AENT):
we may see packet sent by AENT but never received by ENBT.

First, if I need to take this to another thread say so, but I have downloaded the software and am not understanding how you would capture packets without routing them through your computer. But I am not very technical on the details of networking. I get the feeling it has something to do with the port mirroring that was mentioned before