EN2T partial comms drop and max connections reached

Hey All,

We have been having some EN2T modules (all same, up to date, firmware) experience communication drops. Odd thing is that RSLinx and HMIs stop communicating, but Producer-Consumer comms, ping, and Web Interface all still work.

We have a set of cells that this has happened to multiple times, and other sets of cells that have not experienced this at all. All have the same EN2T modules and firmware. The switches are also all configured correctly and do not show any errors.

I went into the web interface and noticed that the TCP connections were at their max (128) and then looked into the encapsulation sessions and noticed that each of the 5 HMIs had about 15-20 identical entries, which filled up the connections to its max.

A hard reset of the EN2T (pull out and reseat) corrects the problem and my TCP connections are back down to normal levels (25).

Any idea if the multiple entries for the HMIs is a cause or result of the problem? If it's the cause, any idea how it happens? Any ideas are appreciated.

Thanks!

We recently had some EN2T's replaced in connection with a known issue detailed in Rockwell product notice PN_2014-08-002. The issue described in the notice was nothing to do with communication problems but it was interesting to find that replacing the units completely resolved an annoying problem with lost connections to EIP devices connected to these cards. If your modules are from the affected series I would talk to Rockwell about replacing them.

Thanks for the reply Andy. Having a hard time finding the PN via google. Do you have a link?

That Product Notice is in the RA Knowledgebase as Article # 616044, Product Notice 2014-08-002 Revision B. (Access Level: Everyone).

Because that Product Notice describes a hardware component failure that results in a major nonrecoverable fault of the module or the CPU connected to it, I doubt it is related to your network connection overflow. But it can't hurt to check your serial numbers and datecodes.

Your information about the TCP connections going over the limit is great troubleshooting info. Now we just have to figure out if the 1756-EN2T is closing the existing connections incorrectly, or if the HMI is.

What sort of HMI devices are you using ?

The web interface diagnostics for the 1756-ENxT modules are really useful.

If you can catch this malfunction in action with Wireshark that would be awesome. Do you have a managed switch that can give you a mirror port at the 1756-EN2T ?

First, see if the encapsulation connections show a Start time or a connection lifetime value. It would be very interesting to see if the extra TCP connections start on a timed basis (once a day, every 72 hours, something like that) or if they all start together, which would suggest a network-wide disruption.

Hi Ken,

Thanks for the replies. Looks like the Product Notice doesn't apply to our cards, so we can scratch that off the list.

We are using Siemens MP377s and MP370s (WinCC 2008) in the affected areas and also in the areas that are not experiencing any issues. Wondering if there is a network setting in the HMIs that differ between the two groups. Will check that out tomorrow. If you're familiar with the models, do you know of anything specific I should look for?

The encapsulation connections do not show a start time or lifetime value. The only fields available are Session, Host Address, Inbound/outbound, and TCP inactivity timeout. There really hasn't been a pattern to the comm drops, only that we've observed them in specific areas and even then, they don't all happen at the same time or intervals. One card may drop out one day and then another on a separate day, and then maybe the first card again, or another card in the area.

The Hirschmann switches have not detected any fragments, CRC errors, or collisions. Do you think Wireshark will help?

The switches can enable port mirroring. I haven't used Wireshark, I'm assuming that you're suggesting using Wireshark on a mirrored port, correct?

Again, thanks for the replies! Any and all ideas are welcome and appreciated.

So, we still have this issue going on, and the production date is coming up with no solution found yet. Was hoping one of you fine members would have any ideas. I was lucky enough to catch one of the malfunctions in wireshark and sent it on to our central team, who were not able to identify the root cause. I have about 12GBs of data, but have narrowed the malfunction down to a 10min block (still about 150MB).

Any of you familiar enough with Wireshark to try and make sense of what is going on?