EN2T Connection Failure

A couple months, we started noticing that our RSView stations were failing to update. A simple fix at the time was to reset the EN2T/C module by pulling it out of the rack, and reinserting it. Eventually, we replaced the EN2T and upgraded our RSView stations to Windows 7 in hopes that we could alleviate the issue, but to no avail. Eventually, we noticed that it wasn't just RSView that was losing communication; anything that connected to that specific EN2T card was dropping the connection.

This led me to believe that we were having some sort of networking issue. Being that we have a rather expansive Ethernet network, I thought that there was a chance that some rogue device was flooding our network with garbage. To test this theory out, I setup port monitoring on the switch that the EN2T card was connected to, and setup a computer with Wireshark running in hopes that I could capture the event that was causing the comm losses.

Going through the Wireshark logs, I noticed a CIP connection failure around the time that comm loss had occurred. The error contained the following info:

Extended status: No buffer memory available (0x0301)
This message gets sent from the EN2T to any device that was connected to it.

After searching, I came across an article about increasing unconnected buffers in the processor. So I looked into it, and discovered that the L73 in question was already programmed to use the maximum of 40 unconnected buffers. The counters show that the high water mark is only 19 though.

So... My question is, what else could the issue be? I'm at a loss right now...

Maybe you've already checked, but are you using hand crimped e-net cables or a factory made e-net cable? I've fixed a number of mysterious e-net problems just by re-crimping a hand-crimped RJ45 connector.

Yup, already tested the Ethernet cabling with a Fluke DTX-1800. Everything passed with flying colors.

What type of traffic are you running through the respective bridge?
Strictly SCADA or you are also passing implicit type (I/O, including Produce/Consume)communications?

I believe the L Series is a max of 40 buffers, but the default is 10 from what I was told by Rockwell. You can change the buffer size tech note 21192.

What type of traffic are you running through the respective bridge?
Strictly SCADA or you are also passing implicit type (I/O, including Produce/Consume)communications?

Click to expand...

The card in question is strictly SCADA.

I believe the L Series is a max of 40 buffers, but the default is 10 from what I was told by Rockwell. You can change the buffer size tech note 21192.

Click to expand...

The L73 is already set to use 40 unconnected buffers. The high water mark is at 19 though. The default for the older controllers was 10. I am not sure about all of Rockwell's processors, but the L73 has a default of 20.

I would go after this with the Logix 5000 Task Monitor tool. It should be able to let you see if the number of buffers used or the CPU capacity of the 1756-EN2T module is changing with events or time.

Do you have any RSLogix 5000 users running TrendX objects ? Those are notorious for using up comms buffers.

Well, I have been using the Task Monitoring tool, and it hasn't shown anything out of the ordinary. Max buffers used is 19, EN2T CPU utilization peaks around 15%, and the L73 peaks around 68%. Nothing stands out as being problematic.

kubiej21 said:

Yup, already tested the Ethernet cabling with a Fluke DTX-1800. Everything passed with flying colors.

Click to expand...

The cables are not the only "media" involved, here.
The switches count, and, for Ethernet/IP they must be pretty beefy.
AB has more than a few White Papers on this subject.

The best one is:
Publication ENET-RM002B-EN-P

Yes, our network is built on the following Cisco switches: 3750, 3508, 3524, 3550, 3560, and 3560 V2. I analyzed all of the counters and buffers on the switches that would route the traffic between the EN2T card and the RSView stations. No errors could be found. Just for the heck of it, we tried moving the EN2T and several RSView stations to a new 10GB switch, however, this did not improve the communication issues that we were seeing.

What firmware are the EN2T's? They should be at major rev 4 or 5. Anything major rev 3.xxx and older should be flashed up. We had issues with EN2T's locking up at the older revs. Flashing them up to at least 4.003 solved the issues (just got done flashing 20 or so up to 5.008). Must be some anomaly that was corrected in the more recent revisions. Just a thought.

Was there a resolution found to this situation?