ControlLogix 1756-L61 communication error

We have Allen Bradley Control Logix 1756-L61B controller with RSLogix5000 v16.03.00 (CPR9) software installed on a reciprocating compressor at a remote site. We are monitoring the operational parameters of the compressor over a Wireless Ethernet Bridge of 25Mbps on Factory Talk View Studio 5.00.00 (CPR9) HMI. The PLC communicates with the Wireless link with its 1756-ENBT module over ethernet.

We are often faced with loss of communication resulting in disappearing of the HMI tags values (real-time) on the Factory Taklk View Studio SE HMI. Following error messages occur during such a situation:

CIP connection (3) open rejected (Error 2040101) on route ControlLogix in slot 0 of the chasis at 172.19.11.101
Driver Ethernet lost communications to 172.19.11.101:socket error on send
CIP connection (3) open rejected (Error 2040101) on route ControlLogix in slot 0 of the chasis via cip://Ethernet:172.19.11.77/1:0/2:22

Please help us in troubleshooting this communication loss problem which hinders our parameters monitoring.

I've been fighting a similar issue for months on a RSViewME PV+. Rockwell support hasn't targeted the exact cause yet.

The only thing we've determined so far is that the ENBT should be set for Automatic speed and handshake rather than forcing it to 100 and Full Duplex. Change this in RSLinx by right-clicking and configuring module.

Which part stops communicating? The HMI or the PLC? Can you connect to the PLC with RSLogix5000 during this comms outage? Or does it shut you out? Does cycling the power to the PLC rack bring it back?

Check the PLC connections with the website for the ENBT... just type the IP address of the PLC into a browser window. Observe if the "Open Connections Requests" keep ratcheting up while the HMI is operating properly.

You need to diagnose how many packets your wireless link is losing, and investigate how to improve it.

The 1756-ENBT has some Ethernet statistics you'll want to look at. The wireless access point probably does as well. There may be no frames or packets lost at all on the local Ethernet; it might be entirely over wireless, so the only statistical evidence will be at the TCP protocol level.

Error code "0204" is a simple Timeout.

Don't go looking for some exotic firmware problem or product defect when a low-reliability wireless link is right in front of you.

@KILLER: Most probably the HMI PC with FactoryTalk View Studio SE software stops communication. During this communication problem if we ping any of the PLCs via the wireless link, we receive replies alongwith some timeouts indicating that the Computer is still connected to the PLCs.

We checked the CIP Connections Statistics which are as follows:
Conn. Open: 28236
Open error: 1084
Conn. Close: 110
Close error: 19
Conn. timeout: 3483
Uptime: 39 days 21hrs 09mins 25sec

We also checked ENBT Module configuration and found that the option "Auto-negotiate port speed and duplex" was checked.

Please also find attached the connection architecture of our PLCs and HMIs.

Thank you for posting that sketch of your system layout and those diagnostic statistics from one of the 1756-ENBT modules.

I have some architectural and configuration questions.

1. Are both the Remote Site and Base Site instances of FactoryTalk View SE the Standalone version, or is one of them a Client and the other a Server ? This will make a difference about how much and what type of data flows between them.

2. Does the Remote Site HMI experience any of these timeouts ? From your description it sounds like the poor performance is exclusively over the wireless link.

3. Is the Ethernet switch a Managed switch ? Have you done any configuration to be certain that broadcast and multicast traffic that should not be on the wireless link does not reach the wireless link ?

4. Does the wireless link give you any packet statistics about dropped or retransmitted packets ?

I think you will need to use some network analysis software (Wireshark is a free and extremely powerful tool) to make sure the wireless link is not overloaded and to measure what effect changes in configuration and software have on the error rate over the wireless link.

@ Ken Roach: Thanx a lot for taking interest in this issue, following are the answers to your architectural and configuration questions:

1. Both the Remote Site and Base Site have FactoryTalk View SE Standalone version

2. In remote site HMI, this behaviour is very rare

3. No, the Ethernet switch doesnot have any configuration. Its a simple commercial switch (no IP address of switch)

4. Yes, the wireless link gives packet statistics about dropped or retransmitted packets.
When we ping the link on base-site or remote site from base-site HMI, no lost packets/time outs are detected.
However when we ping the link on remote site from the remote site HMI, we get several timeouts alongwith many replies.
Sometimes even the bandwidth of the link also goes to 0.0.

Has it something to do with the link or the FactoryTalk configuration on base-site PC.

It is very unlikely that the problems on the wireless link have anything to do with the FactoryTalk View configuration, except in that the amount of data requested across the link is configurable in the FactoryTalk project.

Do you know if there are any Ethernet I/O connections, or Produced/Consumed Tags on the Remote Site network ?

It would be extremely useful to run Wireshark software connected both at the remote site and at the base site to see if there is multicast or broadcast traffic clogging the wireless link.