ControlLogix 5575 (L75): Sudden/Repeating "Unconnected Message Timeout" (0204)

ControlLogix 5575 (L75): Sudden/Repeating "Unconnected Message Timeout" (0204)

This ControlLogix PLC (L75/5575) has been in operation for at least 6 years now, is non-redundant, and is directly Ethernet-wired to two "remote" racks that only about 40' of cable away. A couple of weeks ago it started scrolling the "I/O Fault ... Unconnected Message Timeout 0204". The PLC chassis looked fine otherwise, I could remote-connect to it, and the PLC was not Faulted, but in the software - the I/O on one of the remote racks was showing the yellow triangles/"!" on all the I/O. At the cabinet, all the modules had red lights, so I pulled the network plug out of the remote rack, plugged it back in, and everything started working again. About a week or so later, same thing happened, except this time it was the other remote rack directly above (in the same cabinet). Again, I pulled the network cable, which cleared that rack, but then the other rack got hung up ... so I pulled that cable, and everything started working again. I guess these two are daisey-chained?

This has happened now at least two more times. I've seen similar complaints online about this kind of thing. Should I try re-seating the Ethernet Card on the main PLC chassis (since that's the common denominator here), or the card on the remote chassis, or both? And then what ... replace the Ethernet card(s) if the problem returns? How can you tell for sure which end has the problem?

The Ethernet Card on the main PLC chassis is a 1756-EN2TR 10/100 Mbps Ethernet Bridge, 2-port, twisted pair media, rev 10.007.

The PLC itself is a 1756-L75 ControlLogix 5575, Rev 20.54 in a 17-slot chassis.

The message is a timeout message. The controller is not getting a response from the remote chassis.

My first question would be, is there an Ethernet switch that each of these three chassis plug into? In other words, are there three separate cables plugged into the switch? Daisy-chaining is not typical on Ethernet. Are there other devices like an HMI connected? Does this system connect to a larger plant network?

I would start by troubleshooting the cabling and make sure it hasn't been damaged. Also look and see if there is anything generating electrical noise that could be interfering with the signal.

Since it is a relatively short distance, maybe when you get a chance, run a new temporary cable.

OG

@Operaghost:

Oops! Sorry, forgot to mention there is no local Ethernet Switch in either cabinet! We have that situation elsewhere here, and those 4-port switches inside the cabinets have usually been the problem.

Here's the wiring config:

Main PLC Ethernet Card (5th slot) Port #2 wired to Port #2 on Upper Remote Rack.

Main PLC Ethernet Card (5th slot) Port #1 wired to Port #1 on Lower Remote Rack.

Then, the two remote racks are connected together:

Upper Remote Rack Port #1 connected to Lower Remote Rack Port #2.

Question: is the connection between the two Remote Racks really needed? Both racks just have an Ethernet Card in Slot 0, followed by some combination of I/O Cards. They're just sending field I/O back to the Main PLC, right? Why would they need to be connected together? Isn't the Main PLC just requesting data packets for I/O from each rack? The Racks themselves aren't requesting data from each other.

Or does this just provide a redundant path for the Main PLC to get the data?

NOTE: on the Upper Rack, I noticed Link#1 indicator is NOT blinking! But otherwise, the Main PLC's links and the Lower Rack's links are both blinking fine. Does this mean that the cross connected wire is being used right now?

This is not related to your issue, but you should not be using Redundant firmware (20.054) for non-redundant system.

I'm guessing that the three chassis are each using a 1756-EN2TR. If so, you have what is called a "ring" configuration. The idea is that lets say the cable was cut between your PLC and the Upper Rack we could no longer send data directly between those two devices. But the data would be automatically routed in the other direction through the Lower Rack and then on to the Upper Rack. This configuration requires that connection between the Upper and Lower Racks. In most networks we would not want that, but you are using redundant ethernet modules so that must have been what they wanted.

I would still look at troubleshooting the cables themselves. They are the most likely culprit. But I would be curious what others more experienced with ring networks might have to say.

OG

Contr_Conn said:

This is not related to your issue, but you should not be using Redundant firmware (20.054) for non-redundant system.

Click to expand...

Huh! As Johnny Carson used to say, "I did not know that!".

But this system was previously a PLC-5, and the previous owner of this Plant paid a Contractor to migrate it to ControlLogix, so that's how it ended up with that firmware. Interestingly, this Contractor did some other PLC-5 conversions, too, and some of those were redundant, so maybe that's how the same firmware ended up in all of them?

Is there a handy firmware table of version numbers that shows what's for redundant vs. non-redundant systems?

Operaghost said:

I'm guessing that the three chassis are each using a 1756-EN2TR.

Click to expand...

Yes, that's correct!

Operaghost said:

In most networks we would not want that, but you are using redundant ethernet modules so that must have been what they wanted.

Click to expand...

Yeah, I don't know ... that's just what the Contractor did 6 years ago when these PLC-5's got migrated to ControlLogix platforms.

Operaghost said:

I would still look at troubleshooting the cables themselves. They are the most likely culprit.

Click to expand...

I suppose I could run some new cables, but even though the cabinets are only about 15' apart (stradling an entry door, but at 90 degrees to one another), the cables go out the tops of the cabinets, through steel conduits, about 10' above, before turning horizontally, and then back down. Do people just temporarily 'string cables' across from cabinet to cabinet to prove wiring problems? If the cables are inside conduits it would seem that would not be acceptable ... even for temporary testing purposes.

And to my knowledge, these cables have never been moved or removed (except recently to "reset" the I/O hangups). Do the Ethernet cables just dry rot over time? It's not a hot environment they're living in. The room temperature is 77 degrees ... so not hot. I think I have seen the connectors on the ends of Ethernet cables "wear out" ... maybe that's all it is?

It is very common to string cables outside of cabinets and conduits to prove out a network architecture error; the problem comes when you leave the system that way forever !

You mentioned that all the ring ports are blinking except one: that's normal. That port is the "blocked" port in the ring.

https://rockwellautomation.custhelp.com/app/answers/answer_view/a_id/1071668/loc/en_US#__highlight

There's a DLR diagnostic tool you can use to examine the configuration and status of the ring network. It's usually installed along with Studio 5000 Logix Designer but you can get it from RA Tech Support and install it separately.

You mentioned that the devices that were not communicating with the ControlLogix had red status LEDs. What sort of I/O devices are they (1756, POINT, FLEX, etc) ? Were you able to get any additional status information from them, or note precisely which indicators where in which states ? If there's a 1756-EN2TR module involved it should have some kind of scrolling status information.

If it weren't a ring network I would think there's a rogue duplicate IP devices on the network. Ring networks are usually *more* reliable than networks without that media fault tolerance.


https://rockwellautomation.custhelp.com/app/answers/answer_view/a_id/1091930/loc/en_US#__highlight

Colt Hero said:

Is there a handy firmware table of version numbers that shows what's for redundant vs. non-redundant systems?

Click to expand...

I don't work with redundant systems, but every one that I've seen mentioned around here was firmware rev xx.5x. Non redundant tend to be xx.1x (or xx.0x?)

And to Ken's first point, I heard someone once say, there is nothing more permanent than a temporary fix.

To clarify: if your DLR ring is configured properly (and the DLR tool will tell you if it is), you should be able to disconnect any one of the Ethernet cables in the ring and the system should function without interruption or error.

A failure to the I/O modules in a remote chassis would require the failure of two or more of the cable connections, or the failure of the network Adapter module, the chassis, or the chassis power supply itself.

Ken Roach said:

It is very common to string cables outside of cabinets and conduits to prove out a network architecture error; the problem comes when you leave the system that way forever !

Click to expand...

Ha! Ha! ... I would never do such a thing ...

Ken Roach said:

You mentioned that all the ring ports are blinking except one: that's normal. That port is the "blocked" port in the ring.

Click to expand...

Oh. OK! Yeah ... I was just starting to believe the solid green didn't matter ... only that there was at least one path of blinking green lights for the Main PLC to reach both Remote Racks.

Here's what I just observed:

....................................... Port 1 Port 2

Main PLC (MPLC) .............. blinking blinking

Lower Remote Rack (LRR).. blinking blinking

Upper Remote Rack(URR).. SOLID blinking

----------

Since the "cross-connected" 5' cable runs from URR Port 1 to LRR Port2, I pulled this cable completely off and tested it with "L-Com ELA568 Cable Tester Model DXB64A". Tester says "Green light = Good Pair", but it was showing orange light intermittently on "7&8", so I replaced this wire with a shorter, minimum-length 2' molded cable that continually tested out good. NOTE: pulling this cross-cable off completely did not affect the running system!

But then this is what the lights looked like:

...................................... Port 1 Port 2

Main PLC (MPLC)............... SOLID blinking

Lower Remote Rack (LRR)... blinking blinking

Upper Remote Rack(URR)... blinking SOLID


So it appears the "live path" right now is from MPLC (Link2), through the LRR (on both ports), to the URR (on Link1).

And this is normal? So even though the Ethernet wires may be physically connected to particular ports, communication can "cross over" from one port to the other on either side? Is that correct? Appears to be the case.

And it may switch ports at some point and the solid green light will move to the other port, or both ports will be blinking (again) at some point? Is that also true?

Ken Roach said:

You mentioned that the devices that were not communicating with the ControlLogix had red status LEDs. What sort of I/O devices are they (1756, POINT, FLEX, etc) ? Were you able to get any additional status information from them, or note precisely which indicators where in which states ? If there's a 1756-EN2TR module involved it should have some kind of scrolling status information.

Click to expand...

The two remote racks are both populated with 1756 I/O Modules. The URR is 10-slot with 3 AI's (orange), 5 AO's (yellow), and 1 blank. The LRR is a 17-slot with 3 DC Ins (blue), 9 DC Outs (green), 2 blanks, and 2 AO's (yellow). When this failure happens, every module in that rack has the red status light lit! And I haven't gotten any status info from them so far, but I can look for that on the next failure. Both of these Racks have a 1756-EN2TR in their Slot 0's, but I didn't notice any scrolling status information on either one of them. All I saw was the scrolling "I/O Fault 204" message on the Main PLC. I'll have to pay better attention next time, I guess ...

Ken Roach said:

If it weren't a ring network I would think there's a rogue duplicate IP devices on the network. Ring networks are usually *more* reliable than networks without that media fault tolerance

Click to expand...

Yeah, the Main PLC has a 2nd ENT2TR Card (but only 1 port on this one) that connects it to one of the Process Networks. I've seen other PLCs on this network that have their own unique IP address, but then if they have any remote racks, they sometimes have the oddball, what I'll call, "default" IP addresses being used at the lower levels. Maybe one of these "default" IP addresses is duplicated somewhere on this particular Process Network? How would I find that ... just hunt around and manually start recording IP addresses?


And I see the DLR Tool on the machine that I use to get on this Process Network to access the Main PLC (to bring up RSLogix 5000 Professional Network Edition V20.04.00 (CPR 9 SR 5)), but when I start it, it's asking for a "valid DLR Network Participant". I tried drilling down to the Main PLC's IP ... nope ... his Ethernet Card in Slot 4 that connects to the two Remote Racks ... nope ... then all the way down to his IP Address ... and it says "device is not in the ring". Huh? Shouldn't that get me in?

Colt Hero said:

....So it appears the "live path" right now is from MPLC (Link2), through the LRR (on both ports), to the URR (on Link1).

And this is normal? So even though the Ethernet wires may be physically connected to particular ports, communication can "cross over" from one port to the other on either side? Is that correct? Appears to be the case.

Click to expand...

The EN2TR module actually has a two-port switch integrated into it. This allows data to pass thru from one port to another.

Colt Hero said:

....And it may switch ports at some point and the solid green light will move to the other port, or both ports will be blinking (again) at some point? Is that also true?

Click to expand...

Yes. Think of the ring as a simple circle. Let's say your comms between your PLC and your URR is traveling in a clockwise direction. What you don't want is to have data travel that circle and then continue circling over and over. This causes all sorts of problems. Once it makes one full rotation you want it to stop. That is what the blocked port (red light) does.

So now let's say you remove a cable that prevents your PLC from talking to your URR in that clockwise direction. The blocked port becomes unblocked and now our traffic travels in the counter (anti) clockwise direction. When we fix our cable issue, we will continue traveling in the same direction and a different port will be the blocked port. It won't automatically go back the way it was before.

Colt Hero said:

....When this failure happens, every module in that rack has the red status light lit!

Click to expand...

This is likely because they have lost their connection to their "owner" (the PLC). Once the connection is reestablished do these go away?

Colt Hero said:

Yeah, the Main PLC has a 2nd ENT2TR Card (but only 1 port on this one) that connects it to one of the Process Networks. I've seen other PLCs on this network that have their own unique IP address, but then if they have any remote racks, they sometimes have the oddball, what I'll call, "default" IP addresses being used at the lower levels. Maybe one of these "default" IP addresses is duplicated somewhere on this particular Process Network? How would I find that ... just hunt around and manually start recording IP addresses?

Click to expand...

It sounds like your network is limited to just these three devices. The PLC connecting to another network is pretty common. But that shouldn't affect your remote racks.


OG

Operaghost said:

The EN2TR module actually has a two-port switch integrated into it. This allows data to pass thru from one port to another.

Click to expand...

OK. Good to know. Thanks!

Operaghost said:

Yes. Think of the ring as a simple circle. Let's say your comms between your PLC and your URR is traveling in a clockwise direction. What you don't want is to have data travel that circle and then continue circling over and over. This causes all sorts of problems. Once it makes one full rotation you want it to stop. That is what the blocked port (red light) does.

Click to expand...

Wait! There is no "red light" appearing on the Ethernet Cards (any of them). Just solid green lights (vs "blinking" green lights). Did you mean to say "green" there? The only red lights have been on the I/O Cards themselves when that Rack stopped communicating ... and yeah - the red lights disappear once comms are reestablished.

Operaghost said:

So now let's say you remove a cable that prevents your PLC from talking to your URR in that clockwise direction. The blocked port becomes unblocked and now our traffic travels in the counter (anti) clockwise direction. When we fix our cable issue, we will continue traveling in the same direction and a different port will be the blocked port. It won't automatically go back the way it was before.

Click to expand...

OK, but are we absolutely sure this is a DLR network ... because Ken mentioned that DLR tool and when I tried to use it I couldn't get it to connect. It acted like the tool was looking for something the network wasn't providing.

Maybe this is just a standard Ethernet configuration with that extra path between the two Remote Racks for network redundancy ... so if the Main PLC traffic goes out its Link #2 (because Link #1 is down), but Link #2 is physically connected to the "wrong" Rack for this data packet, the packet can be re-routed to the other/correct Rack over this "cross-connected" cable?

Operaghost said:

It sounds like your network is limited to just these three devices.

Click to expand...

These three "Racks", yes (Main PLC and two Remote Racks). Doesn't the term "devices" connote "field devices"?

Operaghost said:

The PLC connecting to another network is pretty common. But that shouldn't affect your remote racks.

Click to expand...

Yeah, the other (single port) Ethernet Card just gets the Main PLC out onto the larger Process Network. But here's a question though (back to what Ken mentioned about a duplicate IP):

Isn't this PLC (and any other PLC on the larger Process Network) smart enough to keep its network I/O traffic local? In other words ... only to its local, remote racks?

Could another PLC on the larger Process Network have its own Remote Racks that also use "default" (or out-of-the-box) IP addresses locally that are duplicates of these Remote Racks, and cause an IP conflict? Wouldn't these PLCs know enough to keep their I/O network requests local, and not send them out on the larger Process Network where they could possibly end up on another Remote Rack of another PLC with a "duplicate" address?

As far as the IP of the Main PLC (and all the other "Main PLCs" on the Process Network) ... I'm fairly certain there are no IP conflicts there. I have a master list of all the IP addresses and have never suspected any duplicates. But these sub-remote racks underneath them ... with the 'default' IP addresses ... I'm less sure. Could be duplicates, but like I said ... those should be isolated because the PLC shouldn't be sending data requests anyplace but on that one Card connected to those 'local' remote racks.

Colt Hero said:

...Did you mean to say "green" there?

Click to expand...

I should have said blinking green versus solid green. I had red in my brain for some reason.

Colt Hero said:

...OK, but are we absolutely sure this is a DLR network ... because Ken mentioned that DLR tool and when I tried to use it I couldn't get it to connect. It acted like the tool was looking for something the network wasn't providing.

Maybe this is just a standard Ethernet configuration with that extra path between the two Remote Racks for some level of network redundancy? And maybe this "extra connection" is causing the problems and needs to be removed?

Click to expand...

It is possible. But the network would have failed a long time ago. A ring configuration that isn't setup as a ring would fail pretty quickly. These devices are intended for redundant cabling or a ring style network. The wiring layout you describe is how a ring would be configured. So, the hardware you have, and the layout all say "ring" to me. If you go online with your controller and open the EN2TR properties you can see on the Network tab if that module is the ring supervisor. You should be able to check all three EN2TR modules.

Colt Hero said:

..These three "Racks", yes (Main PLC and two Remote Racks). Doesn't the term "devices" connote "field devices"?

Click to expand...

You're catching me being lazy with terms. "Nodes" would be the best term.

Colt Hero said:

..Isn't this PLC (and any other PLC on the larger Process Network) smart enough to keep its network I/O traffic local? In other words ... only to its local, remote racks?

Could another PLC on the larger Process Network have its own Remote Racks that also use "default" (or out-of-the-box) IP addresses locally that are duplicates of these Remote Racks, and cause an IP conflict? Wouldn't these PLCs know enough to keep their I/O network requests local, and not send them out on the larger Process Network where they could possibly end up on another Remote Rack of another PLC with a "duplicate" address?

Click to expand...

The short explanation is that the controller uses a messaging protocol called UDP to talk with I/O. To get data to pass from separate networks you have to use a router. Routers block UDP messages. This keeps all I/O data "local". There are ways around this block, but it is highly unlikely someone would go to those lengths as it introduces too many potential issues.

As for being able to connect to the DLR using the software tool, I'll have to defer to Ken or someone else as I have never used it.

OG

Operaghost said:

If you go online with your controller and open the EN2TR properties you can see on the Network tab if that module is the ring supervisor. You should be able to check all three EN2TR modules.

Click to expand...

The Main PLC is showing:

Network Mode: .............Device Level Ring (DLR)
Network Topology: ........Linear/Star
Network Status: ............Normal

Active Ring Supervisor: ..... grayed out

Enable Supervisor Mode: NOT checked

===========

UPPER Remote Rack (same as above)

===========

LOWER Remote Rack (same as above)

===========

So ... there appears to NOT be a Ring Supervisor?? Or is this not really a DLR even though it shows "DLR"?