PLC5 MSG Read and Write failed

The Ethernet has 3 pieces of PLC5 processors (PLC5-40E), for easy explanations, we name A,B and C. All pings are OK and RSLogix5 can connect all Processors. Intouch I/O server can connect all PLCs. MSG Read/Write via A to B and B to A are OK, but A to C and C to A failed. Previously all MSG work well but after a new downloading to processor B, the problem comes out. How to troubleshooting and where is the problem?

I will try to keep my advice on-topic, but first: if you aren't skilled with safety and boiler and generator systems as well as the PLC-5 you should not be working on this. If this is a Solar Turbines system, then they should pay the contracts for Solar Turbines to support it instead of hiring local inexperienced integrators or using in-house staff.


The IP address "1.1.1.11" is an Internet address belonging to Cloudflare's DNS server in Miami, Florida, and should not be used on any automation system that could ever connect to the Internet. I can PING it right now, which means that any PING results you get are not necessarily returning a result from your PLC-5.

Any device that has an IP address should also have a Subnet Mask, even if it doesn't have a Default Gateway. In this system (if you were going to use these addresses, which you should not) you may wish to use 255.255.255.0 as your subnet mask.

The messaging logic is likely intended to send that MSG just once per second, but it also makes it almost impossible to see when an error has occurred because it immediately resets the /ER bit.

One way to resolve if your system is PINGing CloudFlare instead of that PLC-5 is to do this in a CMD prompt window:

TRACERT 1.1.1.11

If it shows multiple hops out to the Internet, your PLC-5 is probably sending messages to Miami. If it shows just one hop, it may not be.

Here's what a normal Trace Route to a DNS server somewhere on the Internet looks like, versus one to a device that's on your LAN:

Ken Roach said:

I will try to keep my advice on-topic, but first: if you aren't skilled with safety and boiler and generator systems as well as the PLC-5 you should not be working on this. If this is a Solar Turbines system, then they should pay the contracts for Solar Turbines to support it instead of hiring local inexperienced integrators or using in-house staff.

The IP address "1.1.1.11" is an Internet address belonging to Cloudflare's DNS server in Miami, Florida, and should not be used on any automation system that could ever connect to the Internet. I can PING it right now, which means that any PING results you get are not necessarily returning a result from your PLC-5.

Click to expand...

Noted. Thanks for this info.
1.1.1.xxx was assigned at 20 years ago in Project time by one USA team:handball:. But our ethernet is an internal LAN w/o internet connection. I may change or may leave like this.

Ken Roach said:

Any device that has an IP address should also have a Subnet Mask, even if it doesn't have a Default Gateway. In this system (if you were going to use these addresses, which you should not) you may wish to use 255.255.255.0 as your subnet mask.

The messaging logic is likely intended to send that MSG just once per second, but it also makes it almost impossible to see when an error has occurred because it immediately resets the /ER bit.

Click to expand...

I remember I have filled 1.1.1.11 and 255.255.255.0 during downloading.
In "Edit Channel Properties" page, the below Mask is for broadcast message, nothing with MSG instruction.

Ken Roach said:

One way to resolve if your system is PINGing CloudFlare instead of that PLC-5 is to do this in a CMD prompt window:

TRACERT 1.1.1.11

If it shows multiple hops out to the Internet, your PLC-5 is probably sending messages to Miami. If it shows just one hop, it may not be.

Click to expand...

I have run Tracert, it returns one hop only. So it is not the root cause.

In RSLOGIX5 "Edit Channel Properties", the diagnostic File for PLC A (1.1.1.11) is 25, the diagnostic File for PLC C (1.1.1.13) is also 25, any issue?

LANs almost never stay static or without network connections to the outside world for 20 years, even if everyone there swears that they are isolated.

It's a terrible idea to use Internet addresses for your controllers, even if you blame someone else decades ago. Someone did that to me in 1995 and nearly killed me in Texas and I am still angry about it.

If you filled out the Subnet Mask differently during download, then go get a screenshot to prove it.

You can use the same diagnostic file number in each PLC; it's not a "global data" file or anything.

If you modify the logic so it does not immediately Unlatch the /ER bit, you could get an error code from the MSG control block that might be useful, to tell if it's timing out or of the destination controller is replying with an error or exception code.

Ken Roach said:

LANs almost never stay static or without network connections to the outside world for 20 years, even if everyone there swears that they are isolated.

It's a terrible idea to use Internet addresses for your controllers, even if you blame someone else decades ago. Someone did that to me in 1995 and nearly killed me in Texas and I am still angry about it.

If you filled out the Subnet Mask differently during download, then go get a screenshot to prove it.

You can use the same diagnostic file number in each PLC; it's not a "global data" file or anything.

If you modify the logic so it does not immediately Unlatch the /ER bit, you could get an error code from the MSG control block that might be useful, to tell if it's timing out or of the destination controller is replying with an error or exception code.

Click to expand...

The
MG32:1.ERR is 21.

21 decimal = 0x0015 hex, which acccording to the RSLogix 5 Help file means "connection not completed before user specified timeout".

The Channel 2 configuration screenshot you posted suggests that the TCP connection timeout is 15000 milliseconds. So a connection timeout suggests that the PLC "A" at 1.1.1.11 cannot reach the PLC "C"at 1.1.1.13 through the network at all.


Your post suggested that a download to Processor "B" is what caused the communications between A and C to fail. The simplest explanation for that is for PLC "B" to have a duplicate IP address as PLC "C", but you say that both RSLogix 5 and InTouch can communicate with all three.

One of the tools I find very useful instead of PING is "TCPING", which creates a TCP connection instead of an ordinary ICMP "Ping". It can help you narrow down which TCP Ports a target device has open, or that might be blocked by a firewall. It is a free download: TCPing

PLC-5E controllers use TCP Port 2222 for their inter-controller communication, which is the old Allen Bradley "CSPv4" protocol. PLC-5E controllers built within the past 25 years or so will also support TCP Port 44818, which is modern ControlLogix "CIP" EtherNet/IP protocol.

If you can bring a computer to the same switch as each of the PLCs and do some testing with TCPing, this is an example of what you would expect to see.

Note that TCPing is a standalone executable program, not an operating system service or system command. I have a copy of TCPing.exe in my home folder in Windows so I can just open a default CMD window and use it without navigating to where it is or creating any path links.

Also, I recommend putting a Counter (CTU) in a parallel rung to the MSG instruction to illustrate how that logic will work.

If the MSG instruction is completing every time, the logic only re-triggers the MSG once per second.

If the MSG instruction fails, the /TT bit of the timer stays high for one full second, during which the /EN and /ER bits cause the MSG to re-trigger as fast as possible (tens or hundreds of times per second).

One of the most difficult things to do in our industry is to know when to question or re-consider the work of the people who came before you. In this situation with a network that can break if it's connected to the Internet, and MSG instructions among multiple logic controllers to exchange data for possibly hazardous systems like a boiler, I would take it very seriously and get the tools to measure, monitor, and analyze the network traffic.

Ken Roach said:

If you can bring a computer to the same switch as each of the PLCs and do some testing with TCPing, this is an example of what you would expect to see.

Note that TCPing is a standalone executable program, not an operating system service or system command. I have a copy of TCPing.exe in my home folder in Windows so I can just open a default CMD window and use it without navigating to where it is or creating any path links.

Click to expand...

FYI, TCPing all PLCs' IP address and port 2222 are OK. I will recycle power when we have stoppage.

Thanks for that detail ! I realize how hard these things are when it's a continuous process system and can't be easily shut down. If there's an over-flowed buffer in PLC A or PLC C then maybe that will solve the issue.

Using TCPING on each of the IP addresses should be a good indicator that the IPs are set correctly, but the architecture of your network might still mean that PLC A and C are not physically able to connect to one another.

Maybe it's just a coincidence that a download to PLC B happened when the comms was lost to those other controllers.

If you can, use this problem as an opportunity to map out the control network at his facility. If it has 20-year-old IP addresses that use actual public Internet server addresses, there are probably links and switches and devices you don't even know about.

I like to use Angry IP Scanner set up to look for various ports that are common in industrial automation:

80 HTTP for web servers
8080 HTTP alternate
22, 23 SSL
502 Modbus/TCP
503 Modbus/TCP alternate
2222 Allen Bradley CSPv4
44818 Allen Bradley EtherNet/IP

And I would use PING and TCPING and ARP -A and RSLogix to be sure I knew the actual MAC ID of each of those PLC-5 controllers.

A web browser, TCPing, and Angry IP Scanner are just simple free tools that most controls folks can use. They are only toys compared to heavy-duty network monitoring and management tools, which is why IT departments exist.

This morning I read a fascinating story that touches on the topic of public IP addresses:

https://www.washingtonpost.com/technology/2021/04/24/pentagon-internet-address-mystery/

In the early 1990s when IP addressing was very new, I used to field questions all the time where people had followed the user manual exactly and couldn't get their Ethernet messaging to work with PLC-5 and SLC-5/05 controllers. Often they had followed the directions too exactly, and had used an address "192.168.130.30", which was in the manual because the tech support lab was in room 130 on the first floor at Allen-Bradley in Mayfield Heights, so you could roll in equipment through the same loading dock as the cafeteria used.

Yesterday, I found one of MSG (total 4) failed to response on Processor B, it is MSG Read from A. But 3 MSG Write to A are OK.
On Today morning, finally, I got a chance to recycle the power on C, then A to C MSGs (one Read, one Write) are back to normal. But one of MSG Read on B still fails to response, I recycle processor B, then All MSGs are back to normal state.

Through Further investigation, I find the root cause (possible) --short of memory, the processor C is 1785-L20E with memory 16384 and used memory is 15676 but free memory is only 708 Words. No wonder there're many mystery problems along 20+ years according to senior staffs. Now I replace with L40E of triple memory 49152 Words, so far so good.

Thank you, Mr.Ken. I learned a lot from you, specifically the tools you introduced. I will keep your advice and will find a time to change those potential risky IP addresses.

FYI, regarding to MSG, according to manual "For the Ethernet PLC-5 processors, a MSG instruction going through Port 2, the Ethernet port, uses two consecutive message elements (i.e., MG10:0 and MG10:1)", it means we use MG10:0, we must leave MG10:1 unused.

@Ken Roach