ControlLogix to PLC-5 message anomaly

[mylespetro]

Hey everyone,

Got called out tonight on an issue with a ControlLogix L62 processor, curious if anyone has seen anything like this before. I've posted a couple threads about this particular system before, but it's a central data collector (Integrator) for a power plant, acting essentially as a multiplexer to read/write between a bunch of field systems (mix of PLC-5, CompactLogix, and ControlLogix PLCs in the field) and the plant DCS (Emerson Ovation).

They were having issues with some (but not all, this is key) I/O points in their sootblower system, so I came out to take a look at the Integrator program. They had the sootblower system locked out under permit earlier in the day (or maybe yesterday), and part of this procedure was powering down the sootblower system PLC. At first we noticed one of eleven messages going to the sootblower PLC-5 (DH+) was erroring out, with the error message "16#0200 - An unknown error has occurred." Then after a couple of minutes we realized that actually three of the 11 messages were erroring out (messages 1, 5 and 10 in the sequence), however, the rest of the messages were executing without issue. We know this to be the case because the I/O contained in the addresses in those messages was updating in the DCS.

When I investigated further, message 1 had the error "16#00fe - Message timeout," while both messages 5 and 10 had the previous 16#0200 error code. First we tried a power cycle on the sootblower PLC-5, which didn't work. I then tried disabling the messages from executing and then re-enabling them, but this didn't solve the issue either.

What eventually solved it was pulling the Integrator ControlLogix processor and reseating it, and the message sequence started back up again with no issues, and they were able to read all of the I/O in the sootblower again, as well as start the sootblowers as required.

Has anyone ever run into this issue where some (but not all) messages going to the same processor are erroring out? If I recall correctly, they were all PLC-5 Word Range Read MSG instructions.

Just to reiterate the sequence of events as I understand them:

1. System was running without issue
2. Sootblower system was taken out of service for mechanical maintenance, PLC-5 powered down
3. Sootblower system and PLC were powered back up to return to operation
4. Of eleven (11) sequential messages from CLX to PLC-5, messages 1, 5 and 10 were erroring out but all others were executing without issue
5. Had to pull CLX processor to re-establish message function

Again, just curious if anyone has run into this before or may have any insight as to why this would happen. Not sure if the PLC-5 power cycling caused it or not.

Thanks in advance!

 

[jstolaruk]

I wonder if an internal message queue got filled up.

Does the CLX logic continue to try and send read requests without the PLC-5 powered up? Or Is there logic to disable them if its determined that the connection to the PLC-5 doesn't exist?

Is there anything uniquely different between the message instructions? Polling times?

 

[chelton]

OT slightly, as your working with DH+, but i recently discovered that I had numerous connections to a PLC5 coming from IP addresses that shouldn't have been communicating with the device. I found this by browsing the embedded web page diagnostics. The connection came from old MSG write instructions that had been programmed in at some stage on different logix processors. Although the MSG instruction had been deleted from the running code, the MSG data block was still populated. So i'm not sure why, but the MSG was still creating a connection to the PLC5. Deleting the unused tag (of Message data type ) broke the connection.

 

[mylespetro]

I wonder if an internal message queue got filled up.

Does the CLX logic continue to try and send read requests without the PLC-5 powered up? Or Is there logic to disable them if its determined that the connection to the PLC-5 doesn't exist?

Is there anything uniquely different between the message instructions? Polling times?

That’s a good thought, definitely a possibility. Just strange that it only affected a couple messages. Configurations are identical aside from the addresses being read.

OT slightly, as your working with DH+, but i recently discovered that I had numerous connections to a PLC5 coming from IP addresses that shouldn't have been communicating with the device. I found this by browsing the embedded web page diagnostics. The connection came from old MSG write instructions that had been programmed in at some stage on different logix processors. Although the MSG instruction had been deleted from the running code, the MSG data block was still populated. So i'm not sure why, but the MSG was still creating a connection to the PLC5. Deleting the unused tag (of Message data type ) broke the connection.

That’s interesting, definitely a strange hiccup regarding the message data block. I didn’t even think to check the web page for the CLX.

Thanks for your input guys, I’m just wondering if it’s something that can be reproduced reliably, would be a good “gotcha” to keep in mind.