DHRIO Issues

mylespetro

Member
Join Date
Dec 2015
Location
NS
Posts
740
Hey everyone,

Working on a system with a newly installed ControlLogix 5571 (Rev 24) which replaced an old SLC (5/04 I believe). The SLC originally communicated I/O back to a DCS integrator PLC, with a PLC-5 piggybacking onto it over DH+. The DCS integrator PLC is another ControlLogix, but it's a 5562 (Rev 15.04), and I seem to be having a bit of trouble talking to the integrator PLC remotely. When I go into RSLinx, I can see the integrator PLC by going into the new PLC through USB, then out of the DHRIO card on Channel A, which shows me the PLC-5 and integrator PLC, but I can't go online with the processor. It gives me Error 701-8004250A, and says "No open connection." There are a fair bit of MSG instructions happening in the integrator PLC, could that be hogging the connections?

Thanks
 
Some screenshots of the system layout, sorry for the links, I can't seem to get the upload function to work.

https://ibb.co/c7krvf
https://ibb.co/nLVrvf

The first image is a network layout of the original system. I've circled the area I'm working in, with the SLC 5/04 at node address 05 being replaced with the new CLX PLC.

The second image is what I see in RSLinx plugged into the new CLX ("SCREW_COOLER"), and you can see that both the PLC-5 (node 10) and the integrator PLC (node 51) appear. However, I can't get online with node 51.

EDIT: I can also expand node 51 in Linx and see the backplane and the rest of the cards in the rack.
 
First, let's make sure that you're pointed to the correct device. That second screenshot cut off the part that shows the backplane of the new Ash Screws controller in the backplane with the 1756-DHRIO at Node 51.

You need to highlight the CPU itself, not the 1756-DHRIO. That exact error code will show up if you point to a communications module instead of a CPU.

Second, have you tried this with a first-hop on Ethernet instead of the USB port of a ControlLogix ? I don't know of anything specific that would keep this from working, but it would be another data point in the investigation.

But I think your instinct is correct on this; the "Integrator" CPU is doing a bunch of messaging through multiple comms modules, and it would not surprise me if the CPU or the DHRIO is out of available connections. Fixing that would require a careful review of the whole Integrator application to find out if it's genuinely out of resources, or if there are programming techniques that could recover some wasted resources.
 
First, let's make sure that you're pointed to the correct device. That second screenshot cut off the part that shows the backplane of the new Ash Screws controller in the backplane with the 1756-DHRIO at Node 51.

You need to highlight the CPU itself, not the 1756-DHRIO. That exact error code will show up if you point to a communications module instead of a CPU.

Second, have you tried this with a first-hop on Ethernet instead of the USB port of a ControlLogix ? I don't know of anything specific that would keep this from working, but it would be another data point in the investigation.

But I think your instinct is correct on this; the "Integrator" CPU is doing a bunch of messaging through multiple comms modules, and it would not surprise me if the CPU or the DHRIO is out of available connections. Fixing that would require a careful review of the whole Integrator application to find out if it's genuinely out of resources, or if there are programming techniques that could recover some wasted resources.

I haven't tried it using Ethernet, it's an option. But my guess would be an excess of messages. The integrator program uses a bunch of read/writes titled for example "AshMsgBlk1" etc, and then executes them using an XIO of their own DN bit, example below:
|--]/[---------------------[AshMsgBlk1]-|
| AshMsgBlk1.DN------------------------|
|--]/[---------------------[AshMsgBlk2]-|
| AshMsgBlk2.DN------------------------|

I feel like this may not be the best way to go about doing it, but it's a rather large program. My initial thought would be to execute them top to bottom, based on the status of the previous rung, but I've never dealt with data transfer on this scale before, especially using DH+.

Also, I think I saw somewhere else on the forum that anything in excess of 16 messages can cause issues, it that true? If so, that may be the case here, as there are a lot of message instructions.
 
Just as an aside, I've forced some points in the new CLX and they are changing in the DCS, so I think aside from not being able to go online with the integrator PLC remotely, we are good on communication.
 
I feel like this may not be the best way to go about doing it

Your intuition is correct.

That sort of self-re-triggering is the logical equivalent of hammer-fanning a single-action revolver. It's the fastest way to get the load downrange in a hurry, but it uses up all the buffer resources that are available.

My old job included fixing these sorts of messes professionally. The first thing I would do is undertake a mapping endeavor to figure out which messages go to which controllers, then try to understand how fast they need to arrive.

I strongly prefer to trigger messages based on a timer or sequencer, and prevent them from being re-triggered until the previous message has completed or has returned an error code.
 
Your intuition is correct.

That sort of self-re-triggering is the logical equivalent of hammer-fanning a single-action revolver. It's the fastest way to get the load downrange in a hurry, but it uses up all the buffer resources that are available.

My old job included fixing these sorts of messes professionally. The first thing I would do is undertake a mapping endeavor to figure out which messages go to which controllers, then try to understand how fast they need to arrive.

I strongly prefer to trigger messages based on a timer or sequencer, and prevent them from being re-triggered until the previous message has completed or has returned an error code.


The good thing is that the integrator PLC is the final destination for the data, the DCS system then polls data from the processor. This means that there's no further mapping required, just changing how the messages are executed. My boss actually said just before I left work that we should redo how the messages are sequenced at some point, so I'm going to look into that.


Thanks for the replies!
 
Hey everyone,


A new issue popped up with this same PLC/DHRIO card. The weekend before last I got called out to troubleshoot a loss of comms with the DHRIO card in question, the other DHRIO cards in the rack were still communicating. I unplugged the DHRIO card and plugged it back in (like a true professional) and the comms came back online. Everything worked smoothly until last night when the same thing happened. The controls tech onsite did the same thing I did (unplug and plug back in) and comms returned again. Naturally we're now trying to figure out what is causing the issue.


As I said previously in the thread, there are a lot of MSG instructions happening simultaneously, and I'm wondering if some kind of buildup of errors causes the card to crash or something along those lines. I'm going to work on reducing the number of instructions and try using either the DN bit of the previous message or a sequencer to execute the program, but has anybody ever ran into something like this?


Also, I didn't get to edit the previous post where I said they used each instruction's DN bit as an XIO, but it's actually each instruction's EN bit, which is how the DHRIO manual says to have a constantly updating integer value.
 
How many is "a lot"?

I'd suggest following Ken's advice and build your message logic to cascade them based on the ER or DN bits from the previous message, perhaps with a little time delay between each. If you are bombarding the connection it is probably choking due to lack of bandwidth/buffer space.

I normally program a state engine to drive the messages where each step is triggered by the completion of the DN or ER bit of the previous message along with the completion of a timer that gets reset whenever the state changes steps. I can also add in branches (if needed) to skip over nodes that are out of service or non responsive.

Then the MSG instructions are all listed with a single state engine bit in front of them. This makes it easy to follow the flow, monitor the total time to poll everything, and when dealing with the Micrologix (which does not allow online editing of the MSG rungs) I can alter the sequence logic online as long as I stick with the same control bits for each MSG instruction.
 
How many is "a lot"?

I'd suggest following Ken's advice and build your message logic to cascade them based on the ER or DN bits from the previous message, perhaps with a little time delay between each. If you are bombarding the connection it is probably choking due to lack of bandwidth/buffer space.

I normally program a state engine to drive the messages where each step is triggered by the completion of the DN or ER bit of the previous message along with the completion of a timer that gets reset whenever the state changes steps. I can also add in branches (if needed) to skip over nodes that are out of service or non responsive.

Then the MSG instructions are all listed with a single state engine bit in front of them. This makes it easy to follow the flow, monitor the total time to poll everything, and when dealing with the Micrologix (which does not allow online editing of the MSG rungs) I can alter the sequence logic online as long as I stick with the same control bits for each MSG instruction.


In the program there are 51 MSG instructions. Not all of them are being directed to the problematic DH card, but at least 27 of them are. I think your method sounds like a good solution, as I think you and Ken are correct in that the current method is eating all of the bandwidth/resources. I'm heading out to site tomorrow to gather some info and I'll post anything I find.


Thanks!
 
Just an update, we read through some Knowledgebase articles and found that the limit of cached connections is 32 which I assume is at least partly responsible for the issues we're experiencing. I'm currently working on sequencing each system's messages to execute in order instead of constantly together. I've attached a screenshot of a small snippet of code from the program if anyone wants to take a look. Is there anything else I should add to this logic? There's error logic after the messaging, I'll also attach some screenshots of how that works.


Thanks!

screenshot.PNG screenshot2.PNG screenshot3.PNG
 
Just for a little more info, we were able to reduce the total number of MSG instructions from 51 to 43. Would it be helpful to change some of them to uncached connections to reduce the number to 32? I believe that I read that you can't send cached and uncached connections to the same module, but between the 3 DHRIO cards I think I could change all of the messages going to a single card to uncached. Not sure if I'm understanding that correctly or not.
 
Rockwell Knowledgebase Technote ID 30621 contains a Word document regarding DH+/RIO Message Management in ControlLogix. This has been a help to me in the past.

Edit: I've also discovered this Logix 5000 Controllers Messages PDF from Rockwell that may be of some use.


Hi Ron,


Thanks for the info. I'm going to take a look through later on tonight.


I tried to apply the changes to the logic in the screenshots above and the messages went into a steady error state. My boss said that he thinks that the MSG instruction needs to see a false to true and also cannot be held true (by a .DN bit for example). So I'm going to look into a bit of interlocking in addition to the sequencing to ensure that the .DN bit only registers through a ONS instruction, and then when it is done have an XIO on the .DN bit to reset the MSG.
 
Would it be helpful to change some of them to uncached connections to reduce the number to 32?
Yes, I would think so.
I believe that I read that you can't send cached and uncached connections to the same module
Agreed.
...between the 3 DHRIO cards I think I could change all of the messages going to a single card to uncached. Not sure if I'm understanding that correctly or not.
I'm not sure I understand what you mean by "messages to a single card". I could see that taken as a single DHRIO or a single destination card.

Either way, what I've tried to put into practice is the following through my message manager logic:
1. For any particular destination (PLC, analog module, etc.), execute no more than one MSG at a time.
2. For any Remote I/O adaptor, execute no more than one Block Transfer MSG at a time. (ASB's can only handle 1 at a time, to my knowledge)
3. In general, I cache my Block Transfer messages rather than my inter-PLC messages because the BT updates are more critical and need to occur more often.
 

Similar Topics

Hey guys, We recently picked up a new customer that just finished a controls upgrade with some other systems integrator company. They upgraded to...
Replies
4
Views
1,729
hello, I have a problem with my AN-X2-AB-DHRIO Prosoft module, I can't connect with the module because I lost a microSD card that has firmware and...
Replies
12
Views
345
We have a remove PLC rack that is being used to collect data from older equipment via a 1756-DHRIO module. This module occasionally faults out...
Replies
1
Views
393
I have a 1756-L81e v32 using a DHRIO v7.001 in RIO scanner Driving (7) 1771 ASB Modules @ 115k baud. The Racks RPI are set at 48msecs. This system...
Replies
2
Views
877
I am trying to send some datas from PLC-5 to control logix plc through 1756-DHRIO. When I go online to PLC-5, Message instruction gives error &...
Replies
15
Views
3,305
Back
Top Bottom