1756 dh-rio card display

I recently(about an hour ago) found one of our(ControlLogix) 1756 DH-RIO cards not functioning on channel A(channel B was fine). When I got to the Rack, the display was on channel A, and showed "28F1". I unplugged the card and plugged it back in, and everything is working for now....... Any ideas?

re-flash it

I had problems with some of the earlier versions of this card.
For example the diagnostic counters did not work properly.
There were also some issues with the amount of data being passed through it. Once I re-flashed to the latest Version these problems
got fixed. So when in doubt I suggest that you re-flash.
One important thing here to remember is that you need to follow the compatibility matrix as it applies to your specific hardware / software. All processors and intelligent modules in the system need to be on the same firmware level.

Certainly "28F1" isn't a standard error code. I would have expected the whole module to have shut down if there was a firmware or OS crash. You will see a hex code in the case of a firmware crash but "28F1" doesn't correspond with anything I've heard of.

When you say "the display was on Channel A", what do you mean ? The display usually alternates between channel A status and channel B status.

There is the usual litany of diagnostic questions:

1. What were the states of the three LEDs on the module: CHA, CHB, and OK ?

2. What exactly was showing on the text display. Was it only "28F1" or was other information scrolling ?

3. What Series and firmware level is the DHRIO ?

4. What mode is each channel in (DH+ or RIO) ?

and those are just the first few questions.

Jiri,
Thanks for the input. I'll look at the firmware versions later today.

Ken,
Here is the list of what I know:
Upon re-reading my original post, dyslexia strikes again(this was the last item I worked on that day, and I started fairly early in the morning). Channel B was not functioning, ch A was fine.
1. I'm afraid I didn't write down the led status bits, so I'm not absolutely sure what the status lights were doing, but I believe A was on, B was off, OK was on, and nothing was changing during the period I watched it.

2. I watched the test display for about 30 seconds, and it never changed. It could have said 2B01 instead of 2801 perhaps?

3. 1756-DHRIO/B, product type 12, product code 18(not sure what those two mean), and revision 2.21. The card is in slot 4 on a 10 slot rack.

4. Both channels are dh+.

5. The cards are used for collecting run-time information for two different dh+ segments, and to pass that information to the controllogix processor for collection by an oracle database through rssql.

6. Recently(5 days ago) our contractors set up a passthrough messaging to send information between one of the processors(5/04) on this card(ch B, node 2) and a processor on a different card(slot 5, ch A, node 4) to support some changes in our conveyor layout. Since we haven't had any problems previously, this is probably the source of our issue. The contractors have since hardwired in relays to take the place of the messaging, but we are still trying to figure out what happened for future reference.

7. I wish now(20-20 hindsight) that I had taken a look with the diagnostics through rslinx. At the time however, I just found out that the comms weren't working, and went out to the processor to find out why.

Okay, if there was no other data then you're getting a firmware crash code. Only RA Technical Support is going to be able to tell you exactly what part of the firmware crashed (evidently it's one of the routines that handles Channel B), and then you can try to figure out exactly why.

1756-DHRIO/B
Product type 12 = Communications Interface
Product code 18 = 1756-DHRIO
Revision 2.21

I think that's the most recent firmware for the Series B hardware.

Useful diagnostics about the activity inside the module are available from the "Module Statistics" (right-click on the DHRIO in an RSWho browse) feature in RSLinx. This will help you get an idea of how busy the DHRIO is on each channel and maybe that will help.

Thanks Ken. I'll contact them(probably tomorrow), and find out what they have to say.
Do the diagnostic counters automatically reset by removing the card? Or is the only way to reset those through rslinx? I'm going to reset all the counters to get a clean start, but I'm curious how long the information currently displayed has been accumulating.

Diagnostic counters in Logix modules generally reset when the module is rebooted, which is usually at powerup.

You can reset and lock the counters from the RSLinx Module Statistics page, and I think you might be able to do so programmatically from the Logix controller too.

I think 28F1 error was an old firmware bug when trying to send MSG with both ports set as DH+ and both networks have the same node target address. It sounds familiar...
Ken, can you confirm this? Anything in "AB internal" KB?

Based on that description, I did find a few instances of a bug in the ControlLogix CPU firmware that resulted in that error on the DHRIO when both channels have the same DH+ node number and several other coincidences exist as well (same target data table, simultaneous messages, etc).

I'm not convinced that's Tom's problem because he describes this as beginning when a channel-to-channel bridged message connection was added to the system. Maybe it's a combination of things.

Either way, RA Technical Support is going to know more about it than we can get.

To narrow this down; Tom, are Channel A and Channel B set for the same DH+ Node Number on separate DH+ networks ?

No, I'm afraid not. Channel A is set for 42, and Channel B is set for 20. However, since it looks like it just went out again (while I was checking to see which node numbers were what), I should be able to get a little more information now.

Channel B Statistics:
35903 messages sent, 34803 received, 22430 commands sent, 12370 commands received, 12370 replies sent, 22430 replies received.
79 transmit retries exhausted, 60 timeout w/no ack received
DH+ interface state: quit
Items preventing normal state: blank(the working channel shows :none
Module state: run

Channel A: On steady
Channel B: Off
OK: On steady
Display: 28F1, not changing

I think this happening when you have same message target address on both networks, not necessary both DHRIO channels use the same address.
Let say you are sending message to the device node 22 on the "B" channel. If another device with node 22 also exists on the network "A" DHRIO will fault time to time.
May be it happen only if device A-22 also involved in messaging and messages crossed some how.

Techsupport will definatly have info about this, try to call them, but prepare to be asked for a support contract

We do have about 10 different node addresses overlapping when you compare network to network, and every single one of them are messaging to the CLX processor. We do have a support contract for the plant(M-F, 9-5), and actually, for this piece of machinery, it is a 24-7 contract(from somewhere in Canada). I just haven't called in a ticket yet because after the relays were added, it is no longer a critical item, so I have the time to look into it myself.

I found some references to the bug that Control_Conn is referring to, and it was supposedly fixed in Revision 12 of the Logix firmware. What revision are you running ?

Error 28F1:

2 = Channel B
8 = DH+ Interface Task
F1 = Connection-related fault

Do you have any diagnostic counters on your MSG instructions ? If you can figure out which MSG instruction is responsible for those 60 timeout errors, you can examine the PLC/SLC associated with that message.

I've heard that this sort of fault can happen if a node goes offline right in the middle of a message. Maybe there's been some damage to one of the DH+ nodes or its wiring, though I would expect more noise-related counters to be incrementing. Remember, it's not the absolute number in those counters that matters, but the rate at which that number is increasing.

Ummmmm.. Looks like revision 11.28. I'm going to disable the passthrough message statements, reset the card again, and then go check for any loose connections on that network segment.
I'm afraid that there aren't any diagnostic counters setup in the program yet.

After disabling the passthrough messaging, and checking the connections on 12 different processors, I re-arranged a couple of wires ie.. shield wire touching the white wire, someone stripped too much insulation off the connector:
0 transmit retries exhausted and 0 timeout w/no ack received after 35000 messages sent.
I re-enabled the passthrough messaging, and after resetting the counters, it is now at 30,100 messages sent with 0 errors continuing to happen.

Ken and Control_Conn and Jiri, thanks for all your help! I'm going to continue monitoring this for the next few days, but hopefully this is a done deal(as soon as I address a 'minor'!!?? training issue with wire-stripping).
Thanks again!