Issue on RIO Rack

I have a SLC 5/03 system with a RIO rack (1747-SN to 1747-ASB). There is only one digital output card with one output that is holding in a remote shutdown relay on another skid. The output is momentarily shutting off, which is shutting down the remote skid, and I have been pulling my hair out trying to isolate the problem. It occurs erratically (sometimes multiple times a day, and has gone a week without a problem). Here is what I have done so far.

The existing circuit on the skid had a pilot light to show that it was losing the enable signal from the PLC RIO rack. I added a latch, so if the remote enable was lost, it would require a pushbutton reset to re-enable. Showed problem was from PLC and not internal.

Replaced remote shutdown relay on the skid. Added a relay in parallel to the Output distribution block at RIO rack. Latched in - Reset with jumper wire, and showed that I was losing power at RIO rack. Moved output to another channel on the card.

Added alarms to Hmi - previously had feedback, but no alarms when it was told to shutdown. - Did not get alarms when problem occurred.

Moved Latched test relay to the power feed to my relay output. - Did not lose power when problem occurred.

Programed a 1 sec watchdog pulse into PLC and used a loop back circuit on a spare output, to a spare input on the RIO. I record the max time difference between the output and the input. Loopback test has been consistent at about 0.12 s and no extra delays when problem occurs or drop outs causing a larger time.

Changed Output Card and added test flags in PLC in case the output was being told to turn off and back on, faster than the HMI could alarm.

Yesterday, I moved the output card to a slot in the remote rack, and changed the dip switch on the RIO adaptor to hold last state. Also moved my latched test relay to the output directly on the card.

I think, that I am getting down to the RIO cards or the rack. I am doubtful that either would cause such a specific problem, especially after I moved the output. There are a number of analog inputs and outputs, as well as digital inputs, and I have not seen any problems with any of these.

Note: The system is powered by a active UPS, so storms or power issues shouldn't be a problem. Also I did not show that I was losing power to the relay card with my test relay.

If anyone has had experience, that may help; it would be greatly appreciated. Site is manned during the day, but they are getting call outs on nights and weekends to re-start this skid. It is a burner oiler heater, so both of us our a little leery of enabling them to start it online, and not on site.

Are you seeing any errors or other problems in the channel diagnostics?
What type of output card is it?
Did the fault return after you set the rack for hold last state?
It is not all that uncommon to have a backplane problem be that specific, but usually a rack fault will occur. It is possible for the system to be set up so that a rack fault does not fault the main PLC, matter of fact, I think you have to add logic to trap rack errors with a SLC scanner, at least I seem to recall adding a lot of error trapping to logic done by others.

First place I would look is for loose wire, and capture minimum volts on the supply terminal for the output and nail it down to that particular output "blinking"...prove it is the output turning off and not the AC common terminal first. I think you may have done that, reading your list of steps, but I am not clear on what changed, if anything between steps.

Thanks Okie,

I haven't checked channel diagnostics... I have worked on a few RIO systems, but do not have a lot of experience with them (seems they are normally fairly robust). I'll check into it.

It is an 8 Channel isolated relay output (OX8)

Changing the hold last state is the last thing I did... along with changing the slot that the output card was in. It's only been a couple of days, so not sure yet.

I will check for ladder for remote rack fault logic.... When the problem does occur, the output does turn back on, without any action from the operator.

Most of the steps have been to isolate the problem. I have verified that the output is shutting off, and that the power feeding the output is not. I have moved the output on the same card, and then changed the card. The last thing I did, was to change the hold last output dip switch and move the card to another slot. I'm waiting to see what happens.

You can add logic to monitor the channel status, using the S:XX bits for the channel in question, I believe you can do this down to each node as well, go to the 'Processor Status' and look at the I/O, Chan0, and Chan1 Tabs.

It's pretty handy for troubleshooting, but can add quite a few tags and alarms to HMI applications, In the case of remote I/O I can see quite a few alarms when we get a power outages or blips. Our processors are on battery backups but most of the remote I/O is not.

No Calls, yesterday or today, so I think that is a good sign. Either moving the card in the rack, or changing the dip switch to hold last state seems to have helped the issue. We are using a 1747-SN scanner in the main rack, and a 1747-ASB adaptor in the remote rack. So, any communication issues are going to be a little more difficult, than just looking in the status bits.