Controllogix L72/A randomly cycling power

[jthompson20]

Hey everyone,

I've tried the search function for similar issues but couldn't find any relevant info.

Issue: I have a Controllogix 1756-L72/A FW:21.3 in a 17 slot backplane that controls our product conveying system that will cycle power and was losing its program.

Series of events:
*Tuesday or Wednesday morning one of our large 50HP conveying blowers blew a 400 amp fuse going into the drive which is switched on by a 1756-OA16 output 0 in slot 11. Inspected system, all seemed well so we replaced the fuse and started production again.

*Wednesday night the controller lost its program and I was called in. Major fault T01-C60 was present. I loaded the program and away she went without issue until Thursday night, same thing. I spent both nights babysitting the system to minimize downtime.

(Type 01) Power-up Fault
(Code 60) Non-recoverable Fault.
Fault Information Dump:
Info[0]: f4078d84
Info[1]: 0000007e
Info[2]: 00000000
Info[3]: f464e798

*Friday night we only ran half a shift so i had time to go in a troubleshoot. What I found was the SD card being used for NVM was shot. The controller wouldn't recognize it, flashing red twice then nothing. Controller properties NVM showed no card present. Replaced the card and tested. That issue has been resolved. For good measure I replaced 1756-OA16 connected to the drive even though the output is isolated with a relay.

The ESM is bad and im waiting on the replacement. I confirmed by cycling the power at different tine-off intervals and with each cycle, it loses the program and has to load from the SD card. Worth noting the controller is connected to a working UPS so the power issue is within controller.

They will be running full 24 hour production all week starting tomorrow so ill be able to provide more info as it becomes available. I suspect it will happen again. Its like pulling teeth trying to find any correlation between what was being done in the system at the time of failure except the times it happened. 7:00PM-ish Wed and 8:00PM-ish. I ask the night shift guys and all I get is "no idea. I wasn't using it" I'll note that nothing in the system is time activated. Operators call for product from the system as they need it.

Questions:

* Has anyone ran into a similar issue or could point me in the right direction?
* Has anyone had an SD card (1784-SD1) go bad around the same time their ESM dies out? We had a 9 hours a plant wide power outage a few weeks back and the system recovered without issue after the UPS depleted.
* I should know this one but... Any concerns with swapping a 1756-OA16 FW 2.6 with a newer FW3.003? The controller is set to compatibility mode so it should be fine. It doesn't show any errors or faults in the module properties, only that its a mismatch.

Thanks,
Jeremy

 

[Jlandwerlen]

I honestly didn't read through all of your post, but in my experience program dumps are usually caused from firmware issues or I've seen ground issues and power spikes cause problems.

If you can flash firmware to the highest possible I would start there.

 

[jthompson20]

I honestly did read through all of your post, but in my experience program dumps are usually caused from firmware issues or I've seen ground issues and power spikes cause problems.

If you can flash firmware to the highest possible I would start there.

Thanks. I wont rule out the FW being part of the issue and it just didn't come to light until the NVM died. Night shift maintenance is quick to power cycle and go on about their business without looking into the issue. Ill update it to the highest possible between shifts tomorrow.

Grounding is good, one of the first things I checked.

As for power spikes, I hope not. That'll be a lot of fun to track down.

 

[Jlandwerlen]

I had one customer that would repeatedly lose their program. It seemed like it would always happen on late shift and when they were welding in the vicinity. However, once they flashed firmware it never happened again.

I think Rockwell has a utility you can run to determine why the program was dumped. However, it sounds like maintenance will probably not use it.

 

[I_Automation]

When this happens on CLX and SLC racks the first thing I do is replace the power supply for the rack. 98% of the time this fixes it, the other 2% on SLC racks it frequently is an bad analog card taking down the rack.