AB ControlLogix Crashes - Program Lost

We have a newly installed AB ControlLogix PLC that crashes after running a few hours and the program is completely deleted.
Each time this happens it is necessary to download the entire program and restart the PLC. We don't know if the problem is hardware or software and are looking for advice.

What we have done so far:
Replaced the CPU module - we still have the problem
Obtaining a new Power Supply module to test. Not yet available.

Can the problem be software related? Is there any PLC instructions that could wipe out the program?

Somebody has suggested that it is a feature (read BUG) of the latest AB PLC's to wipe the program in the event of catastrophic failure rather than risking continuing to control. Any ideas?

How can we diagnose this problem?

First thing I would check is the firmware. You can try to flash it with the latest updated firmware.

Not sure about Controllogix PLCS.
I had a PLC5/40 do this when there were incoming power upsets. Installed a UPS and this solved the problem.

Shot in the Dark

Found these:
http://domino.automation.rockwell.com/applications/kb/RAKB.nsf/0/6A82C7F079B628A885256AFB00688717?OpenDocument
http://domino.automation.rockwell.com/applications/kb/RAKB.nsf/0/814C9AE8B85C016885257188005DC5BA?OpenDocument

Logix controller marks program as "invalid" if it determined that it is not safe to execute program anylonger.
This is done by design.
This will prevent catastrofic actions as I/O remains in a safe state.

The answer to the Logix memory loss:
Contact techsupport for a fault log extraction utility.
Extracted log file along with your project file needs to be sent to techsupport for review.

Make sure that you are running the latest firmware revision for your major version. Based on your description this should be easily duplicated by RA.

It's is almost certainly a power issue. Check your incoming power and I bet you will find the source of the problem.

Also, if you have a bunch of unsuppressed contacts all opening at once, an arc could be getting onto the power circuit and getting back to the controller and corrupting memory. I have duplicated this repeatedly on a test bench.

OG

maybe Mickey Mouse has right - I will tray with UPS; after, consider another possibilities

What`s about working temperature outside in the cabinet and inside PLC? can you feel more heat than other PLC ???

Don't waste your money and time - UPS will not help. This is not power up issue. This is most likely FW action that needs to be looked at.

Extract fault log!

IF you tell us what controller and FW rev you have then we can look at rev history to see if it is fixed.

Do you have a Message function in your code? If it is configured to cause a major fault, it will wipe your program off.

I hate these kinds of problems. A manufacturers point of view.

When booting up the PLC should do a CRC check on the program to make sure it isn't corrupted. That is what we do. I would bet Rockwell does something similar. In the past we had cases where the power supplies did not start up with stable voltages. In these cases the controller would be doing a CRC check when a brown out occurs just after startup. This would cause the CRC to fail. Since the controller thought the CRC was bad it would erase memory and not flash. In our case one could turn the power off and turn it on again and if the CRC with the flash was good this time then everything can proceed. However, if the customer thought his program was truly gone he would entry a new program over the old and everything would be lost.

We had to add more code to check if a brown out was detected to restart or until the brown out bit goes away. Then we can continue with the CRC. The hard part was reproducing the problem. I think I turned on and off power to one of our controllers for 20 minutes before I could reproduce the brown out conditions. I had to make the power fail during the time the CRC was being calculated. Then I was able to reproduce the problem for the hardware engineers once. I could not reproduce the problem again but I didn't try too hard because the hardware engineers had already seen the problem. Then the brainstorming begins.

The point is that these problems can occur during startup. It is usually due to the power going on then off then on again. I know when I was turning the power off and on for twenty minutes I turned the power on and off many more times than the controller usually get turn on and off in a life time.

Another problem that can occurs is break down across the substrate of the memory chips. Memory chips may get old and die now because they are made so small. Actually the rams don't just fail. A bit would change from time to time to time which makes this a difficult problem to find. Sometimes this would cause a checksum to fail. We bought a batch of memory chips from a well know supplier and they started to go bad after 1 to 3 years. One can not detect these problems in the burn in processes. This one cost us a lot of money and was very inconvenient for our customers.

This problem was similar to the more publicized note book battery fiasco where the note book manufactures have laptops that burst into flames because they are using Sony batteries. So is this Apple's or Dell's fault?

If you lose the program it may still be in flash. Turn off the PLC and start it again. The CRC may succeed this time and you can recover your program. If this works good but there is still a hardware problem. It is probably too late for John Gaunt.

It seems to be a Program problem.

Thanks for all your responses, it gives me lots to think about.

Anyway, in the meantime, we have disabled all the uncommissioned PLC functions and are just running the basic plant operating cycle that has been commissioned. Now the problem has gone away. We assume there is something in the more sophisticated code that causes or more likely triggers the problem. The code we have disabled includes several PID control loops and a fair bit of mathematics.

It seems to me to be most "unfriendly" of AB if they do deliberatly wipe out the program in the event of some invalid sequence of instructions. However, if this is the case, we now have to try to locate the offending code. This might not be easy as the plant is in production and they have just suffered 3 days of chaos due to this problem.

Any ideas as to what instructions could cause this problem would be appreciated.

I am not aware of any instruction that will cause the processor to wipe it's memory. It is a design function of most PLCs that if their program becomes corrupt they must dump the unsafe program. I have watched as a customer swapped out a Symax for a PLC-5 only to see the exact same memory dump occur. Only then were they convinced that it wasn't the hardware or logic. Finally they fixed a whole bunch of unsuppressed contacts and their problem went away.

As Control_Conn mentions, there is an extract utility that can help discover what the processor was doing when the fault occurs.

OG

I am not aware of any math instructions that will cause memory loss. In some versions unlatching DN or ER bits of the MSG instruction may cause this. Again without knowing what version you running it is hard to say.
Is controller does it frequently then RA should be able to pinpoint to a specific place in your program, but they will need fault log file.

I had a similar problem on a MicroLogix controller a few months ago while on a service call. The controller would fault due to the program being wiped out. We loaded the program back into the processor 3 times in a half hour period. We tried changing the base unit and the processor. Still no luck. Then I saw the pattern, it was every time the machine got to the last step in its cycle. There is a limit switch that makes on the last step of the cycle (when a cylinder extends) that was shorting to ground. Sure enough every time that short on the input occured we would lose the program. I couldn't believe it, never seen that happen before but it would happen every single time.