ControlLogix 1756-L72 Major Fault Recovery

[kloc5]

Hello,

We have a customer with a 1756-L72 ControlLogix PLC. They have recently got a T01:C62 Fault Code. I am trying to figure out how to recover with the least amount of potential for changes as this is a medical machine.

The HMI buttons indicate that they are not communicating with the PLC. I don't know if this is just due to the major fault, or if the PLC program was lost from RAM altogether.

The PLC has a compactflash card installed, but we didn't have a backup of the program on it (mainly to avoid confusion as to which program is the latest if the main PLC program is lost and the one from the compactflash is loaded.) This fault seems to indicate that the PLC program was saved, but I'm not sure how it is saved. This PLC doesn't have a battery and I believe it uses a capacitor to save the PLC program to non-volatile memory. Is this non-volatile memory where the PLC program is now stored after this major fault? Is this non-volatile memory separate from the compactflash card or is it the compactflash card? How would you restore this? The customer has tried cycling power which didn't help. I also thought that you may be able to clear the fault by flipping the keyswitch from Remote Run to Program-->Run-->Remote.

I believe it also writes an error log to the compactflash card, but I was a little leery to have them pull that out until I knew how this worked on the 1756-Lxx PLCs. Is there any information in the error log in a human readable format that would help us determine the cause of this issue, or is this only for Rockwell to decipher?

I guess after this is back up and running, I can wipe the compactflash card again so there is no confusion as to which is the latest.

Any help would be greatly appreciated.

Thanks!

 

[plvlce]

The most likely causes of a failure to communicate by the HMI are either communication issues (cabling, failed switch, etc), or the program in the PLC has been dropped. Given the existence of a non-recoverable fault I suspect the latter; a fault by itself will not prevent communication. I have personally seen major faults with the HMI still communicating perfectly.

As far as I am aware, no fault will cause the program itself to be written to the storage. To my understanding, the text for T01:C62 is indicating that there *is* a stored user program, not that one was written -- concerning given you believe there is not one. The fault description also seems to distinguish having a 'Secure Digital' card installed rather than merely a 'memory card'; I don't know if this is relevant.

The capacitor/memory you mention is unrelated to the CF card. When the controller shuts off normally, the capacitor maintains power long enough for the current program to be written to user memory so that it can be loaded at next power-up -- this replaces the use of a battery in previous controllers for maintaining the program. If something went wrong with this process I would expect a T01:60/61/62 fault at next power-up (which one depending on memory card status), but other things can cause these faults as well. All the fault really tells you by itself is 'something went badly wrong at power-up.' Diagnostic information should have been written to the memory card, but I don't know if it's human-readable.

For C60 and C61 non-recoverable faults the user memory is wiped. For C62 this is not mentioned in the fault description, but the remedy Rockwell gives is still 'Clear fault and download project'. 'Recoverable' faults can be cleared by switching into Program and back as you describe (of course depending on what caused the issue it may just immediately fault again), but this is definitely a 'non-recoverable' fault and since you don't have a copy on the CF card you will have to download from a backup.

 

[kloc5]

Sorry, previously I said compactflash card. I believe it is actually an SD Card. The SD card has a file structure on it, but most of the folders are empty.

Logix --> Dxxxxx --> Faults -->CurrentApp:
has two files in it with the name of the PLC processor for this machine (not the PLC program). One ended in p5k, and the other was an XML file.

Logix --> Dxxxxx --> Faults -->Log:
has 3 bin files in it

Logix --> Dxxxxx --> Faults:
has an XML file called "Load"

I don't believe a PLC program was put on the SD card during setup as I stated earlier. Is it possible that the PLC program on this card is from when this major fault happened since it is under the "Faults --> CurrentApp" folder? Is there a way to reload it straight from the SD Card? It was interesting that there is a file called "Load" also in the "Faults" folder. Our customer doesn't have a copy of RSLogix5000 on site. Finally, I believe for whatever reason, the files on the SD Card have a date from 1997.

What would be the best course of action here? We have a backup from our last visit, but it wouldn't include any changes, if any, that were made through the HMI.

Thanks!

 

[plvlce]

Sorry, previously I said compactflash card. I believe it is actually an SD Card. The SD card has a file structure on it, but most of the folders are empty.

Logix --> Dxxxxx --> Faults -->CurrentApp:
has two files in it with the name of the PLC processor for this machine (not the PLC program). One ended in p5k, and the other was an XML file.

Logix --> Dxxxxx --> Faults:
has an XML file called "Load"

This is the correct structure for a stored program, although normally it's not in the Fault folder. The p5k file is a stored binary image of the controller, while Load.xml tells it what conditions to load under and points it to the p5k file. See page 26 or so of this document for further details.

Maybe they do save an image on a C62 fault; I don't think I've ever run into the situation to speak from experience.

I don't believe a PLC program was put on the SD card during setup as I stated earlier. Is it possible that the PLC program on this card is from when this major fault happened since it is under the "Faults --> CurrentApp" folder? Is there a way to reload it straight from the SD Card? It was interesting that there is a file called "Load" also in the "Faults" folder. Our customer doesn't have a copy of RSLogix5000 on site. Finally, I believe for whatever reason, the files on the SD Card have a date from 1997.

I've never heard of a program being saved to the card as a result of a fault, but I don't have a better explanation for its presence.

The 1997 date is probably basically meaningless, iirc some controllers had that as a default date (I remember specifically reading about some NSE compactlogix), which is a little concerning if the controller's internal date/time was previously accurate.

What would be the best course of action here? We have a backup from our last visit, but it wouldn't include any changes, if any, that were made through the HMI.

Without Logix present, it's hard to do much.

You should be able to open the Load.XML file and see under what conditions it will load the image, and potentially edit it to ensure a load on power-up (may require moving it out of the Fault folder, I don't know whereall the controller looks for the load file). Worst case it turns out it's already trying to load the image and that's what is immediately triggering your fault, in which case your last visit backup is the best you've got.

See the document I linked earlier for the XML format and folder layout; it's not something I've ever messed with.

 

[kloc5]

Thank you very much for your reply.

I had the customer send me a copy of the XML file with the same name as the PLC processor and it looks like it was set to "USER_INITIATED" to load the .p5k file in the "\Logix\D05FF290\Fault\CurrentApp" folder.

I did find one other piece of information. In all the pictures that I've come across of this PLC from before this major fault occurred, the SD card light was on RED. Therefore, since they were not videos, I'm assuming that it was on solid red. This seems to indicate that the processor couldn't access the SD card which leaves me even more confused as to where the files came from.

I had the customer reinsert the SD card, power up, and the SD card light is solid red and the OK light is flashing red.

So, it seems like there are files from this machine on the SD card that could be loaded back to the PLC, however the PLC doesn't recognize the SD card and probably hasn't for a while...if ever??

I was thinking the best course of action would be to wipe the SD card, so as to not accidentally load its file in the future and to potentially fix the solid RED SD card light on the processor. Then download our backup of the PLC. Thoughts?

Finally, in the manuals I haven't found anything about wiping the SD card. They show how to format a CF card, or how to format an SD card through the software while copying the program to the SD card using the "store" function. Can I just format with windows using FAT (Default) and the file structure would be created when I reinsert it into the PLC? Maybe I should just buy a new one as maybe it is damaged.

Sorry for rambling and thanks for your help!!

 

[JosephMont]

When a PLC faults, it drops the communication thus the reason the HMI is not communicating.

Why don't you just connect to it? Grab a USB printer cable and see if you can clear the fault and what see what it is.

 

[kloc5]

Joseph,

I believe that there is no program in the PLC and that is why it is not communicating with the HMI. I am just trying to find the best way to reload the PLC program to what the customer has been using and to prevent it from reloading an old file from the SD card in the future.

Also, I would like to find out why it happened in the first place.