1756-L81ES Losing Program

[Bering C Sparky]

We finally got off the dock last night just after midnight and already the fun has started.
The 1756-L81ES processor in our fishmeal plant had no program in the controller when I got on shift.
Reading over the Rockwell literature I see no mention of this processor having a battery. Am I correct in that assumption?
The processor is connected to a UPS though.

No one else onboard with access to the software has touched the controller in the past 24 hours, so I can safely rule out someone writing over the program with a blank file. (unless it was done remotely by the contractor that installed it, which I highly doubt)

Reading though knowledge base search results and archived post here on this forum I do see others saying they have had similar issues, although not with this exact controller model number.

Anyone else experience anything similar, and if so were you able to find the cause?

Regards
BCS

 

[plvlce]

Reading over the Rockwell literature I see no mention of this processor having a battery. Am I correct in that assumption?

Technically it indeed does not have a 'battery'. It has a capacitor-based non-removable 'energy storage module' (ESM) that serves a similar function of maintaining the program through power cycles (see page 29 of the controller technical data).

When the controller loses incoming power, the ESM maintains power for the program to be written to nonvolatile storage before the controller actually shuts down fully.

The first thing to look at would be if the error log shows anything. Usually if the program has been dumped some kind of error will have been reported. A failure of the ESM would cause the program to be lost if power was cycled, but is not the only possible cause.

 

[ianingram]

Hi,

Have you seen product notice PN1562? https://rockwellautomation.custhelp.com/app/answers/answer_view/a_id/1131169
Access Level is everyone

I had an experience with a Compactlogix dumping its program, and this notice explained what was happening.

 

[tlf30]

A fellow Alaskan!
Another reason it may have lost the program is if it had major non-recoverable fault (MNRF).
Although if it did it should have been obvious, but one way to check is to look and see if there are dump files on the SD card.

For the 1756-L8xE (and 1756-L8xES) models there are not many things that can cause the program to be lost. You can also check if there is a red light on the controller for an energy module fault.

 

[Bering C Sparky]

Thanks for the replies guys.
The controller was showing a Major Non-Recoverable fault and had no program. I could not go online with the controller or upload from the controller. It only gave me the option to download to it, which I did not because the vendor would surly want to see what is going on remotely before I made any changes. However as is the case most of the time, they are not able get a remote connection over the VSAT.

I was off shift when this took place so cannot really say what, if anything actually happened in the plant itself. Everyone that was on shift says nothing out of the ordinary took place.

I am just getting back on shift and see that the vendor has emailed the .ACD file, so headed down to recover the plant now.
Will have a look to see if there is anything on the SD card and I give the link ianingram posted a read to see if we may be having the same issue.

@tlf30....I am not getting any of that oil money. LOL
I am actually a Floridian (last 40 years anyway) but I spend almost half my life for the past nearly 2 decades in Alaskan waters. Don't really get to see much of mainland Alaska other than the Anchorage airport, Dutch Harbor and whatever I can observe out the window of the plane.

BCS

 

[I_Automation]

First thing to do is replace the rack power supply.


Happens on CLX and SLC600 racks - something in the power supply takes down the backplane and the backup battery if one and on powerup does a default wipe.


Also had a problem on SLC's that a bad analog card would do the same thing.


EDIT: On the CLX it does it while the PLC is powered up and running, the SLC only does it when the power is turned off

 

[Jeev]

Other than Ian's post above about Non-Recoverable Major Faults (We've seen this since v19), I would have a look at how dirty the power is and if it has been browning out. I have had different pieces of hardware do all sorts of things if they are browned out at the wrong time - Usually this is some point during the start-up that does the scary stuff. If the supply is flaky, it may be time for a small online UPS.

 

[Bering C Sparky]

I will get a new power supply and controller ordered, but unfortunately due to the logistics of where we operate it will likely be a month before we have it in hand.

I have the file loaded back into the controller and everything seems to be back to normal with the exception of 2 drives will not reset, I will have a look at that in a bit.

I am not finding an obvious path to retrieving any error codes off the SD card.(If there is one)
Under the Nonvolatile Memory tab in controller properties it says there is no nonvolatile memory image, the dialog box above that is grayed out and there is no name, type, revision or anything else in the "image of nonvolatile memory" dialog box.
Under the alarms tab it says "last cleared state" and nothing else.

Let me know if I am missing something here, but from what I see it would appear there is no SD card.

Thanks for all your input, its been nearly a decade since I used Rockwell software so I am little more than rusty around the edges.

BCS

 

[Bering C Sparky]

Other than Ian's post above about Non-Recoverable Major Faults (We've seen this since v19), I would have a look at how dirty the power is and if it has been browning out. I have had different pieces of hardware do all sorts of things if they are browned out at the wrong time - Usually this is some point during the start-up that does the scary stuff. If the supply is flaky, it may be time for a small online UPS.

It is very possible, I will get the controller onto a better UPS.
But the plant was not running when this happened, we are steaming north and not in production mode yet.

The shipyard also did a terrible job with the cabling on this brand new vessel. I think they picked their electricians up in front of Home Depot each morning or something. Cable is terrible, installation is terrible and my 13 year old granddaughter can make better terminations than most I am finding on here.

On top of that the powers that be decided it would be a good idea to go with 690v as our primary generated voltage and use 690v motors and drives across the entire vessel. (3 separate factories plus all the engineering equipment)
"690v motors and drives are readily available and easy to source" they said.......turns out they are not and everything is months out on back order.
The manufactures are basically building it as we place orders for spare parts.

Its going to be a **** show this season, but we will do the best we can with what we have.

BCS

 

[Jeev]

It is very possible, I will get the controller onto a better UPS.
But the plant was not running when this happened, we are steaming north and not in production mode yet.

"Never attribute to malice that which is adequately explained by stupidity."

Operators have been known on occasion to aggressively cycle power to things Let's hope this was not the case for you.

 

[Bering C Sparky]

"Never attribute to malice that which is adequately explained by stupidity."

Operators have been known on occasion to aggressively cycle power to things Let's hope this was not the case for you.

It is possible, I would not put anything past them.
That explanation would be better than actual power distribution issues though.
We are still shaking the bugs out of this new vessel, so neither voltage regulation issues or operator error would surprise me at all.

BCS

 

[Bering C Sparky]

Well I can rule out brownouts or operator error.
I have been in the control room the entire time since downloading the .ACD file back in the processor.
Was working on a couple of drives, looked up at the HMI and noticed it was not displaying data again. Opened the cabinet and the controller was displaying Port A > the IP address of the controller > Data Storm Detected >Link 1 - 100/Full.

It repeated this same message many times for about 10 minutes, then the processor stopped displaying any message at all for about 30 seconds. Then it started displaying (very, very slowly at a turtles pace) No Project > Port A > Link 1 - 100/Full.

Controller LEDs stayed green this time but the OK led was red the first time I noticed the project file missing from the controller.

This time I also noticed the Allen Bradley Stratix 5700 managed switch was faulted as well.
Rebooted the switch and now the message on the controller is moving really fast but still saying "No Project > Port A . Link 1 - 100/Full"

I can also ping the controller again without timing out or loss of packets.

So possibly a switch issue instead of a controller issue?
Why would it dump the program?
Does the switch have to be managed due to the safety ring? Or can pull the managed switch and install a non-managed switch that I have onboard?

Its almost wake up time in Denmark, so gonna go shoot the vendor an email letting them know what I found so they can try to get parts before their tech flies to Dutch to meet us in a few days.

BCS

 

[Jeev]

Am I understanding this right? You potentially have a DLR without the ports on the switch and/or PLC's EN card set up for DLR.... and/or without a ring supervisor, which equates to a loop in the network? The 5700 is more than managed enough to handle both the DLR and loop. It should have been trimming one of the ports to stop that, which would have been a different and potentially less dramatic problem. Even your garden variety 1756-EN?TR can be set as a ring supervisor.

As for dumping the memory, I have only suspicions. Guardlogix with a compromised network has no CIP Safety (or CIP Security if used) integrity.

 

[Bering C Sparky]

I did not set any of this up, so I really cannot answer a lot of questions about how it is designed. It was just handed over to me a few days ago without as much as a set of complete drawings, IP address list or a kiss goodbye.
The vendors ran off here like we were sinking when they left.
They are supposed to meet us in Dutch and be onboard for the first trip of the season so the pressure is somewhat off me for now.
But I would still like to get a better understanding of what is going on and how to solve the issue myself. If I am going to need parts then I need to get them sourced right away and gold streaked out to Dutch ASAP.

 

[I_Automation]

OT, but how big is the new boat?