1756-L81ES Losing Program

[Bering C Sparky]

Grounding? And/or surge protection on inductive loads?

I would certainly buy into that as the cable installation is sub par.
But this is happening without anything in the plant running. We are steaming from Seattle to Dutch Harbor and not in production.

BCS

 

[dmroeder]

I'm doing my best to follow along here. I saw some talk about network storm, so we have the possibility of a network issue. I'm pretty sure that when a safety controller faults, you are required to download the program, so it may not be actually dumping the program.

Are you able to share any information, network diagrams, program maybe? I'm willing to give a bit of my time, DM me.

 

[kekrahulik]

I've had similar issues for the last 8 years with an L72 standard processor losing it's brains. No one has ever seen the processor while it happened - so your observation on data storm is interesting.

On AB's advice, the first thing we did was replace Ethernet cards.
Then we replaced the processor.
Then we flashed from v20 to v21.
Then we replaced the power supply.
We finally replaced the chassis. We had been experiencing the pain several times a month, so after a few months we thought it must be fixed. We've never been that bad, but it has happened several times (maybe 1-2 times per year) since.

Please keep us updated on your findings

EDIT: my processor faults. program is erased.

 

[dmroeder]

While "Data Storm Detected" might not be the root of the problem, it definitely is a problem that needs to be addressed. This is minor fault, but since Safety I/O is involved, I could see it causing bigger problems. The cause of this is usually misconfigured DLR.

It's pretty easy to get DLR wrong. The first thing is making sure the loop consistently goes port 1 in, port 2 out. You can't have that swapped anywhere in the ring. You'll want to document baud rates, there can't be any mismatches.

I think you have a stratix 5700 10 port in the ring? The manual says you should use either ports Fa 1/7 and 8, or Gi 1/1 and 2. For the ring. They also recommend the port macro "Multiport automation device" on the DLR ports.

 

[kekrahulik]

For what it's worth, I have no safety and no rings.

 

[Timbert]

I will throw this out there. I had this issue because the safety partner of a GuardLogix processor was not seated fully in the rack.

The two processors disagreed, major non-recoverable fault, then the program gets erased.

The other suggestion would be a Task Watchdog Expired. If there is a data storm it could interrupt the processor long enough for the safety task watchdog to expire.

Or related to that I've seen this happen when the processor was simply task saturated, average 95% CPU usage. Occasionally it would go over 100% which would trigger Task Watchdog Expired as well. Of course this was with an older 1756-L63S.

 

[5618]

A short story to support the broadcast storm faulting the PLC: we had a non-safety L81 that messaged 12 Micro820 from an ENBT. I know the installers duplicated a M820 IP at least once during commissioning without that being a big deal. Fixing shortcomings after commissioning, I swapped out five ENBT to four EN2T and moved this M820 network to the on-board port.

Shortly after that, I fat-fingered a download to one of those M820 and gave it a duplicate IP with another M820 on that network. By the time I went out to the M820 and fixed it locally, the L81 was down for a major fault. I didn't capture the fault code, but I learned at that time that using the on-board port has less forgiveness than using a ENxT.

 

[James Mcquade]

Is the plc on a ups?
if it is, is it a true sine wave output?
a plant that ran siemens had all their plc's on ups systems and kept loosing the system (power supply, plc program) and removed the ups from the panels. their problem went away. the ups units did not out put a true sinewave.
any chance of getting the plc power supply tied into a constant voltage transformer?
james