Compact GuardLogix Auto-Reboot No Logged Errors

[DamianInRochester]

I have an installation with a Compact GuardLogix 5380 CPU [5069-L306ERS2] that is experiencing a strange issue I have never encountered before. It is controlling a network with about 8 other nodes, and is an application we have many of in the field almost identical, except we recently switched from Kinetix 300 drives to Kinetix5100.

The CPU will run for about 5 minutes and then will get an error that causes the OK led to go red. It will lose ethernet communication during this time and not communicate while in error. It will remain in error for several minutes, and then eventually auto-recover, go back into run mode, and begin operating again. It just continuously does this over and over again.

When in error, all other nodes on the network will ping except the CPU. When online with the CPU, no ethernet errors of any kind are logged.

No major or minor faults are logged. I added code to the controller fault handler with the hope of logging faults, but they all come up zero once communication is re-stablished.

I am debugging this remotely via a tunnel setup by a Tosibox. The network is otherwise isolated from external connection.

Anyone have any ideas on how I can determine what the actual fault is?

 

[dmroeder]

Diagnosing remotely is going to be difficult. Anyone on site that can lend a hand?

If I were there, and I happened to have a managed switch, I'd setup port mirroring so that I could wireshark traffic going to the controller. Then I'd see if the last packet sent consistently caused this.

 

[DamianInRochester]

Diagnosing remotely is going to be difficult. Anyone on site that can lend a hand?

If I were there, and I happened to have a managed switch, I'd setup port mirroring so that I could wireshark traffic going to the controller. Then I'd see if the last packet sent consistently caused this.

It uses unmanaged Stratix 2000 [1783-US8T] switches. The customers diagnostic abilities are limited. I am not even sure if it is an Ethernet related fault, or just a error in the CPU that causes it not to be able to communicate while in error.

 

[dmroeder]

Is there a SD card configured to load the program on corrupt memory or something like that? Just thinking out loud, maybe the controller is in fact crashing, you lose the ability to get the fault information because the program is reloaded from the SD card?

 

[plvlce]

Is there a SD card configured to load the program on corrupt memory or something like that? Just thinking out loud, maybe the controller is in fact crashing, you lose the ability to get the fault information because the program is reloaded from the SD card?

Controller resetting for whatever reason is my first thought as well -- the OK led is red during power-up diagnostics.

 

[joseph_e2]

It actually sounds like a power issue to me, but I've never worked with a Compact GuardLogix before (my first one is sealed up in its box right now). I just tested it with my 5370 (1769-L33ER) by cycling power. All LEDs go off immediately with the power, with the OK light flashing red very briefly. As soon as power comes back on, OK comes on red while it reboots. During that time, it can't be pinged or otherwise reached until it boots up. Then it takes off and works.

 

[DamianInRochester]

The controller is not setup to reload the program from the SD card.

The customer has also sent me a video of the CPU entering into the fault condition. It doesn't appear to be a reboot by an interruption in power. It is going through its diagnostics and then the OK light goes solid red.

Some other side notes:

They currently have a second CPU in use. I mailed them a preloaded replacement of the PLC CPU. Swapping out the processor had no effect.

The customer themselves installed a larger power supply (on their own accord trying to figure it out themselves). Nothing changed.

 

[Steve Etter]

Are there any 5069-OW16 cards in the local rack? I had something similar happen (not always auto-recovering and not a safety PLC), but that was my culprit.

 

[DamianInRochester]

Rack Layout Is:

5069-L306ERS2
5069-IB16
5069-OB16
5069-IB16
5069-OB16
5069-IB16S
5069-OBV8S

 

[Steve Etter]

A shot in the dark, but are the MOD and SA power the same or from two different sources? If already different, are their commons connected?

 

[DamianInRochester]

A shot in the dark, but are the MOD and SA power the same or from two different sources? If already different, are their commons connected?

Same source

 

[Steve Etter]

Separating them would be my next step.

 

[DamianInRochester]

What can cause the CPU to fault, the OK light to go red, yet no major or minor faults are in the fault log?

 

[MikeyN]

That sounds like the same process a PLC would go through if it were powering up. I believe that is why Steve suggested separating the MOD and SA power, to be sure that nothing was dragging the power supply down. Alternatively, is the equipment prone to excessive vibration such as on a press or similar equipment? Have you verified that all of your connections are tight and properly seated in all of the terminal strips? We had a similar issue with a piece of equipment a few years ago. We finally found a wire that looked like it was connected properly, but after tugging on the wire we discovered the wire was broken at the insulation and was just barely making enough contact to keep the remote IO block powered up.

 

[DamianInRochester]

Separating the power made no difference.
There is video of it going into the fault, not on a reboot.
I am onsite and have smack and jiggled everything as much as I dare. If it is a loose connection on a connector or backplane then that should have triggered it.