L35E repeated major fault

I have a Allen Bradley L35E compactlogix with the following hardware:

[0]1769-L35E (firmware V16.3)
[1]1769-SDN/B DNET_R00_S01
[2]1769-MODULE SP_R00_S02
[3]1769-MODULE SP_R00_S03
[4]1769-IQ16/A SP_R00_S04
[5]1769-IQ16/A SP_R00_S05
[6]1769-IQ16/A SP_R00_S06
[7]1769-IQ16/A SP_R00_S07
[8]1769-IQ16/A SP_R00_S08
[9]1769-OB16/B SP_R00_S09
----second rack----
[10]1769-OB16/B SP_R01_S10
[11]1769-IF41/A SP_R01_S11
[12]1769-IF41/A SP_R01_S12
[13]1769-IF41/A SP_R01_S13
[14]1769-OF4CI/A SP_R01_S14
[15]1769-OF4CI/A SP_R01_S15
[16]1769-OF4CI/A SP_R01_S16

The installation is about ten years old.

Our problems stated when one morning the processor when into major fault showing:
(Type 03) I/O Fault
(Code 16) I/O module connection failed on required module
Port: Backplane
Module: DNET_R00_S01
Error: Connection timed out
Info: <none>

We changed the devicenet card and reset the fault, however it returned again about six hours later. We then changed the processor, again the fault returned. We noticed when the fault occurred all the cards were showing a yellow ‘!’ next to them not just the one mentioned in the error message.

Also the processor keeps reverting back to 1/1/1998 when the fault occurs but not always, it never loses the program. Sometime the fault message will say that AI_R01_S12 is faulty but again all the cards show ‘!’ next to them. We removed the end caps and cleaned everything with contact cleaner to rule this out. We have also tried swapping out the power supplies in each rack.

In an effort to keep the processor running we unticked ‘Major fault on controller if connection fails whilst in run mode’, on all the cards but now when it goes into fault we just get this message:

(Type 03) I/O Fault
(Code 16) I/O module connection failed on required module
Port: Backplane
Module: Local
Error: Connection timed out
Info: <none>

We have ordered a replacement 1769-IF41/A just in case this could be the cause, but otherwise I’m really struggling to understand what is causing this fault. We have support contract with Rockwell but they couldn’t help either. Does anyone know if there’s something I’m missing here?

Welcome to the forum...

Have you checked all the Bus connections?
Are you working in any sort of corrosive environment that could cause issue with the connections?

What did rockwell support suggest - changing modules until the fault did not return?

How often does the fault now appear?

JohnCalderwood said:

Welcome to the forum...

Have you checked all the Bus connections?
Are you working in any sort of corrosive environment that could cause issue with the connections?

What did rockwell support suggest - changing modules until the fault did not return?

How often does the fault now appear?

Click to expand...

Thanks for the reply, I've been through the whole rack with contact cleaner. It's in a panel in I'd say is in a slightly damp location (external door nearby is often open). Rockwell just suggested swapping each card out in turn. I ordered some line filters for the power supply which I'm going to install this morning. The fault occurs roughly every 6-12 hours. I've also added the fault routine attached from an Allen Bradley document

Just to close this in case anyone else has a similar fault, after installing the filters I've not seen a single major fault. I still don't know why it has worked fine for the last ten years without these, but at least it now it works ok.

I've seen AC powered micrologix processors do weird things when we've had bad power quality. With the increased popularity of VSDs and other non - linear loads, the poor old mains is starting to look less and less like a sinusoid. Have you changed any other non linear loads lately?

All my PLC gear is 24VDC these days

thank you for coming back with that updated report ...