CompactLogix L35E Communication I/O Fault

We've been having an intermittent fault which logs this minor fault:
(Type 03) I/O Fault
(Code 93) I/O task overlap detected.

The obvious immediate symptom is the HMI loses communication with the PLC. I’ve also been able to note that there is no available communication to the PLC; Who Active cannot see it, RSLogix can’t go online and I can’t even ping it. The HMI seems fine and communication to that is active.

The machine keeps running briefly before an E-stop is usually pressed until the operators can cycle the power to the PLC, which seems to fix the problem (though some startup states are different than they were when running just before the fault; notably two devices are on manual). I can’t confirm if the PLC freezes and the outputs do not change, or if control continues and Ethernet comms is the only thing lost, but I believe it is the former based on operator info.

Anyone got any info on these faults or symptoms?

Additional info:
PLC: 1769-L35E CompactLogix5335E Controller
Rev: 20.18
Less than half memory in use
A hand full of Ethernet port Errors In (seen in picture)

These sorts of things can be a little difficult to solve over the internet, as much details about the system you can provide will be helpful (number of devices, what types of devices, switches involved, baud rate settings, cabling, etc).

The typical things are trying to connect too many devices to the L35E, mismatch baud rate settings between devices, overwhelming the port with too much data being transferred (maybe this because of the I/O task overlaps).

The L35E has a web page that will provide some additional information. Just type it's IP address into a web browser. One important thing with it might be utilization of the ethernet port.

You need to look on the controllers webpage (just type in the IP address) and look at the Ethernet statistics. Pay attention to CIP connections, those controllers don't offer a lot of head room for Ethernet connections so you could be maxing that out. I believe it's 32 connections for that processor.

Could be RPI issues to your IO modules as well, if you set your RPI to fast then it might not be able to keep up. Default is usually 10ms or 20ms? Plenty fast so turning it down to say 40ms, 80ms or more might take care of the issues.

Need more information about your setup, screenshot of your hardware tree would be beneficial as well as current RPI settings. Screen shot of the webpage too.

I've attached a few screenshots, thanks for direction in where to get more info on the network connection.

All of the In Errors are CRC errors.

The device at 10.50.1.51 is a PVPlus 1500
The device at 10.50.1.101 is a PC running WinCC for some logging functions

All modules have an RPI setting of 20ms excluding the AI module which is 100ms (and an RTS of 500ms).
I can't modify this while running, so I think the earliest would be Friday.

I think the scan rate is in the low single digits of milliseconds, but I don't have a great way of confirming this, unless someone can help with that.

edit: It might be useful to know that only the DeviceNet module and the Ethernet Port do NOT have the "Major Fault On Controller If Conection Fails While in Run Mode" checked. As the faults originally mentioned were in the Minor Faults section and we lose Ethernet comms, I expected the problem to be on the Ethernet Port.

how many TASKS are you using - and how are they set up? ...

in particular - do you have any "heavy load" tasks set up for a Priority of LESS THAN or EQUAL TO 6? ...

are you allowed to post the ENTIRE project file? (ACD extension) ... (you'll have to zip it first - forum rule) ...

Check if speed and duplex is the same on all ethernet devices and its matching switch ports, especially if they are set to auto-negotiate.

Sometimes auto-negotiate option doesn't work well and the result is similar to what you are experiencing.

Thanks for the help and suggestions.

Today I have:
- Increased the System Overhead Time Slice to 35%
- Increased the DeviceNet module RPI to 30ms
- Increased all other RPI's to 25ms


The main task has a Max scan time of 20.18ms and a Max interval of 20.52ms.
The only other task is priority 10 and runs every 20ms with a max scan of 1.3ms (it's normally 0.2, but has quite a few peaks of around 0.7ms).
Sorry, I'm not able to post the project, thanks for the offer to look at it though.

Ethernet is set to 100Mbs, Full Duplex. This is the same as everything communicating with the PLC except the programming computer, but this has never happened while connected.


I'll report back next week. If this doesn't fix the problem, I'll get a bit more drastic with the RPI settings, there just a couple of time critical inputs I need to monitor, so we might have to move them to one card and prioritise that card.

Thanks.

An additional information here:

TN493328 - CompactLogix minor fault Type 3 Code 93 (Access Level: Everyone)