How to troubleshoot and reduce communication overhead on Rockwell ControlLogix

Need help. We have a plc that's cpu keeps getting maxed out. We've already upgraded it once. Now we need work on optimize it.
We have over 50 outgoing msg instructions, 60 incoming, and 103 number of ethernet devices (flow meters, drives, etc) I've gone through and tried to make sure everything is cached that can be, only instructions that are currently needed are running, and communication to the same plc happen in the same scan, but I haven't made a dent.
I'm having trouble identifying which instructions are significant. It seems the connections will be consolidated so lots of msgs shouldn't be too big of a problem. Considering Produced & Consumed tags but our team isn't very familiar with them and I believe you have to do a download to modify them, which is a problem. Our IO module RPIs are all set to around 200ms, but that didn't seem to make a difference (from 5ms).
We have a shutdown this weekend and I plan on disabling everything and turning it back on one part at a time to see where the load is really coming from.
Does anyone have any suggestions? The task monitor doesn't have a lot of detail that I can understand, i.e. It's either too summarized or too instant for me to make heads or tales of it. Here is a couple screens from the Task Monitor to shed some light on what I'm seeing.

Add more ENET adapters.

The first thing that I always look at is the Ethernet modules web page. The Diagnostic Overview will have some good information like number of connections, CPU usage, packets per seconds, etc. This information will help try to narrow down what needs to be addressed.

Maybe you can post some screenshots of the Ethernet modules pages and we can take a look at the data.

I agree with dmroeder, my first inclination is too many connections for one card.

So its really not the card. The usage of the card is only about 60%. We also have multiple cards, one for IO, one for Peer-to-Peer, and one for the HMI/Historian. Our problem is even less about communication failures and more about just code changes and browsing the PLC takes forever since the CPU is at 100%.

What is your system overhead time slice set to? May need to increase that. Can you post a pic of that screen?

Something to keep in mind too, when you add an Ethernet device it always defaults to 20ms scan time. If your just monitoring a Flex input for some flow panel proxes then 500ms is plenty fast enough.

Can you test by unplug your HMI/Historian to see if CPU usage go down significantly?

can you go through the different tasks and programs to see in the monitor tab how long they are taking to execute?

If you’re using that many messages I would make sure you read up on the difference between explicit and implicit messages and their affect on the cache. Take a look at this document starting on page 11.

https://literature.rockwellautomation.com/idc/groups/literature/documents/pm/1756-pm012_-en-p.pdf

In that same document head down to page 23 and review how to setup some simple sequencing logic for managing multiple messages.

Are your messages staged so they don't all just automatically retrigger?
You can have 10,000 MSG instructions that will have no impact on communications, as long as there are only 10 or 20 active at a time.


Also, make sure they aren't cached.


Also, keep in mind, generally, WRITES take up at least twice the bandwidth of READS.

From watching the task monitor, I can see that only about 10 or 20 are active at a time. We have 16 PLCs we need to communicate to constantly at least though.