Now questioning his knowledge, I have checked Rockwell Automation Publication 1769-TD005K-EN-P - January 2017, pg 9, line 5, column 3, Power supply distance rating for the L30ER compact logix controller, " 1769 Compact I/O modules: 4…8, depending on module".
This document specifies a distance rating of four for the CPU, ("Controller and 1769-SDN: 4"), clarified in the next column over as "3 I/O modules between controller and power supply". I/O modules have distance ratings varying from 4-8 as noted.
This being the case, your setup is not technically correct - there should be only three modules between CPU and PS on your main chassis. I think it's
unlikely that this is related to your problem, but if you get RA tech support involved, that'll likely be the first thing they'll point to. The good news is that moving the power supply does not affect your slot numbering, so if your wiring length allows, you could move it one slot to the left without affecting anything, and at least you can rule that part of the puzzle out. Your OW16's have a distance rating of 8, so no issues with the modules on the right of the PS.
Publication 1769-TD008A-EN-P - January 2010 for the 1769 PA4, pg 7, line 14, column 3, "8 I/O modules can be connected on either side of the power supply for a maximum of 16 modules"
True, although not quite the whole story. You couldn't connect 8x OB32's to one side of a power supply, because they only have a distance rating of 6. You could connect 2x OW16's and 6x OB32's, as long as you put the OB32's closest to the PS and the OW16's at the end. In any case, all of your I/O modules are within their respective power supply distance rating, so it's unlikely that this is the cause of your problem.
Rockwell Automation Publication 1769-TD007D-EN-P - May 2014, pg 6 Technical Specifications - 1768 and 1769 EtherNet/IP Modules, the 1769 AENTR doesn't have a distance rating, just a PS recommendation, 1769 PA4
This document states on page 37: "The adapter has a distance rating of five, therefore the adapter must be within five modules of the power supply." As before, this means only four modules are permitted between the AENTR and the power supply - and you have five. This could quite likely be the cause of your problems - and again, even if it's not, RA will be no help until you make it right either way. Once again, you should be able to move the PS without upsetting anything else - all your modules will still be within their required distance rating.
Things we have tried and checked today, uncheck the “major fault” box on the connections tab of the AENTR, and lower all the RPI’s to 2ms on the remote side, and checked to make sure the port speed on the controller and the AENTR are the same, and lower them if possible. I haven't heard back yet on the impact these changes made, if any.
Unchecking the major fault will stop the PLC from faulting out, but does not address the cause of the problem. It only masks it. What are the consequences of the machine trying to run with three I/O modules out of action?
Lowering the RPI will make things worse if anything - if you're already struggling to communicate with the cards, trying to do it faster won't help at all.
My approach from here would be:
1. Restore all settings to what they were previously to start from a level playing field
2. Move power supply modules to correct locations and monitor
3. If no improvement, make sure all bus connecting plugs/levers are clean, dry, corrosion free, and properly engaged. Unplug and replug a few times and make sure it feel solidly engaged
4. If no improvement, swap some of the suspect modules with known good ones from elsewhere in the chassis, to see if the problem follows the module or the position
5. Go to the pub and drink beer until you come up with another idea that's so crazy it just might work