GE 90-30 CPU 372 & 374 Plus - Odd Comm Issue

Hi everyone, curious if anyone has ever witnessed the following comm error:



A little history, I had a department PC die, WIN 7. Had to upgrade to WIN 10 for Corporate standards and security. Rack upon upgrade with Wonderware comms started exhibiting comm issues in a cyclical fashion like this error above. Not saying it's to do with Wonderware, just this started with that driver communication hiccupping about every 5 mins cyclically.

That was with my CPU 374 at the time. I swapped this system with a 372 CPU thinking maybe it's the CPU...same problem exhibited. I found it odd that both CPU's would be bad in the same manner, so I upgraded the rack to an RX3I, problem gone.

Current problem is a plating line PC died, same situation...had to upgrade to WIN 10 and new IO drivers don't talk to old GE 90-30 Serial anymore. I ended up transplanting my now spare 374 CPU to the rack and now the same communication hiccup is happening again as the error above shows. At first I thought it was something to do with WIN 10 then I realized this error is exhibited within the GE 90-30 rack itself, so that's not at play here. I suppose it's possible around the same time, both my GE 90-30 CPU's are exhibiting comm failure and showing their age perhaps? I don't think it's firmware either because these were in lines for over a decade working fine until a few months ago. The 372 was a spare but is exhibiting the same issue. No other module firmware's or anything has changed.

Unfortunately, the path forward is to upgrade because it's long overdue but current lead times are three months about for conformal modules (bad plating department, so intend to protect best I can as none of these cabinets last forever in that department, just how it is). Will probably just have to deal with it for a little while until new hardware can be installed.

Just curious if anyone has ever seen this and resolved it or if it's indeed bad hardware like I think it is. Not sure if memory mismatches or any particular oddities in code could cause processing hiccups in the sweep?

How rapidly is WW trying to update its data? It could be a little too fast for the 90-30 CPUs to process every request, periodically requiring clearing the buffer of pending requests.

Yes, I was wondering that. It normally polls its data every 1000ms. I did try moving that up to 10000ms though and it still keeps logging the communication error. Starting to wonder if new OS protocols, firewalls, security software from the company or something is interfering. Also wondering if the port is being interrupted somehow.

The programs and drivers were added to the firewall though and wonderware and its IO cores are on an administrator account, so there shouldn't be any blocks or hiccupping. That's why I'm kind of at a loss. I could try slowing the poll rate down even further though if you think it's worth testing but even back on WIN 7 days, we would us the 1000ms and not had any issues. Got the newest Wonderware IO core patches too, so not like they fixed anything being aware of any compatibility issues...at least yet if that were a concern.

I have run into those LAN errors in the past with no apparent performance issues. Just the nuisance of frequently clearing the errors and the risk of a real error getting lost in the noise.
I agree that a 1000 mS update rate should not cause problems. How well organized are the addresses WW is trying to read? By that I mean, are the addresses reasonably close to each other (fewer requests for data required for a complete update) or are they pretty random?
Also, are there a lot of alarms and a lot of data logged? I'm not entirely sure about how WW operates 'under the hood', but in many HMIs, alarms and logged variables need to be updated regardless of which screen is being displayed. Data that is only shown on a screen that isn't currently displayed doesn't need to be updated.

I was wondering that myself. I inherited the files from our server and my predecessor or I should say the Engineering Manager at the time that programmed the interface. He was doing a ton of scripting and crazy things that should just be handled by the PLC. There were plenty of memory areas not being used...essentially I've done a LOT of cleaning up on these files even with the error exhibiting.

He did spread a lot of items out when they don't need to be. Cards are AI 0-10 then AI 54-X. He does have a lot of gaps in memory. He chose to assign a lot of memory %M locations to 1,000 series, then after a few hundred jumps to 2,000, then 3,000, etc. (probably just for his own peace of mind to associate certain items starting in a certain memory range). It's not as organized or packed as it could be. Therefore probably requesting more memory capacity in the CPU designation as well because of it. I've been slowly cleaning up the nightmare.

You know...now that you say it...you reminded me of a message in the Wonderware local log:



I did note this slow poll mode indication in the log when it seems to fail. I moved the poll rate over to 10000ms and it still exhibited though which to me I said, "That's insane to have to go slower than that.", so I said forget it and moved it back to its former 1000ms and left it alone. Perhaps it's worth testing even slower if that's the case. I could try cleaning up the file more but admittedly with the hardware config and file, it would take some time to clean up due to the Wonderware memory location designations. A LOT has to get updated, basically all redesignated in the HMI and deployed.