FactoryTalk HMI dual redundant servers switching with McAfee

[martinshaw]

Hi all
I have a couple of virtual servers with Windows 2016 Standard operating system. The servers are configured as dual redundant FactoryTalk HMI servers (FT View Site Edition V11.00).


Each morning at approximately 10:02, the servers switch over from ACTIVE to STANDBY, so for example, if server01 is ACTIVE, it switches to STANDBY and server02 which was STANDBY switches to ACTIVE. It happens every morning, even though there are no stored procedures or scheduled tasks to do this on any of the servers.


Some investigation has shown that at this time, McAfee EPo AV (V8.8) downloads and installs its daily DAT file. When this happens, the rnaaeserver.exe on the ACTIVE server starts communicating to all FT clients, spikes the bandwidth, loses connection to the clients and switches to the STANDBY server.


Has anyone else seen this happening, and more importantly, how did you fix it?
Whats the connection between the rnaaeserver.exe and the AV DAT update that send it into a bandwidth spike?

 

[martinshaw]

Problem fixed

Hi all


I found the issue. It was CPU and RAM resources on the two servers. Although they met the recommended Rockwell specification, that recommendation does not take into account what the IT department also run on the servers, ie Anti Virus protection and backups.


The specs are now 8 cores with 16Gb RAM with Win Server 2016.


Martin.

 

[drbitboy]

Thanks for the follow-up.


Why is IT so often clueless as the to needs of industrial computing?


E.g. why not only update the system on standby, and inhibit the active system, and find a way to trigger a failover every few days?


We could write a PLC program to this

 

[martinshaw]

Its funny how all this started way back. In the old days, PLCs and HMI's were generally on private or closed networks. Then when companies realised that we produced some good data that they could use, they were transported onto corporate networks for ease of access. With that though, brought the IT involvement with anti virus, windows updates and scheduled backups etc, and to them, a PC is something you use for email and spreadsheets and not controlling a 160 tonne continuous steel caster! Needless to say, a few seconds delay on a spreadsheet or sending an email is inconvenient, but loss of control for a few seconds when poring molten metal at plus 1600 degrees can be catastrophic.