Cimplicity blackout screens

Hi everybody,
I´m facing a trouble with my Cimplicity HMI v8.2, screens gets blackout frequently (I mean once a day, at 9pm most of times, this is when operators begins the turn).
I´ve sent ping to my PLCs and server, at trouble moment, and they answer at maximum time of 27ms.
But alarm message is still "Comm error or dynamic disable, Server:E07"
That´s the name of one of my missing PLCs, the other is E03, but in total there are nine (the total in my network also) PLCs having this fail.

Previously:
-We updated cimplicity from v6 to v8.2 using Windows Server 2003, (everything fine).
-Then, we updated Windows (to Server 2008 R2) version and PC (everything was fine also, at least during a month).
-Finally, 10 days ago more or less, the ethernet switch was updated and the blackout screens problem began.

Steps:
-We´ve monitored the network speed, and everything looks fine, switch doesn't reports failure or reset at any device according to IT.
-We've checked cimplicity's configuration: cimhosts file, timeout and retries are by default, and stale data is enabled at each port, which are nine, one for each PLC.
-We've compared the old model with new and new switch seems to hold proper configuration (of course).

I don´t know where is the problem, I've thought it could be physical but It's not strange that it happens at same time all days?.

Thanks in advance, and I'm sorry for so long speech.

I can't answer directly, but if this were mine, I would have put the old switch back in about five days ago to see if the problem went away.

Are all these devices connected to the same hardwired LAN ?

27 milliseconds is far too long for a PING to take. Any single-switch LAN should have a PING time under 1 ms.

How fast is the PING during normal operation ?

This sounds very much like a network problem. You need to focus on actually measuring retries and delays during the time of the network failure.

Lots of things to ponder - just some ideas/questions to consider here:

When you say 'blackout screens', I assume you mean that the data values being displayed in Cimplicity go black? Or does the whole display go black?

Which PLC models are you talking to?
Which drivers are you using?
Are they the GE drivers or aftermarket drivers?
If aftermarket, what licensing scheme do they use?

If it is the data values going black, is it for all PLCs?
Are all PLCs connected via Ethernet - or do you have some connected via serial modbus that you can separately test?
Many Ethernet-based PLCs have a web page - can you access that web page when the screen goes black?

How is your PLC network separated from the control network?
With those long ping times, is there a bandwidth intensive job (like backups) running on your network at that time that is burying your bandwidth?

Does your switch, router and network card keep statistics? Are you seeing any errors or collisions? You should see zero on a well designed network.

Thanks for your answer.

RussB, we're looking doing the change for the old switch however it depends of my customer by the moment. Before doing it we would like to be sure there isn't a wrong configuration on cimplicity (it seems everything's correct) or another reason for screens to being blacking out.

And Ken, the switch is the same for my PLCs and PCs and yes is the same LAN.
I read a checklist "Server-Viewer Communications" it says that less than 100ms is a good speed for network. But you're right, PING during normal operation is 1ms or less in my case.

IT says that my control equipment is in the same VLAN than other measurement equipment which uses IPs too. But network segments are different.
As you both say, it seems a network issue (and according to this information) maybe a wrong VLAN configuration on the switch.

Thank you very much, I hope finding solution using your suggestions.

As usual, Ken is on the right path.

VLANs are great at separating multicast and broadcast traffic into their own domains but are often configured to combine all the traffic into a trunk to travel to another switch and/or router. So a VLAN trunk can eat away all your bandwidth when some bandwidth-hungry application runs (like backups).

Its also easy to accidentally configure a VLAN port as a 'trunk' port instead of an 'access' port and bury your communication on that port. The 'trunk' ports should be connected between switches and the 'access' ports to devices, plus you need to carefully consider which VLAN IDs flow across the trunk.

nwboson, thanks.

* When I say "blackout screens" I mean displayed values goes black.
* I'm using GE Series 90-30 PLCs.
* I'm using GE drivers.
*All my PLCs (nine) are over ethernet.
When failure happens my nine PLCs goes down. But previously I had only two port configured on my CIMPLICITY (one with one PLC and the other with nine PLCs, I don't know why the configuration was in this way), when it began fail I supposed it was for the network traffic and I separated the PLCs using nine ports.

Unfortunately I don't have any access to the switch (policies) but IT says switch doesn't presents any reset or error counting.

I haven't seen if 90-30 PLC has a web page as you say, I will check, I'm sure it will be helpful for monitoring failure when problem occurs.

Thanks,

nwboson said:

As usual, Ken is on the right path.

VLANs are great at separating multicast and broadcast traffic into their own domains but are often configured to combine all the traffic into a trunk to travel to another switch and/or router. So a VLAN trunk can eat away all your bandwidth when some bandwidth-hungry application runs (like backups).

Its also easy to accidentally configure a VLAN port as a 'trunk' port instead of an 'access' port and bury your communication on that port. The 'trunk' ports should be connected between switches and the 'access' ports to devices, plus you need to carefully consider which VLAN IDs flow across the trunk.

Click to expand...

This makes all the sense, it seems they configured the switch to let pass all the traffic, so at the moment of starting operation, network speed goes slow and cimplicity can't access correctly.
I've a conference call today, I will go for this path with them.
Thank you all.

VRincon said:

* I'm using GE drivers.

Click to expand...

Are you using the S90_Triplex driver or the older S90_TCP/IP driver? If using the older one, consider converting to the Triplex. v8.2 has a conversion utility to update the driver.

Thanks mbartoli, I'm using triplex. Drivers were converted as a part of HMI upgrading.

They sent me this information from the server Status Log:

6/20/2014 1:50:31 PM Success 10328 W32RTR RtrStart 0
CIMPLICITY Router V8.20.20483 (ENU) starting on SERVER
Error of type: COR_IPC_ERR, Code: 0

6/20/2014 1:45:23 PM Success 13024 W32RTR RouterExit 0
CIMPLICITY Router Stopping on SERVER
Error of type: COR_IPC_ERR, Code: 0

6/17/2014 10:33:22 AM Success 13024 W32RTR RtrStart 0
CIMPLICITY Router V8.20.20483 (ENU) starting on SERVER
Error of type: COR_IPC_ERR, Code: 0

6/17/2014 10:28:42 AM Success 4944 W32RTR RouterExit 0
CIMPLICITY Router Stopping on SERVER
Error of type: COR_IPC_ERR, Code: 0

* Why would CIMPLICITY Router stop by itself?
* And now they say the problem is coming more frequently, the alarm banner is always indicating "Comm error or dynamic disable, SERVER:E0X"... E01 to E09 are my PLCs.

Please, if you know about this error in status log... i can´t find enough information

Problem solved

If somebody helps...
We went to the plant with IT personal, after reviewing status log and cimplicity's configuration, we were sure everything was fine in cimplicity.
By the other hand they monitored the network and found that server had been losing packages and response time had increased more than 10 ms.
So this specialist configured something for server don't to lose packages.

Response is still high (I mean, 20-30 ms) sometimes, and it happens still at the same hour of night. But now, Cimplicity is no longer missing devices.

Thanks those who answered.