Network Architecture Design Issues

Dear All,

I am doing our plants network upgrading works and have difficulty doing the design.

First please see the attached PDF files.
Basically, we have 2 plants in our Factory.
I named it Plant 1 & Plant 2.

Both plants were initially designed to be network of its own.
Due to the recent upgrading, more of the SCADA HMIs were introduced to both plants.

For Plant 1, we found that the HMI screens start to show "lagging" sympthom. I believe more control screens added with more I/Os and PLCs on the network would cause this to happen.
FYI, we are running on 100MBits networks unmanaged switches.

Please see attached PDF file - com. With more automation system upgraded, we tried to installed a remote station at Engineering offices to monitor the two plants PLC & HMIs. At the same time, the managing director requested to be able to view the HMIs in the office. We plan to link up both plants using fiber optics cable.

My first question:
By installing Gigabit unmanaged switches & Cat6 cables for Plant 1, will it improve on the data transfer? will this overcome "lagging" problem?

Second question:
Combining the both plants into single network may congest the two plants network. We are worry of this combination. We have to view both PLC and HMI from the single network due to budget constraint. Also we are not able to use internet switch for the time being. I was thinking of installing routers for combining the two plants. Will this avoid data congested issue?

I hope for some advices here.
Really hope to learn on network design.

Thanks & Regards,
Jason

First, unmanaged switches are are not the answer. You need to swap those for quality managed switches. AB Stratix, NTron, Moxa...

Second,in my experience "Auto-negotiate" settings for the network speed always add in a "lag", this is especially true if one device is set for "auto-negotiate" and another is a fixed speed, or doesn't have the "auto-negotiate" feature. You may have devices that the only speed they are capable of is 10M, half duplex...

Managed switches will allow you to configure each ports network speed. I suggest matching port speed with the speed of the device it is connected too.

Managed switches will also allow you to setup igmp snooping. Google that if you aren't familar.

Managed switches will allow you the ability to setup VLANS (isolating networks), this might be of value when you merge your networks.

How are your HMIs talking to the PLC? Are they all communicating through the IO sever? And the servers are the only ones talking to the PLCs? Too many computers trying to talk to a PLC will max out the PLCs ability to communicate effectively as communication bandwidth can be a function of the PLC scan time settings.

What software is running the on the HMIs? Have you reviewed the HMI program design to determine it is within the recommended design specs for communications? (IE aware of tag limits, folder limits...)

What are you using for communications? OPC? Kepware? RSLinx? DAServers?....

Are you using a ControlLogix as your picture indicates? Are you making online tag changes to the ControlLogix? If so, this can cause problems, best practice is to re-start your HMIs/data servers if you make tag changes so the communications can be re-optimized.

As you can see, there are a lot of items that can contribute to a slow network. You'll need to provide more details, and start with the basics. I bet that the problem is with network optimization rather than available bandwidth.

More than likely your primary problem is the unmanaged switch usage and if any of the projects are more than 2 years old they would almost in certain be setup for multicast ethernet communications.

Changing to unicast would halp but would be a good deal of work. A managed switch with IGMP switch will in my opionion make things a lot better.

As paully's 5.0 said check port speeds it needs to be all auto negotiate or setup per port and device.

If you are using things on dhcp that adds lag also mostly on startup though. Static ip is best or at least ip reservation

All those unmanaged switch on multicast are producting loads of garbage traffic that is taking all your bandwidth IMHO.

You have designed a star structured network.
That means there is no redundancy if one of the central switches fails.
Also, if the plant spans a large area, it will be costly to install cables from every point and back to the central switch.
For these two reasons I would consider to have a ring network, with several smaller switches placed strategically over the plant.

Dear Paully 5.0, PLC kid & JesperMP,

Thank you so much for your swift advice to my questions.
I will make much more effort learning the networking basic.
Here are some details:

How are your HMIs talking to the PLC?
1. Most of the control screen linked up directly to Contrologix through DASABCIP.
2. Some of the screen linked up to two PLC5/15 Classic through RSlinx.

Are they all communicating through the IO sever?
Most of them communicating through the IO server. Except for the two PLC5/15 which is link up to COntrologix via DHRIO. (not shown in the pics)
No choice we have to communicate through RSlinx.

What software is running the on the HMIs?
Wonderware Intouch 10.1 Runtime 60k.
Our tag limit is 60K. We have recently upgraded from 3k. Our application now is 4k IO tags.
in coming months, it would be 5k IO tags with the upgrading system.

Are you making online tag changes to the ControlLogix?
We normally reboot the IO servers after the modification made online to contrologix. Still problem persist.

You have designed a star structured network.
The two plants data does not cross over each other.
The remote monitoring from two offices aren't important.
Therefore I would design the ring network both each of the plant to provide redundancy.
So far, we use redundant power supply to the unmanaged switch.

I need to learn basic networking.
Now google learning how to set up Vlan.
That will be the first thing I will do while trying to add in the manageswitch.
Will it be good if I use three unmanage switches for the link of the two plants?

Thanks a lot for help.

Regards,
Jason

Be sure that you are setting up your network architecture adhereing to the S95 model.

IMHO i would no use unmanaged switches anywhere if it were me.

Any dumb (unmanaged)switch on a multicast ethernet network is going to cause problems. Even if you were setup primarily unicast i would still use managed switches for their diagnostic capabilities.

Also i would get managed switches in and setup and see how the traffic is before you start with vlans.

Here are a few good document you can look at:

http://www.cisco.com/en/US/docs/solutions/Verticals/CPwE/CPwE_DIG.html

http://www.odva.org/Portals/0/Library/Publications_Numbered/PUB00035R0_Infrastructure_Guide.pdf

IP Address Ranges
If you ever connect this network to the Internet, those IP addresses are problematic. 188.188.188.x is a valid address range on the Internet belonging to a Belgium wireless provider. So if the 'remote' connections are going to be via an Internet-connected VPN, there is a potential problem with IP address ambiguity.

There are only three ranges of IP addresses that should EVER be used for a LAN:
10.0.0.0 - 10.255.255.255
172.16.0.0. - 172.16.255.255
192.168.0.0 - 192.168.255.255

PLC Intercommunication
Your diagrams are detailed however they do not clarify what devices need to intercommunicate. If the PLC's are each in their own little world and do not use Ethernet/IP to intercommunicate between each other, an unmanaged switch is capable of performing fine here with proper configuration of the PLC's. However, if the PLC's need to seemlessly share data between each other (using Ethernet/IP multicasting instead of explicit messaging), then a managed switch with VLANs will be absolutely necessary - as suggested by others.

As soon as you use managed switch VLANs (to properly segment the networks into smaller subnetworks), you need a router to bring the layer 3 networks back together at some point in your architecture.

As mentioned by the others, Ethernet/IP multicasting is the killer item for an unmanaged switch that forces you into using a managed switch (with IGMP snooping or VLANs to limit the multicasting to just the appropriate clients). However, if all your network does is communicate between HMI's and PLC's using peer-to-peer messages, an unmanaged switch is up to the task if you are careful to turn off all multicasting (which would be doing nothing of value).

Routers in Place of Managed Switches
To keep costs down, I will frequently use 'local' unmanaged switches that feed to higher level routers or managed switches. Allen Bradley Ethernet/IP is quite good about telling you how much bandwidth you have remaining in a multicasting environment so I use their tools to know when I have taxed the network excessively: the beauty of Ethernet/IP is its determinism.

So to answer your question about using a router to add the new management visibility nodes: yes, that will work with proper design. The two plants should exist in their own domains (IP address ranges) and the router would forward just those ports necessary for the management SCADA stations to operate. This would ensure the two plants do not affect each other's performance. But since you indicated you already have a perceptible problem with 'lag', you should probably consider the managed switch approach anyhow.

The Bottom Line
So this was all a long-winded way of saying that managed switches will get you where you need to be. However, they are not the only possible solution. To keep costs down, you could move the PLC's to separate unmanaged networks (with unique IP address domains) with their own unmanaged switches and use a router to connect everything back together: that approach just requires more intellectual challenge to accomplish.

Its my opinion that the IP address ranges used in your network should reflect the intercommunication needs. If PLC's do not need to intercommunicate, they should be in their own IP address domain and a router should be used to explicitly move traffic between the networks. This is essentially the model that makes the entire Internet work properly and just because we are building small networks (that always have a way of growing), we should not ignore the domain model that was designed to make everything work together.