Process control LAN v.s. business LAN

Hey fellas,

I'm looking for some feedback on how you have set up your process control LAN (ethernet). About a year ago we upgraded our Wonderware Intouch HMIs from old NT4 boxes with KT cards for DH+ to WinXP and instead of buying 15 new PCI slot KTX cards, our Rockwell rep (along with other Rockwell tech types) convinced us that the way to go was to install a couple of CLX chassis containing ENBT and DHRIO cards to act as Ethernet to dh+ bridges.

After attending a special "networking for process control" seminar put on by our corporate IT department (large multi national), and with their assistance, we rearranged our ethernet network so that the bridges and HMIs and all process related equipment was on a physically separate network. All we had left to do was the configuration of a firewall between the business LAN and our process LAN when things suddenly changed.

New owners took over our facility. The new owners were a relatively small company with an IT department made up of only a few guys. Their IT department decided that is was best to switch us to how they do things at their original facility...VLAN. Since the change over to VLAN we have had a number of issues with connections dropping, slow access times and such. However, I have no proof that any of our issues are related to us being on VLAN.

I admit that my ethernet LAN experience is limited, so I'm just looking for advice from others out there.

1. How are you set up? Why?
2. What ethernet hardware do you use and/or recommend? Why?
3. Can you provide pros/cons for segregated and VLAN setups please?

Many thanks in advance!

I suppose that with VLAN you mean "Virtual LAN" and not "Wireless LAN".

I dont have much to share. Only that with the Ethernet/IP that is the protocol used with ControlLogix and the ENBT module, it is very important to use switches that have "IGMP snooping".
Does your system have that ?
I dont even know if IGMP snooping works on a VLAN.

Appears VLAN may be recommended.
http://domino.automation.rockwell.com/applications/kb/RAKB.nsf/0/84379FB8AFD936FC85256CEF005586D0?OpenDocument

rsdoran said:

Appears VLAN may be recommended.

Click to expand...

I believe their recommendation of VLan is as an improvement over having process control on the same network as the business traffic (worst case).

One thing I question (and this is from an ethernet newbie) is their statement that "IP multicast traffic from VLAN 1 will not reach VLAN 2". Logically, yes but are not all ports in a switched physically connected to one another i.e. a backbone of sorts? I guess they call it fabric. If traffic on VLAN1 increases, does that not affect the bandwidth of VLAN2? The traffic from one does not reach the other but the switch's ability to deal with one is affected by it's need to service the other. Does that make sense?

It looks like you were on the right track before the takeover. My opinion is your best bet is to have a physically seperate network from the business network. If data needs to transfer between the two install a firewall.

The requirements and equipment are completely different on an automation network as compared to a business network. A good IT guy recognizes those differences and will work with the controls guys to come up with workable solutions.

In my experience very few people in IT have practical experience in manufacturing or control system networks. Their lack of experice leads them to believe they can adopt the same methods and policies to a control system network that they already use in the office. I am sure the VLAN setup could be made to work ok, but it sounds like you already had a decent setup before they changed it.

Wildswing, you’re asking the right question. In a switch (or VLAN) there is a part of the switch which connects the Input Ports to the Output Ports (these are not the Ports that you plug your Ethernet cable into). This part of the switch is called the switching fabric. Packet queuing is performed on both the input and output ports. What could be happening is that the switching fabric in the VLAN is not fast enough to handle the business and process traffic at the same time. When the switching fabric isn’t fast enough to handle the incoming packets, they will be stored in the input port’s buffer, when this buffer gets full, incoming packets are dropped.

The people on the business LAN won’t notice these dropped packets because they are using TCP. You on the other hand will because you are using UDP. When a TCP packet is dropped, the receiver will request that the packet be retransmitted, hence TCP is a reliable protocol. When a UDP packet gets dropped it’s gone for good, an unreliable protocol. The Rockwell Automation document that Ron posted a link to could be considered a bit misleading. UDP is the right communication protocol for I/O communication because of it’s speed, but it is not a reliable protocol. Rockwell is doing things to make UDP as reliable as possible, but in the end it’s still an unreliable protocol.

You need to make sure your IT people understand that your traffic isn’t coming in bursts as most business LAN traffic does, you have a constant flow of traffic. This could be the cause of your dropped packets.

We also have become attune to network traffic resulting in comms problems. Our main issue is with HMI comms to the PLC, which is a serious problem, as the HMI is the only possible way to shutdown several parts of these lines, aside from a hardwired estop. Our plan sounds exactly congruent to what you were doing before the takeover, with some minor changes. We currently have a separate business lan and process lan, but there is really no security separation between the two. We also will have another separate network that we are calling the "machine lan". This machine lan is where all PLCs and HMI's will sit. All devices on this "machine lan" will be assigned non-routable IP addresses. While some of this is more IT knowledge, my understanding is that we can have problems with the business lan, or process lan, and still have unaffected comms on the machine lan. Please note that each machine lan is separate for each line. We have lines in the plant containing as many as 5-7 PLC's and 10-15 HMI's. This is the reason for each of these lines having their own dedicated "machine lan."

Tark is on the right track. Data from VLAN 1 won't show up on VLAN 2, but all the traffic is still being processed by the same CPU in the switch and the CPU can only do things so fast (and only one at a time) so some packets could be getting dropped and others could simply be tied up in the switch's buffer long enough for the data they contain to be useless to the receiving device.

The easy way to fix your problems (probably) is to buy a new network switch (preferably industrial grade) and connect everything on the "process" network to the new switch, then connect the new switch to the existing VLAN switch. This gives you a dedicated switch just for the process network, but retains the existing link (and address assignments) to the business network.

Thanks for the feedback fellas. Excellent discussion. Keep 'em comin'.

Too late to ask, but why didn't you make the network a CONTROL-NET network for your process....and your ENET for Wonderware...etc at the desk?
Our plant is comprised of 3 network platforms. Controlnet-Ethernet-Business-LAN. The control is completely seperate from the Business LAN.

This was not a new installation. DH+ was the existing network connecting our PLC5s and HMIs. We would have had to replace the dh+ as well so it probably would not have been considered an option (cost was a big factor), although I must admit that, back when the bridge option was presented to us, we were not aware that controlnet was was even an option.

Is there a ControlNet adapter for the PLC5s? How about ControlNet PCI cards? The dollar signs are starting to pile up huh?

Yes, and no. ControlNet has been around for quite sometime.
Very good, robust network, that will outlast many others.

I understand RS Ethernet IP is pretty robust utilizing all 7 layers of the protocol to ensure throughput and security. However I also understand that due to traffic its highly recomended to use Managed ethernet switches, wihich will cost 10-20 times as much as an un-managed switch.

Managed switches utilize admin config tools to set paths and prioritization of packets througput from IP to IP( PLC to HMI). The switch uses IGMP to detect the QOS (quality of signal) data within each packet to analyze transfer performance. The managed switch can "heal" a bad path and reroute packet via other ethernet paths to the same destination, if it is aware of a better QOS path. The managed switch can ensure certain packet make it thru ahead of other unimportant packets i.e plc traffic over security camera traffic!

[I'm implying that the each PLC or local group is connected to a simple switch, which can detect QOS data. This switch is linked to 2 or more switches via differnet ethernet cables following diffent physical routes to other switches. At least 1 managed switch in the network can decide how to best route the packets from between locations]

Note I havent done this yet, but its a hot subject right now and this is my recent education on the subject.

Look at Phoenix Contact or Hirschmann or others for industrial managed switches.

Keep in mind that if cost is a big factor, you do not need industrial managed switches like the ones from Hirschmann. As long as they are a good quality managed switch; you'll be fine. We have all Cisco 2950 Catalyst series managed switches and have zero problems. Of course, we don't have them mounted in or on equipment in the plant. They are in the "attic" with the rest of the networking equipment. If this is not your case then you will have to go the industrial switch route. Keep in mind though that the price for an industrial switch is about four to five times higher vs a Cisco switch or something similar. You can pull a lot of extra cable for that price!

Patrick

Tark said:

"When a UDP packet gets dropped it’s gone for good, an unreliable protocol."

Click to expand...

nR--> unless the receiver specifically requests a retransmit (usually in the form of a NACK). Error detection and correction are solely up to the receiver in the UDP protocol.

Tark said:

"The Rockwell Automation document that Ron posted a link to could be considered a bit misleading. UDP is the right communication protocol for I/O communication because of it’s speed, but it is not a reliable protocol.

Click to expand...

nR--> UDP is a good protocol to use on small networks where real-time data is required where congestion is not significant. TCP has a throttling mechanism that only allows it to transmit 64-512kbps of unacknowledged data before waiting to make sure the receiving end is getting the transmission. If there is no ACK received, TCP assumes the network is congested and stops transmitting. Also, if latency on the network increases, TCP automatically throttles back its transmission rate... these characteristics are not a good thing when the data is controlling processes that require accurate real-time data and fast response times across the network. The slow-start and congestion throttling mechanisms of TCP/IP are also the reason that data communications over satellite are slow despite large amounts of available bandwidth.

Tark said:

"Rockwell is doing things to make UDP as reliable as possible, but in the end it’s still an unreliable protocol.".

Click to expand...

nR--> Adding error detection/correction in the form of NACKs or selective NACKs (more bandwidth efficient) can pretty much negate the unreliability problems introduced when using a connection-less oriented (UDP) over a connection-oriented (TCP) protocol. UDP also insures that data is sent out of the sending buffers no matter what the state of the network is in (congested or whatever), which is desirable when dealing in real-time processes.

Tark said:

You need to make sure your IT people understand that your traffic isn’t coming in bursts as most business LAN traffic does, you have a constant flow of traffic. This could be the cause of your dropped packets.

Click to expand...

I am thinking Tark is right on in thinking congestion.