Network separation

[smiller]

I am currently part of a group that is fighting the corporate IT department for control of the processing network. We have a sprawling network covering about 50 square miles of pretty rugged terrain (a very large copper mine in the southwest USA). The network consists of miles of fiber and some radio telemetry for the more remote sites. We mainly use Modbus TCP. My question is: what is the standard for Admin and control over processing networks. The IT dept wants to use a VLAN and share the hardware for the business and processing networks with them having all Admin rights of course. We (the process group) want physical separation of the networks with IT keeping total control of the business network and us having total control of the processing network. Do any of you Gurus have any thoughts to share with me? I will be attending daily meetings this week to plead the process side of this case so I'm looking for some industry ammo to take.
thanks.
Steve

 

[DesertDog]

I would think it would be possible to share the main body of hardware while still keeping the networks seperate for administrative purposes. It may also be a benefit to have hardware capable of this for each operation so if one goes down they could be combined.

 

[PLucas]

Steve

Your story is very similar to what has happened where I work.

The engineering department installed a large fibre optic and radio lan network around our site to enable us to do remote diagnostics and crane messaging. When this network was installed and working to our satisfaction, we then find out that the IT department is laying claim to this network and want control over it. We attempted to resist this but was over ruled by the powers that be. It's corporate policy we were told.

This network was kept seperate from the corporate LAN when we installed it, but the IT department decided that they would connect onto the existing corporate LAN and they then changed the configuration of our 'access points' and 'station adaptors' on the radio lan. Our main bug bare, was the fact that they reduced the bandwidth on the radio lan from 3 meg to 1 meg, stating that we did not need that amount of bandwidth as the LAN traffic was not that high. They then proceeded to put everything onto VLAN's, which to us was a pain in the *ss as this meant that our connections slowed even more because it double the amount of switches and hubs our 'comms' had to go through. The 'slower' connections were really noticable, especially on our mobile plant which has to hop from one AP to another as it travels round the site.

Last Friday, I managed to convince our 'Network Analyst' that the amount of hops through switches and hub was causing us a problem. For one link I counted 11 hops, which was far too many and the result of which was lost comms with our remote PLC's and loss of pings. The network analyst was not totally convinced at this stage, so I persuaded him to re-route some of the fibres (actually back to how we had originally installed them or near enough!). We re-ran the same test and the result was a vast improvement in the comms with practically no lost pings. But he still wasn't convinced as he was the network analyst and he knows more about networking than someone in the engineering department!! So he put the fibres back to how he had them after they took control of the network.

My advice to you would be to resist this as much as possible, I am not convinced that handing over an engineering network to a corporate network administrator is a good idea! Our main PLC is constantly 'ARPing' the remote PLC's and the ET200S's (on profinet) to ensure that the connection stays alive so any loss of comms, even for a split second causes us problems.

I do not claim to be a network expert, but I know what I see and 'our' network worked fine with no real problems until our IT department got hold of it and messed about with the configuration.

I hope your does not go the same way.

Paul

 

[panic mode]

Steve,

I agree with previous posts but if you ask me, I would fight
for phisically separate networks because sooner or later you
will run into trouble.

There are so many possibilities for something going wrong and
when it does let someone else take the blame.

We had similar case where someone would come to troubleshoot
one problem, unplug wrong cable, use ping to see that his
nodes are reachable and walk away. By the time problem is
traced it's too late. Another possible problem is traffic
load and communication timeouts you will have to deal with.
You might have to add more logic to handle bussy situations
which depending on number of systems in use might be more costly
than running new lines.

You just can't predict or keep in line what the 'other' guys do.

Another problem is that either side is not quite familiar
with hardware/software used by others and troubleshooting
often could require involvement of members from both sides
which is costly as well.

Or just turn it arround. Ask them to have their own network
but so you can use it as backup if existing one is down.
See how they like that.

 

[rdrast]

100% Pure physical seperation between data acq / control networks and enterprise level 'user' networks.

If that is impossible, one solution is to switch to a completely non-TCP/IP based network for controls, be it Profibus, Devicenet, Controlnet, bla bla bla. One big problem is that control networks generally spew out a tremendous amount of very small packets, which are usually fine in a properly segmented network, but throw medium volume large packets on top of that from the enterprise network, you are asking for trouble.

I always try to set up a completely seperate domain for my control networks, and on one of the machines, put in a seperate NIC or two so it and IT ALONE can act as a gateway up to higher level enterprise systems.

If IT want's full control over the entire network, try to get them to sign off on accepting 100% responsibilities for all communications systems and comms failure-related downtimes.

 

[jamesau]

This is a classic problem.

There are a few things you can do to ward off IT control.

1) Sit down with the operations manager and explain your concerns; work with him to quantify in $$$ how much downtime will cost the business. You help him/her understand the nature of potential downtime scenarios, he/she puts the $ figure on it. Basically, identify and cultivate influential allies.

2) Consider renaming/euphemising the network/computing components to thwart the IT guys' natural propensity to wrest control of the system.
For example, an MMI PC becomes a 'machine controller'. An ethernet LAN becomes a 'mining control and production network'.

3) If your situation starts to look bad, ensure at least that written protocols are established for doing all maintenance and upgrades. Make the IT guys author these subject to your review/approval. In addition, consider a redesign of the network so that your area of concern is made more robust - have it come out of IT's budget. (they may have a real business case for using it... be prepared to understand it and then help them or block them).

Good luck

 

[Borte]

Hello Guys!

I would definetly go for separate networks.

At the plant where I work we have separate networks, one for business/office use and one for data acq (profibus and ethernet), the only connection between these two networks goes via a single gateway computer (no physical connection only data comming from services on that gateway computer).

If you start mixing these two networks all kinds of problems may araise (as earlier pointed out), since things like big network load from office applications can cause the fragile plantfloor networks to go down. Theese networks tends to be really "busy" so any extra traffic should be avoided, even though the traffic only consists of small network packets.
Like for the plant I described earlier, we have about 2000 control points from 50 different sources and this is in fact causing a considerable amount of traffic.

In addition you don't have to worry about "outside" (outside could be internet and office and IT) interference on the data acq network.

Regards
Borte

 

[Lancie1]

smiller,
You should fight them until your last dying breath. I fear that it is a losing battle. The IT departments are rapidly taking over the world, and now to make practically any change on an official networked computer, you have to have permission from IT, which you get by going through channels, and a month later you might get that PLC programming software loaded (that you needed for last month's project).

The IT procedures of course are all blamed on the threat from hackers who might break in and do some terrible damage, and who in the company can argue with that logic, so it is inevitiable that the IT departments will be right up there next to the Chairman of the Board in the butt-lickin contest.

As a result, we no longer have "personal" computers on which a person can load, manage, run, and delete his personal programs at will. Where we are is right back where we started out with the old IBM 360 mainframes, where you submitted a request on punch cards, and got a response at the whim of the IT operator, usually several days later.

As for me, to get any actual work done, I have started carrying my laptop (bought from personal funds) in to work every day, and they are going to have to walk over my dead body to take that one away also.

 

[corkers]

There are many arguements for and against joining the production and IT networks together into one entity under the control of the IT deparment. Unfortunately the against outweigh the fors ten to one.

1. You need admin access to change some settings such as com ports and applying patches to software. Waiting for IT to arrive at a breakdown so they can make the changes can cost big $$ esp if downtime costs US$15k per min...

2. Depending on what sort of data you are moving around ie how critical, network overload can occur very quickly due to the number of small packets that PLCs tend to send.

3. IT doesn't consider the implications of making changes to the network, from a production viewpoint.

4. IT departments often stick there nose in and tell you to use a particular product or else they won't support it, then don't support it anyway.

5. Standard PCs in IT often don't have the required version of windows to run the latest (or oldest, read 6200 series) programing software.

Some suggestions for arguements to use.

Tell IT they have to buy replace all the network gear as they don't comply with IT standard... watch them run for cover.

Tell IT they have to replace all the HMI PCs and in turn, all the old ISA network cards (like DataHighway, Devicenet, ControlNet) as the new IT compliant PCs dont have ISA or EISA slots only PCI.

Tell IT they have to upgrade all the PLC and HMI software to the latest versions to run on the new IT compliant PCs.

Tell IT you need and admin level password. Then tell them how much per 10secs downtime you will bill them while waiting for an IT guy to attend a breakdown when they refuse.

Good luck, fight to your dying breath not to do it and if you lose make sure you get new toys out of it...

Cheers

Andrew

 

[JesperMP]

There is a small technical detail that can help ward off the IT demons (at least in the EU):
In EU all electrical equipment must meet CE regulations. That means that everything that you can lay your hand on has a litte "CE" mark stamped on it.
What many people doesnt know is that the CE regulations is divided into two groups, "home/office" and "industrial". The industrial regulations allows more EMC to be radiated from a device, but in turn demands that a device is more robust against EMC (it makes sense).
It means that network component in an industrial application must meet the EN 50081-2 EMC - Generic emission standard, Part 2 - Industrial environment and EN 50082-2 EMC - Generic immunity standard, Part 2 - Industrial environment
Tell that to the IT department. Tell them its the law !
Secondly, tell them that as network problems can propagate from the office to the production part of the network, that the entire network must meet the same requirements.
Thirdly, present them as example some industrial grade ethernet switches (with price tag).
Fourthly, have them take all responsibility over the production network under the condition that production network issues takes presedence over all other issues.

 

[BobB]

Definately fight them off. The last time I used Ethernet (and if I have my way it will be the last time) for peer to peer, I configured the whole network in the "10" group. This included the PLCs with password protection. IT winged like you would not believe but they could not change anything because of the passwords in the PLCs. HA!!! HA!!!!

I continue to use proprietary networks for peer to peer communications but allow an Ethernet connection to one PLC on the network for data extraction. Password protection stops them doing anything else but extract data. My peer to peer network operates beautifully and if the IT part crashes or slows down, it is their problem not mine.

One of the problems we all encounter is that if it is a computer or Ethernet network IT want to own it. But they do not want to take any responsibility for anything that goes wrong. It will always be the industrial type equipment that causes the problem, or so they tell the boss.

 

[Jim Dungar]

My two cents worth

Make sure that IT can guarantee uptime of the network. In one case we needed to gather data between two buildings separated by 5 miles, the only connection was an IT network. IT said we could use their network, no problem but "you do know that we take down our network every weekend for maintenance/upgrade, don't you?

 

[Doug_Adam]

Keep the network separate.
Two separate networks, one to connect to scada and one to the business unit work very well.
If you lay fibre with enough cores, you can always allow IT to use some of your spares.

Another argument that you can use is that the network is not connected to the 'outside' so is not going to be hacked from the 'net'.

One thing I have to say about my company's IT department was that they quickly realised that they didn't want to touch the production network. This was a good result IMHO.

Doug