PDA

View Full Version : CtrlLgx Network Congestion ???


darkesha
July 23rd, 2009, 04:24 PM
Hi everyone.

We changed our line from PLC5 to CtrlLgx, and in the same time changed our Hydro Tester machine. We added a panelview that eliminated lots of digital I/Os and it all communicate through ethernet. We have 2 internet card in PLC chassis, one for all IO including the hmi, and other for bussiness network (data collection).

Program on our PLC consist of one 10ms scheduled routine that includes 25 subroutines and controls our hydro tester, and 4 continues tasks that control other machines on the line.

I want to add that on the panelview we monitor real-time test pressure operation set up as graph and as a digital display of a current value of test pressure. Beside that we have 30-40 different IO points on the HMI itself (start cycle, pause, etc..)
Everything run fine in first couple of weeks, but now it looks like the network is deteriorating and losing the speed and the bandwidth without adding any new IOs or major logic changes.
First we had delay on the graph itself where the test pressure just stops going up and becomes straight line (because new data is not coming in and it keeps showing the number where it got stuck), which confused hell out of our Op, but the process itself worked good with proper cycle and testing.
Recently we started getting error messages on the HMI saying:
CIP conection (1) open rejection (error 2040101) on route ControlLogix in slot 0 of the chassis at XXX.XXX...rest doesnt show but I believe its our internet address of IO internet card.
Also when the OP change mode from Manual to Auto or anything like that it takes forever for the screen to show and change color from green to red etc..

Are we hitting the wall with our Ethernet bandwidth and panelview requirements or something similar ?

I am debating to suggest getting one more Internet card (it would be 3 in total, and putting the third one and a HMI on separate subnet, but am not sure if it would help since it is still the same PLC that has to do the work and poll the data and do the calculation ??? Also that would require pulling separate ethernet cables so those two network dont share the same switches either.

I am not sure if I miss anything, but if I did, I would be glad to add more explanation.

Thanks for reading,

Darko

Robot_Man
July 23rd, 2009, 06:12 PM
Hi! My first suggestion would be to look at using an 'intelligent switch' rather than the one you've got.

In itself, this might solve your problem.

darkesha
July 23rd, 2009, 07:23 PM
Hi! My first suggestion would be to look at using an 'intelligent switch' rather than the one you've got.

In itself, this might solve your problem.

You mean some IGMP snooping switches ?
I have no experience with those at all. We have 3 unmanaged switches (Hirchman's rail types).

KP_EENG
July 23rd, 2009, 07:41 PM
I agree with the previous post. Try using a managed switch. Hirschman has them, so does Moxa, N-tron, wiedmuller, etc. Your local distributor or tech support should be able to assist you in the configuration. I would confirm that help before purchasing the equipment.

Regards

Kev77
July 23rd, 2009, 08:35 PM
Try hooking up to the network with your pc and open internet explorer. Type in the address of the ENBT cards and use that to help troubleshoot the problem. It will show you if you are at the limit of the cards. It doesn't sound like you should need another card, you just need to manage the network.

I use managed hirschmann switches on my network and they have worked great. When I first came here they had a linksys dumb switch and the system seemed to work fine. What are you using for the business network? Do you have a firewall installed to seperate the two networks?

Kevin

darkesha
July 23rd, 2009, 08:44 PM
Try hooking up to the network with your pc and open internet explorer. Type in the address of the ENBT cards and use that to help troubleshoot the problem. It will show you if you are at the limit of the cards. It doesn't sound like you should need another card, you just need to manage the network.

I use managed hirschmann switches on my network and they have worked great. When I first came here they had a linksys dumb switch and the system seemed to work fine. What are you using for the business network? Do you have a firewall installed to seperate the two networks?

Kevin

We have two different ENBTs with different subnets.

Kev77
July 23rd, 2009, 09:00 PM
We have two different ENBTs with different subnets.

Try looking at each ENBT address in your browser and use the diagnostics to help.

It also could have something to do with how you have your screens set up for the panelview. Do you have the displays set to replace or on top? The problem could exist in the panelview port and adding another ENBT will not correct that. Do you have any datalogs set up?

plc noob
July 23rd, 2009, 10:42 PM
Look at your RPI's see if any are lower than 10 msec.

Check produced/consumed tags each tag uses 1 TCP connection and 1 CIP connection

You need a managed switch.Hands down

Like kev77 said check your hmi application for replace or on top

You may see problems with the hmi depending on which screens are active.If set to replace only the tags associated with objests on the currently displayed screen are being used by the hmi so depending on how much is on a screen or how busy it it it can change the traffic in a major way.

Check the refresh and update times of objects on the hmi app such as graphs and numeric displays.

Make sure no scaling is taking place in the hmi

For any type of remote i/o like flex or point try to setup as rack optimized if no analog or intelligent cards used in that rack.

Use a fluke or other brand of cable verify tools to check cable quality.Bad connectors or ethernet cables running close to high voltage wiring causes major intermittant problems.

Check ENBT firmware for issues as well as any comm cards or remote i/o adapters.

Make sure you mare not maxing out connections on any of the ethernet cards. Try to stay below 80 percent load if above 80 you need an adapter that supports more connections.

Make sure you cables/connectors are right stranded cable/connector for stranded some connectors do both but most are one or the other.

Make sure there is not a lot of excess cable. a few feet okay but i went on one job where they used a 250 spool for each run and piled the excess in the bottom of the cabinet.This causes problems also.

Make sure nothing is connected to a POE Device.

My big money is on the switches though.I use ab stratix as you can specify what equipment is on each port and helps build a more accurate routing table.

Robot_Man
July 24th, 2009, 01:55 AM
Agreeing with all of the above: I'm currently working with a networked system of 8 PLC's and one SCADA PC. They are networked with a Hirschman Managed Switch, and each taget local control panel has a Hirschman unmanaged switch with the PLC and 4 VSD's (inverters) plugged in.

There was a problem with the managed switch, so I temporarily put in an unmanaged switch and it could only cope with the main PLC, my laptop and 2 local control panels. Adding another I kept getting comms timeouts.

When the managed switch was put back in (with a bit of tweaking to filter out broadcasts) the network load was something like 3% of capacity!!

Managed switches cost a bomb compared with bog-standard unmanaged, but I tell you my friend, that is the way to go.

Definitely check the quality of the network cabling. The same job used 'made to measure' cables rather than off the shelf and 50% of them were intermittently causing drop-outs due to poor crimping.

danny.michael
July 24th, 2009, 07:09 AM
Agreeing with all of the above, I would also suggest that you change the continuous task to a periodic task running at 100-200 ms if you can tolerate that rate. Do you really need the main task running at 10ms? Are there any overlaps?

I also suggest setting the number of unconnected buffers in the ControlLogix to the maximum of 40. Search the knowledgebase for "unconnected buffers". There is a KB item that describes how to do that.

darkesha
July 27th, 2009, 10:42 AM
Guys, thank you so much for so many helpful tips. As soon as production allows I will start working on them, one by one.
As for managed switches, what would you recommend beside the mentioned ab stratix switch?
As for the position of the this particular switch, does it go right after the ENBT card in particular, or it can go anywhere on the network.
We have 2, one is right in the PLC cabinet, and the other one is in the control room. The one in the control room is used to connect to HMI and lots of flexes which implies it's the "busier" one.

Robot_Man
July 27th, 2009, 11:18 AM
I only have experience with the Hirschmann managed switch. The basic setup software is sufficient, however if you want to do any clever monitoring the 'management' software is quite expensive.

In terms of connection, nominate a sensible place for the Managed Switch to live: There is no hierachy in the network as everything is effectively 'peer to peer'. Use unmanaged switches within a local control cabinet. From the unmanaged switch (I also used Hirschmann) connect to any port on the Managed Switch.

It was sensible in my case to group connections in the Local Control Panel where the local PLC was communicating with four VSD's, then one port fed back to the the Managed Switch.

As we were using a CompactLogix system with 1969-L32E CPU, we did not need a separate Ethernet card in the PLC rack and simply used the CPU Ethernet port. In the Local Control Panels this connected to the unmanaged switches and in our 'main' panel we plugged the one identical PLC straight into the Managed Switch.

Remember to check everything has unique Static IP addresses: If your PC somehow clashes with IP address of course you will have problems also.

Good luck!

darkesha
July 27th, 2009, 12:14 PM
Agreeing with all of the above, I would also suggest that you change the continuous task to a periodic task running at 100-200 ms if you can tolerate that rate. Do you really need the main task running at 10ms? Are there any overlaps?

I also suggest setting the number of unconnected buffers in the ControlLogix to the maximum of 40. Search the knowledgebase for "unconnected buffers". There is a KB item that describes how to do that.

Yes, the 10ms task is adequate if not too slow in order to control water pressure going from 0-10k PSI.
As for the other tasks and setting them up to 100 - 200ms, I am afraid we would see the delays which would confuse the OPs as well as machine in the Auto Cycle. They all work good as is..

mellis
July 27th, 2009, 01:33 PM
Some additional data would be helpful:

Main Task - Properties - Monitor:
What are the scan time and interval times?

Periodic Task - Properties - Monitor:
What are the scan time and interval times?

Specifically, look at the Main Task interval time. This is the time for one complete pass thru the continuous task. If your Periodic Task is taking a significant portion of 10ms, it can be surprisingly long.

Comms compete with the continuous task, same thing applies. If the periodic task isn't leaving much time, they have problems.

danny.michael
July 27th, 2009, 07:09 PM
Mellis is correct. Comms compete with the continuous task. The "Communications Time Slice" setting in the controller properties dialog box controls how much time the controller spends on communications relative to the time it takes to complete one cycle of the continuous task. If the Time Slice is set to the default of 20%, then the controller will allot 20% of the time it takes to complete a scan of the continuous task to communications. You might try setting the Time Slice to 50% or more, but your best bet is to change the continuous task to periodic. Without a continuous task, the controller allots all the time left between scans of the periodic tasks to communications.

In my opinion, you should move everything out of the 10ms task that does not need to operate at that scan rate. Are all 25 subroutines devoted to controlling the water pressure? If not, try moving them to a slower task.

You do have to be careful, however, to be sure digital signals generated in one task are available if needed in the other tasks. For instance, a one shot generated in a 100ms task will be on for 10 scans of a 10ms task. On the other hand, a one shot generated in the 10ms task may not be see at all in a 100ms task.

In my experience, operating without a continuous task is best for communication speed. For most everything I do a 50 to 100 ms task works fine.

Ken Roach
July 27th, 2009, 07:58 PM
Because the system worked well for several weeks and you did not make any major program changes, I do not think that your problem is a ControlLogix performance or configuration problem.

This sounds very much like an Ethernet media problem.

The embedded web pages in the 1756-ENBT modules will give you a lot of very good information about errors and lost packets, as well as giving you CPU utilization information about the 1756-ENBT itself.

Problems like this are exactly why I always recommend managed switches, because they give you diagnostic features per port, instead of just relying on a link LED and assuming everything's working fine.

The best thing you could do for the Forum is to use the WGET (http://en.wikipedia.org/wiki/Wget) utility to save a copy of the 1756-ENBT web pages files and post an archive of them to the Forum.

darkesha
July 27th, 2009, 08:11 PM
Because the system worked well for several weeks and you did not make any major program changes, I do not think that your problem is a ControlLogix performance or configuration problem.

This sounds very much like an Ethernet media problem.

The embedded web pages in the 1756-ENBT modules will give you a lot of very good information about errors and lost packets, as well as giving you CPU utilization information about the 1756-ENBT itself.

Problems like this are exactly why I always recommend managed switches, because they give you diagnostic features per port, instead of just relying on a link LED and assuming everything's working fine.

The best thing you could do for the Forum is to use the WGET (http://en.wikipedia.org/wiki/Wget) utility to save a copy of the 1756-ENBT web pages files and post an archive of them to the Forum.


We will install and configure one of managged switches as soon as we get one, and production allows.
Instead of WGET, will one or more screen capture shots be enough ?
I tend not to install much of unapproved software with an exception of browser even though I am all pro-GnuLinux guy.

Ken Roach
July 27th, 2009, 08:34 PM
I very strongly prefer actual HTML pages to screencaptures, don't you ?

For a typical 1756-ENBT diagnostic page:
HTML = 7 kB
PNG = 81 kB
BMP = 4.1 MB

If you don't want to use WGET or another "website downloader", consider saving the HTTP pages individually (I use right-click and "Save Frame As" in Firefox) instead of taking a screenshot of them.

If you are responsible for an automation network that uses Ethernet, you should have WGET and Wireshark installed at a minimum.

darkesha
July 28th, 2009, 06:40 PM
I very strongly prefer actual HTML pages to screencaptures, don't you ?

If you are responsible for an automation network that uses Ethernet, you should have WGET and Wireshark installed at a minimum.

Acknowledged !
Thanks for the tip for a good tool.


Back to my problem; We got N-Tron508TX managed switch today and I installed it through the lunch break, and it did wonders.
Now it runs same way as it did before.
Hopefully it will not deteriorate over time like unmanaged did.

I suspect that we had IGMP multicast congestion problems that unmannaged switch just could not cope with. Now the panelview looks instantaneously responsive (set for 50ms update) and the graph updates in real time as well.

Thanks to Everyone !

Thanks to everyone

darkesha
July 28th, 2009, 06:44 PM
Problems like this are exactly why I always recommend managed switches, because they give you diagnostic features per port, instead of just relying on a link LED and assuming everything's working fine.

Does the tool for diagnostic has to come from the manufacturer or there are some general ones as well?
I kind of think that there is not, but it doesnt hurt to ask.

Model we got today is basic one and the program in cd they provided is not recognizing switch. Therefore we have no way of diagnostics.

The Plc Kid
July 28th, 2009, 09:01 PM
Ken

What is another good windows tool for downloading the webpages in your opinion?

I need to setup something to do this.

Ken Roach
July 29th, 2009, 02:00 AM
508TX is a rugged, non-managed switch.
508TX-N has the N-View OPC Server diagnostic feature.
508TX-A has the QoS, Trunking, VLAN, IGMP Snooping, Port Mirroring, etc features.

Most straightforward managed switches provides some kind of web-page based diagnostic reporting.

Switches that are made to be enterprise-class, like HP or Cisco, are going to have utilities and remote management software that is vendor-specific and sophisticated. Most automation systems don't get to that level of power and complexity in switches.

I'm a big fan of WGET because of its simplicity and ability to be called from batch files. There are even GUI front-ends you can add to it in Windows if you're not comfortable with the command-line syntax.

I've used HTTrack and LeechGet as well, but scrubbed them in favor of the universal, cross-platform, classic WGET.

darkesha
July 31st, 2009, 04:15 PM
We will stick with this one for now, since it works great, and if any problems arise we will get an -A version and see what is causing problems.