CtrlLgx Network Congestion ???

Hi everyone.

We changed our line from PLC5 to CtrlLgx, and in the same time changed our Hydro Tester machine. We added a panelview that eliminated lots of digital I/Os and it all communicate through ethernet. We have 2 internet card in PLC chassis, one for all IO including the hmi, and other for bussiness network (data collection).

Program on our PLC consist of one 10ms scheduled routine that includes 25 subroutines and controls our hydro tester, and 4 continues tasks that control other machines on the line.

I want to add that on the panelview we monitor real-time test pressure operation set up as graph and as a digital display of a current value of test pressure. Beside that we have 30-40 different IO points on the HMI itself (start cycle, pause, etc..)
Everything run fine in first couple of weeks, but now it looks like the network is deteriorating and losing the speed and the bandwidth without adding any new IOs or major logic changes.
First we had delay on the graph itself where the test pressure just stops going up and becomes straight line (because new data is not coming in and it keeps showing the number where it got stuck), which confused hell out of our Op, but the process itself worked good with proper cycle and testing.
Recently we started getting error messages on the HMI saying:
CIP conection (1) open rejection (error 2040101) on route ControlLogix in slot 0 of the chassis at XXX.XXX...rest doesnt show but I believe its our internet address of IO internet card.
Also when the OP change mode from Manual to Auto or anything like that it takes forever for the screen to show and change color from green to red etc..

Are we hitting the wall with our Ethernet bandwidth and panelview requirements or something similar ?

I am debating to suggest getting one more Internet card (it would be 3 in total, and putting the third one and a HMI on separate subnet, but am not sure if it would help since it is still the same PLC that has to do the work and poll the data and do the calculation ??? Also that would require pulling separate ethernet cables so those two network dont share the same switches either.

I am not sure if I miss anything, but if I did, I would be glad to add more explanation.

Thanks for reading,

Darko

Hi! My first suggestion would be to look at using an 'intelligent switch' rather than the one you've got.

In itself, this might solve your problem.

Robot_Man said:

Hi! My first suggestion would be to look at using an 'intelligent switch' rather than the one you've got.

In itself, this might solve your problem.

Click to expand...

You mean some IGMP snooping switches ?
I have no experience with those at all. We have 3 unmanaged switches (Hirchman's rail types).

I agree with the previous post. Try using a managed switch. Hirschman has them, so does Moxa, N-tron, wiedmuller, etc. Your local distributor or tech support should be able to assist you in the configuration. I would confirm that help before purchasing the equipment.

Regards

Try hooking up to the network with your pc and open internet explorer. Type in the address of the ENBT cards and use that to help troubleshoot the problem. It will show you if you are at the limit of the cards. It doesn't sound like you should need another card, you just need to manage the network.

I use managed hirschmann switches on my network and they have worked great. When I first came here they had a linksys dumb switch and the system seemed to work fine. What are you using for the business network? Do you have a firewall installed to seperate the two networks?

Kevin

Kev77 said:

Try hooking up to the network with your pc and open internet explorer. Type in the address of the ENBT cards and use that to help troubleshoot the problem. It will show you if you are at the limit of the cards. It doesn't sound like you should need another card, you just need to manage the network.

I use managed hirschmann switches on my network and they have worked great. When I first came here they had a linksys dumb switch and the system seemed to work fine. What are you using for the business network? Do you have a firewall installed to seperate the two networks?

Kevin

Click to expand...

We have two different ENBTs with different subnets.

darkesha said:

We have two different ENBTs with different subnets.

Click to expand...

Try looking at each ENBT address in your browser and use the diagnostics to help.

It also could have something to do with how you have your screens set up for the panelview. Do you have the displays set to replace or on top? The problem could exist in the panelview port and adding another ENBT will not correct that. Do you have any datalogs set up?

Look at your RPI's see if any are lower than 10 msec.

Check produced/consumed tags each tag uses 1 TCP connection and 1 CIP connection

You need a managed switch.Hands down

Like kev77 said check your hmi application for replace or on top

You may see problems with the hmi depending on which screens are active.If set to replace only the tags associated with objests on the currently displayed screen are being used by the hmi so depending on how much is on a screen or how busy it it it can change the traffic in a major way.

Check the refresh and update times of objects on the hmi app such as graphs and numeric displays.

Make sure no scaling is taking place in the hmi

For any type of remote i/o like flex or point try to setup as rack optimized if no analog or intelligent cards used in that rack.

Use a fluke or other brand of cable verify tools to check cable quality.Bad connectors or ethernet cables running close to high voltage wiring causes major intermittant problems.

Check ENBT firmware for issues as well as any comm cards or remote i/o adapters.

Make sure you mare not maxing out connections on any of the ethernet cards. Try to stay below 80 percent load if above 80 you need an adapter that supports more connections.

Make sure you cables/connectors are right stranded cable/connector for stranded some connectors do both but most are one or the other.

Make sure there is not a lot of excess cable. a few feet okay but i went on one job where they used a 250 spool for each run and piled the excess in the bottom of the cabinet.This causes problems also.

Make sure nothing is connected to a POE Device.

My big money is on the switches though.I use ab stratix as you can specify what equipment is on each port and helps build a more accurate routing table.

Agreeing with all of the above: I'm currently working with a networked system of 8 PLC's and one SCADA PC. They are networked with a Hirschman Managed Switch, and each taget local control panel has a Hirschman unmanaged switch with the PLC and 4 VSD's (inverters) plugged in.

There was a problem with the managed switch, so I temporarily put in an unmanaged switch and it could only cope with the main PLC, my laptop and 2 local control panels. Adding another I kept getting comms timeouts.

When the managed switch was put back in (with a bit of tweaking to filter out broadcasts) the network load was something like 3% of capacity!!

Managed switches cost a bomb compared with bog-standard unmanaged, but I tell you my friend, that is the way to go.

Definitely check the quality of the network cabling. The same job used 'made to measure' cables rather than off the shelf and 50% of them were intermittently causing drop-outs due to poor crimping.

Agreeing with all of the above, I would also suggest that you change the continuous task to a periodic task running at 100-200 ms if you can tolerate that rate. Do you really need the main task running at 10ms? Are there any overlaps?

I also suggest setting the number of unconnected buffers in the ControlLogix to the maximum of 40. Search the knowledgebase for "unconnected buffers". There is a KB item that describes how to do that.

Guys, thank you so much for so many helpful tips. As soon as production allows I will start working on them, one by one.
As for managed switches, what would you recommend beside the mentioned ab stratix switch?
As for the position of the this particular switch, does it go right after the ENBT card in particular, or it can go anywhere on the network.
We have 2, one is right in the PLC cabinet, and the other one is in the control room. The one in the control room is used to connect to HMI and lots of flexes which implies it's the "busier" one.

I only have experience with the Hirschmann managed switch. The basic setup software is sufficient, however if you want to do any clever monitoring the 'management' software is quite expensive.

In terms of connection, nominate a sensible place for the Managed Switch to live: There is no hierachy in the network as everything is effectively 'peer to peer'. Use unmanaged switches within a local control cabinet. From the unmanaged switch (I also used Hirschmann) connect to any port on the Managed Switch.

It was sensible in my case to group connections in the Local Control Panel where the local PLC was communicating with four VSD's, then one port fed back to the the Managed Switch.

As we were using a CompactLogix system with 1969-L32E CPU, we did not need a separate Ethernet card in the PLC rack and simply used the CPU Ethernet port. In the Local Control Panels this connected to the unmanaged switches and in our 'main' panel we plugged the one identical PLC straight into the Managed Switch.

Remember to check everything has unique Static IP addresses: If your PC somehow clashes with IP address of course you will have problems also.

Good luck!

danny.michael said:

Agreeing with all of the above, I would also suggest that you change the continuous task to a periodic task running at 100-200 ms if you can tolerate that rate. Do you really need the main task running at 10ms? Are there any overlaps?

I also suggest setting the number of unconnected buffers in the ControlLogix to the maximum of 40. Search the knowledgebase for "unconnected buffers". There is a KB item that describes how to do that.

Click to expand...

Yes, the 10ms task is adequate if not too slow in order to control water pressure going from 0-10k PSI.
As for the other tasks and setting them up to 100 - 200ms, I am afraid we would see the delays which would confuse the OPs as well as machine in the Auto Cycle. They all work good as is..

Some additional data would be helpful:

Main Task - Properties - Monitor:
What are the scan time and interval times?

Periodic Task - Properties - Monitor:
What are the scan time and interval times?

Specifically, look at the Main Task interval time. This is the time for one complete pass thru the continuous task. If your Periodic Task is taking a significant portion of 10ms, it can be surprisingly long.

Comms compete with the continuous task, same thing applies. If the periodic task isn't leaving much time, they have problems.

Mellis is correct. Comms compete with the continuous task. The "Communications Time Slice" setting in the controller properties dialog box controls how much time the controller spends on communications relative to the time it takes to complete one cycle of the continuous task. If the Time Slice is set to the default of 20%, then the controller will allot 20% of the time it takes to complete a scan of the continuous task to communications. You might try setting the Time Slice to 50% or more, but your best bet is to change the continuous task to periodic. Without a continuous task, the controller allots all the time left between scans of the periodic tasks to communications.

In my opinion, you should move everything out of the 10ms task that does not need to operate at that scan rate. Are all 25 subroutines devoted to controlling the water pressure? If not, try moving them to a slower task.

You do have to be careful, however, to be sure digital signals generated in one task are available if needed in the other tasks. For instance, a one shot generated in a 100ms task will be on for 10 scans of a 10ms task. On the other hand, a one shot generated in the 10ms task may not be see at all in a 100ms task.

In my experience, operating without a continuous task is best for communication speed. For most everything I do a 50 to 100 ms task works fine.