1756-en2t

I am currently using a 1756-EN2T with firmware 5.28

The CPU utilization starts at about 50% and will slowly creep up over the next 60mins all the way to 100%. The connections remain at 50. After a few minutes at 100%, I am unable to connect to the EN2T to view the diagnostic screen. I am still able to ping it.

At this point, if I go offline, it wont allow me to connect anymore.

The only way around this is to power down the EN2T.

I am using cisco SG-300-10 switches. One configured as Layer 3 while the others are Layer 2. IGMP snooping is enabled. There are about 8 cisco switchs on this machine in a start topology. 10 Redlion HMI's. 10 - 1734-AENT. 3 Kinetix 300 drives. 2 Realtime automation devicenet to ethernet/IP gateways. 11 Festo Ethernet/IP valve banks. 1 DVT camera. and 1 Keyence camera. 3 Labview and 1 VB application communicating ethernet/IP.

When I am able to look at the diagostic screens on the EN2T, I dont see the number of connections increasing, just the cpu utilization.

All the devices on the network are set for 60RPI except the kinetix 300 which is set for 20RPI

From my understanding, TCP and CIP connections have different max connection values. But for some reason, when I cant connect to the EN2T through the webserver, I also cant make additional CIP connection.

Any Ideas?

When the line isnt running the CIP unconnected packets/second is at 300. Once the line starts running, this slowly increases to about 2000 packets/second and then I lose connection. During this time, the connections dont increase. HMI packets/second stay steady at 100-120.

What might cause this increase? How do I narrow it down?

When the line stops running, the CIP packets/second goes back down and I'm able to connect again.

Why do you have 1 switch at layer 3? Do you have different subnets and vlan's?

Check connections and speed and duplex sttings of all devices.

Is you enbt firmware up to date?

Which processor are you using? Most L6 sries will do 250 connections max and L7 will do 500. Not sure how many connectionjs the en2t handles but it is a lot.

Is spanning tree enabled on your switches?

Yes, Our machine is on a seperate vlan then the rest of the plant. Thats why we use one Layer 3 switch at the gateway, the other switches on the machine are layer 2.

Firmware is at latest release

CPU - L62

EN2T is 250 connections, but I dont believe we are using all the connections. We are using 100% of the EN2T cpu. That is where I believe we are having connection issues from.

Rapid spanning tree is enabled

What firmware are you running in the L62? This determins if you are running multicast or unicast by default.

Make sure you point IO is rack optimized and check you speed / duplex and negotiagtion for each device / port and make sure they are the same.

At this point you will need tools to do a proper diagnosis so download wire shark or rent a fluke etherscope for a couple days.

L62 revision 20.11
All Point IO is rack optimized
all port are auto negotiate, full duplex

What is the firmware rev of your en2t?

Are you doing producer /consumer? How many?

He mentioned firmware rev 5.28 in the first sentence of the first post. Come on Kid, you're slipping!

OG

EN2T current release 5.28
No Producer/Consumer tags

It seems like there is a lot going on with that network. Especially with all the IO racks you have with your HMI stations. It does not sound like your IO is isolated from your HMI/Computer stations which raises a red flag.

What is the firmware version? I think 17 or 18 allows you to set you IO connections to use unicast, however with your managed switches this probably won't significantly change your performance.

The other red flag is your RPI settings. RPI directly affects your packets per second, simply having them at defaults is not the best practice. You really need to understand RPI. Grouping RPI speeds is critical in larger applications to best optimize your communications. For instance remote racks with digital IO only could be set to 20ms. Racks containing analog IO set to 40ms, your Devicent/Ethernet adapters to 80ms, drives to 160ms, the festo valve banks to 320ms..etc. It's important to understand this and AB has an Ethernet capacity tool to help calculate your network demands, changing RPI changes the burden on the network and I think you should review it. Consult your Rockwell vendor to see if they have someone that can review and make recommendations.

The other aspect is the PLC program and how it's organized. Rockwell has a tool called a Task Monitor Tool. This will show all the processes in your system to be monitored and report various stats. It shows which tasks/programs are most demanding on your PLC, shows your network connections. So you may have to adjust your overhead time slice if you're running a continuous task to allow for better communication performance. If you have periodic tasks you may have to look at the priority, and the task times, maybe you have a bunch of task overlaps? Maybe the program is just not optimized correctly and it's too demanding leading to the CPU being busy processing logic and your communications is getting left behind.

With all those HMIs hitting your PLC, it seems to be excessive and I wouldn't be surprised if all the traffic from those is causing issues. May need to consider a dedicated IO server so a single connection to the PLC is made and the HMI stations hit the IO server (However I don't know if the Redlion HMIs are capable of OPC comms or other..).

You've got your work cut out for you if you are familiar with that I'm referring to, and to top it off if you can't get online with the processor the Task Monitor Tool is useless. Might need to do some bench testing with some of this if you have a spare processor available.

EDIT: I have to believe you're having intermittent issues with the machine itself? The HMI displays slow/unresponsive? Intermittent/slow IO response??

Operaghost said:

He mentioned firmware rev 5.28 in the first sentence of the first post. Come on Kid, you're slipping!

OG

Click to expand...

Yep I missed that one.This 2-3 hrs of sleep during the week is messing with my head a few days in.

@ The OP the amount of devices you have you may want to put the HMI's on a seperate vlan from IO and drives which is a common practice on large systems, HMI's can be chatty and it is also much easier to troubleshoot issues like this if the network is segmented.

Another gotcha I have seen is the IGMP querier. Make sure only 1 switch is set as querier by default most switches and all cisco do a auto setup of this and it can go wrong. On a automation system I like to set it manually in the front office I don't care. The querier should be the switch with the lowet IP address and I would think that would be the unit you have set at layer 3 which in a sense is your core switch.

Mkae sure each switch is using the same IGMP version aslo and make sure all the switches have the same firmware. On the cisco small buiness series switch that you have it should have dual memory so make sure if youflash firmware that you set the correct memory as active.