RE: FactoryTalkView SE - Running really slow/bad comms.

RE: FactoryTalkView SE - Running really slow/bad comms.

Hello Gents

I'm looking for some indepth ideas on how FactoryTalk is actually working with regards to how much time it spends processing VBA etc and troubleshooting with some very slow comms issues.

The background is that the client had/has and RSView32 SCADA application running on Windows XP - it communicates with 2 x Micrologix PLC's, initially this was probably OK (they are set to 19.2k) but through the years NET-ENI/D devices were added to get the units on Ethernet and presumeably help speed comms up. In my experience the system isn't asking for a mega amount of data and polled at 1 second via RSLinx topics.

My task was to change from RSView to FactoryTalk View SE (8.00) - I'm no newb to FactoryTalk and understand the architecture etc..I've also converted apps before but for better PLC's such as CompactLogix..however my new system runs like a dog, way slower than the old one.....I've tried every trick in the book - slowed the poll rates down in RSLinx, slowed the screen update times in FactoryTalk, removed any tags that had duplicate addresses - all of that kind of stuff but no joy. I believe the slowness of the communications is pulling down my whole system and often is causing VBA code not to execute. I'm also experience things like changing a setpoint on screen never changes at the PLC yet when you do a 'set' for a tag value to the same place from a test button it works. Sometimes values make it to the PLC - sometimes they seem to get lost in the ether or turn up minutes later.

So has anybody got any ideas on what I can do to get this thing to work/speed up.....at this point I'll take any suggestions on the chin of what I should do.

Regards
Russ

Have you tried using RSLinx as the communications server instead of Linx Enterprise?

If you are using Enterprise, put some of the predefined tags on a display to see what the comms rates are for real, and if there are any errors / collisions.

Hi - Yes it's using RSLinx Gateway not Linx Enterprise, what predefined tags ? Are there some system tags I can look at for comms ?

Some tricks are just not in the book!

rtgibson said:

Hi - Yes it's using RSLinx Gateway not Linx Enterprise, what predefined tags ? Are there some system tags I can look at for comms ?

Click to expand...

In RSLinx, for topic comms errors, go to DDE/OPC>Communication Events.... Here you will see the number of logged communication errors for all of your topics. When an error is selected, you can see the total number of occurrences for this error, and when it last occurred at the bottom of the window. See if you have any timeouts or "Out of Buffer Space" errors.

rtgibson said:

...looking for some indepth ideas on how FactoryTalk is actually working with regards to how much time it spends processing VBA etc and troubleshooting with some very slow comms issues...

Click to expand...

Russ,

It's a case of "How long is a piece of string? & "How many knots does it have?"...

The main difference after converting from RSView32 to FTView SE is that all of your VBA code used to be run on the Server and the results pushed out to the Clients, whereas now it's run on the individual Clients. Depending on the amount and complexity of VBA code used, long loops, etc., this may or may not have an impact on the Clients performance after conversion. Also, you need to be sure you have proper error checking in your code. Unchecked errors in your VBA code could impact on the HMI Client's performance significantly.

These factors primarily dictate how long the Client will "spend processing" VBA code and hogging resources.

But, at first glance, I'm not really convinced it's your VBA code that's at fault here? I do have a theory on it, but I'll come back to that later. So if we park the VBA code for a minute and concentrate on some other possible issues first...

rtgibson said:

...In my experience the system isn't asking for a mega amount of data and polled at 1 second via RSLinx topics...

Click to expand...

The second I read slow comms using View SE and ENI in the same setup, alarm bells started going off. From what you've described, the 1761-NET-ENI is your biggest suspect for bottlenecking in this system.

The 1761-NET-ENI is not really designed for use in a FTView SE distributed HMI system, where it could be expected to pass relatively large amounts of data between its host and another device, or lots of smaller data between several devices. It's more for placing a single controller onto an Ethernet network with limited comms to one device, like an Ethernet terminal. Typically you would only setup one OPC topic to an ENI.

In case you don't, you should know the "number of connections" and "packet size" limitations for the ENI...

Connections:
The 1761-NET-ENI supports 6 connections at any one time using one of these configurations...

2 INCOMING
2 OUTGOING
2 EITHER WAY

Or...

4 MAXIMUM INCOMING / 2 OUTGOING RESERVED

Or...

4 MAXIMUM OUTGOING / 2 INCOMING RESERVED

A MSG in from another controller uses 1 incoming connection.
A MSG out to another controller uses 1 outgoing connection.
An OPC topic uses 1 incoming connection.
An RSLinx Classis driver browse uses 1 incoming connection.
A web browser session to the ENI uses 1 incoming connection.

So you can see how easy it might be have several MSG instructions coming and going, OPC topics, browses of one sort or another, all fighting for limited resources.

On top of that, if a connection is established on the ENI, and then goes idle, it has to timeout the connection before freeing it up for another transaction. This timeout is fixed at 20 seconds. For a MSG coming in or going out, the MSG will error on its own timeout, but for an OPC topic, the controller will just seem to "disappear". For outgoing connections, the ENI will timeout a failed connection after a fixed 6 seconds.

Also, the RPI for the ENI is fixed at 5 seconds for polling.

Packets:
Fragmentation...
If "Use Maximum Packet Size (Ethernet)" is selected in RSLinx for the topic, then the likes of an SLC 5/05 controller will attempt to optimize packets for efficient transfer. This can result in an optimum packet size being calculated and larger packets being fragmented before transfer and then defragmented upon arrival.

The ENI/MicroLogix does not support fragmented packets in the same way as the SLC 5/05 does. It also does not support large optimized packets. The above setting could create an optimized packet size of over 100 contiguous words and say a remainder fragmented packet of 20 words. So leaving this setting enabled for an ENI/MicroLogix can create fragmented or oversized packets at the ENI which will simply be discarded. It's possible that certain data will succeed in passing on some occasions and failing to pass on others, resulting in sporadic control behaviour.

If this setting is enabled, then you should be able to monitor how it is otimizing the packets by viewing DDE/OPC>Optimized Packets...

Limit Maximum Packet:
In RSLinx, this sets the maximum number of queued packets to be sent to a controller at any one time. The default is 20 and usually works well for Ethernet, but only when using robust Ethernet controllers like ControlLogix, CompactLogix and SLC 5/05. They have larger connection counts and larger comms buffers.

Depending on what and how many errors you may be getting for your topics in the Communication Event Log, you may need to adjust this down to reduce comms issues. For the likes of the limited ENI/MicroLogix, you should start with a low number and adjust upwardly until you get errors, and then back a touch.

There are other considerations like simultaneously using Polled and Unsolicited data updates to the HMI, but I'll leave it there for now.

If their old system ran reasonably well, and you find all the above to be satisfactory, then we may be back to my theory I mentioned above. It is possible that the additional overhead of the VBA code to the HMI Client may be pushing an already teetering communications configuration over the edge?

Lessons for the day...

Avoid using 1761-NET-ENI interfaces in a View SE distributed HMI system.

Likewise, avoid using Ethernet MicroLogix controllers for heavy data concentration in View SE distributed HMI systems.

Regards,
George

Thanks

Hi George and thanks for an excellent response, give me time to digest it and I'll respond over the next few days.

If you want to try a device that may (probably will) perform better than the NET-ENI, have a look or a trial with a Digi One IAP. I have had some apps in the past that did tons better with the Digi device for a lower cost and increased reliability.

Swapping that one device might make enough difference that it's the only thing you have to do and at under $800 $600 ('merican dollars), it is not going to be money wasted compared to the time you might spend working on other aspects of your project.

If it doesn't work there, you will probably find it useful somewhere else...

RSLinx Enterprise uses up to 4 CIP connections for read operations, and 1 for writes (total of 5 CIP connections). If you have RSLE running on your PC and it is polling data to the NetENI, well you could max out the connections in the NET Eni.

RSLinx Enterprise does have a new feature to change the default, so it only uses 1 CIP for reads and 1 for writes. I would check the CIP connection usage in the Net Eni.


If you migrated from RSView32 to FTView, likely the project was using HMI tags. Those will double your scan time in FTView (ex. update rate is 1 second, then you actually get 2 second updates with FTView with HMI tags). Consider changing the tags to direct reference. I only say consider, because you lose some features such as scaling, HMI tag based alarming, and min/max settings that the HMI tag database provides. But, the update rate performance can be worth it in some cases.

Take a look at the below AB performance document.
RSLinx Enterprise is significantly more efficient with computer resources than RSLinx Classic
https://rockwellautomation.custhelp.com/app/answers/detail/a_id/33990

arlenjacobs said:

RSLinx Enterprise uses up to 4 CIP connections for read operations, and 1 for writes (total of 5 CIP connections). If you have RSLE running on your PC and it is polling data to the NetENI, well you could max out the connections in the NET Eni...

Click to expand...

arlenjacobs,

I'm not entirely sure if you're quoting the above CIP numbers against the connection limitations I've mentioned above, or not, but in case you are, or others make that assumption...

The connection limitations mentioned above, for the 1761-NET-ENI, refer to its TCP connections, not CIP. They are the limitations on the number of physical devices that can be connected to it at any one time. A TCP connection is used to first establish a connection between two devices. This is at the Transport Layer (OSI), as opposed to a CIP connection, which is at the Application Layer (OSI). A TCP connection is required to be established first before a CIP connection can be made. A TCP connection, established between two devices, may allow multiple CIP connections to traverse it.

So, if RSLE is reading from an ENI, one TCP connection is used out of a possible six. If RSLE did end up using four CIP connections for this read, they would still only use one TCP connection.

However, those four CIP connections will never be established because...

Although the limit for each TCP connection on the ENI is four CIP connections and five unconnected sends, a few versions ago in FTV, they introduced restrictions on the maximum number of CIP connections allowed between RSLE and certain limited devices, such as the ENI. Only one CIP connection is now allowed per TCP connection. Another good reason to avoid using the ENI for heavy traffic.

arlenjacobs said:

...RSLinx Enterprise does have a new feature to change the default, so it only uses 1 CIP for reads and 1 for writes. I would check the CIP connection usage in the Net Eni...

Click to expand...

The above mentioned limit of one is done automatically, so the manual method of editing the allowed CIP per TCP connection is not required here.

Regards
George

Thanks for clarifying Geospark.
Yes, I was talking about something new. There are limits for both TCP and CIP connections in the Net-Eni. Max 4 CIP connections through 1 TCP connection. And, I don't recall where it is documented but I thought the Net-eni also has a maximum total CIP connection (6?) limit as well. It was one of the reasons why the RSLE defaults were changed for new installations.


But for you RTGibson, no likely that is not the problem. I just saw Net-eni and had "red flag" reactions. As Geospark points out the Net-eni is a starting point and possible bottleneck. At least you can find the IP addresses of the devices connecting to it to start your troubleshooting path.

arlenjacobs said:

...I don't recall where it is documented but I thought the Net-eni also has a maximum total CIP connection (6?) limit as well. It was one of the reasons why the RSLE defaults were changed for new installations..

Click to expand...

No, the four CIP per TCP connection, that I also mentioned above, can be cumulative.

If we look again while adding the limits per TCP connection...

The 1761-NET-ENI connection limitations are...

6 TCP CONNECTIONS MAX
2 INBOUND
2 OUTBOUND
2 EITHER WAY

4 CIP CONNECTIONS PER TCP CONNECTION
5 UNCONNECTED SENDS PER TCP CONNECTION

So to get our totals...

6 TCP X 4 CIP = 24 CIP MAX.
6 TCP X 5 US = 30 UNCONNECTED SENDS MAX.

These are the ENI's native limits. Other imposed limitations may exist to restrict these limits, as in the case of RSLE only allowing one CIP per TCP connection to an ENI, but these are external to the ENI.

The documentation you will find this in is it's User Manual under Appendix C...
Sorry no link, phone is acting up...Search > 1761-um006_-en-p.pdf

There is good information on message buffers and sizes, on particular PLC packet sizes and typical packets/sec data throughput rates, using a MicroLogix 1500 with an ENI and an RSLinx OPC Test Client. Very pertinent to the matter at hand.

Regards
George

Again thanks for the answers guys. I'm not using RSLinx Enterprise - it's the standalone Gateway version I've heard rumours (just rumours - don't know if there's any truth in it) that Enterprise doesn't handle communications to 'older' devices as good....so we went for a full blown RS Linx license. Anyway....

The system is in Poland and I am now back in the UK to have a rethink....

It can't be my VBA code, it's so simple (nothing fancy) and even when commented out so that it doesn't run the system doesn't behave any differently.

That's interesting what somebody said about HMI tags and what we could call 'direct' addressing. The system is indeed setup as HMI tags as you'd expect - but in the past on a different system I did have to change to directly addressing 'N' registers etc from NumericDisplays on screen rather than HMI tags - it did work but I don't understand why....in the end though we ended up adding a CompactLogix in between the SCADA and the old PLC5's (on DH+) on that system to eventually cure the issues. That may be an option on the job in Poland - what do you guys think about that idea ?

Russ