FactoryTalk Studio Alarm Issue

Sham

Member
Join Date
Sep 2019
Location
Australia
Posts
152
hi all,

So I am using FT studio for huge project with my team, and we are facing alarm issues, which are not making sense to me. We have about 95,000 alarm definitions, and expected to increase only.

Sometimes, the alarm show up in the banner, and they can disappear the next second. Also, we use excel file to import alarms in the FT studio, and that process is getting shady now too. for example, if I import alarms from a file, its successful, but once I export that file back during my backup process, the size of the file increase (by 3 times, which is strange a bit), and I get errors like ''not enough memory resources are available to complete this operation". How/why is it possible?

Also, sometimes the alarm server in ft studio (local server, I mean just using my local VM for making changes in the code before deploying them) start giving me errors like "could not connect to the alarm and event server". It normally gets fie once I start the client and open the alarms again in studio.
Also, I have face the error " failed trying to save the configuration file hr= c004b010". Why?

Why is this happening, any clue? Atm, we have only one alarm server, should I make more server, so that the number of alarms gets divided?

Also, please give me suggestions about how can I structure the project in a better way? atm, we have only one alarm server, should be get another redundant server? use more VMs? It a huge project, and any suggestions are appreciated.

Thanks
 
FactoryTalk View SE?...Which Version?...Which Controller(s)?...

Sham said:
...We have about 95,000 alarm definitions, and expected to increase only...

...we have only one alarm server, should I make more server, so that the number of alarms gets divided?

Also, please give me suggestions about how can I structure the project in a better way? atm, we have only one alarm server, should be get another redundant server? use more VMs? It a huge project, and any suggestions are appreciated...

Had you checked into the limits allowed for your application before designing it and going live?

Which controller(s) are you using and how many alarms are configured per controller?

The ControlLogix L7x platform, for instance, recommendeds 3,000 alarms maximum per controller with up to 3 FactoryTalk Alarms and Events (FTAE) Servers subscribed. They have also been tested up to 100,000 alarms per application. However, each FactoryTalk Alarms and Events (FTAE) Server is recommended to be limited to 20,000 alarms each.

Each HMI Server supports up to 2,000 alarms on an Alarm Summary. Any more and they'll start to drop off, as I think you are experiencing?

Are the Server(s) running on the same machine (PC) or dedicated? Are all machines suitably spec'd (powerful enough)?

Your 95,000 alarm count is getting close to the upper tested alarm limit for an L7x based application. Depending on the power of the Server machine(s), this can start to impact heavily on resources, such as memory, CPU utilization, and also scan times. If using an older platform, these limits drop dramatically.

You should at a minimum look to add more FTAE Servers and split the total alarms across these.

See here for more on the limits...

ID: IN8132 | Access Levels: Everyone
FactoryTalk Alarms and Events Performance Limits

Regards,
George
 
We are using control Logix L75-1756. How do I check my limit on the CPU? by noting the modules I have downloaded in that PLC, and count the alarms? surely its gonna be more than 3000.

So you are saying that we should divide the alarms into more servers, right? Our Rockwell support guy said that it should be ideally 10,000 alarms in one server, but it looks difficult. My plan is to make 7 different servers, and it will be around 16k alarms in each.

Also, by alarm summary, you are referring to the Alarm Banner right? It has a limit of displaying 2,000 alarms, right?

Thanks a ton!
 
Had you checked into the limits allowed for your application before designing it and going live?

Which controller(s) are you using and how many alarms are configured per controller?

control logix L75-1756. I think around 5-7000 alarms per controller.

[/QUOTE]
Are the Server(s) running on the same machine (PC) or dedicated? Are all machines suitably spec'd (powerful enough)?
[/QUOTE]

Its running on the same machine as the studio, not a dedicated machine for the alarm server. The machine has 24 gb ram and 200 gb hd. Good enough? what do you recommend?

[/QUOTE]
Your 95,000 alarm count is getting close to the upper tested alarm limit for an L7x based application. Depending on the power of the Server machine(s), this can start to impact heavily on resources, such as memory, CPU utilization, and also scan times. If using an older platform, these limits drop dramatically.
[/QUOTE]

What are you recommendation for the machine resources?
 
Geo Server 1: 10k word limit...

Apologies, it's been a hectic few days, and so says the human race...

Sham said:
control logix L75-1756. I think around 5-7000 alarms per controller...

You mean 1756-L75 :)

That is double or more than the recommended limit of 3,000 alarms per L7x controller. Depending on how many alarms are on scan at any one time, the controller's scan time and CPU utilization may be heavily impacted. To further reduce this impact - on top of trying to limit alarms to 3,000 per controller, it is recommended to limit the number of these alarms in "transition" at any one time to just 1000 per controller i.e. alarms being triggered, acknowledged, suppressed, reset, etc. This condition, of an accumulation of concurrent alarms in transition, is known as an alarm "burst".

Monitor L7 CPU performance...

While you can use the Logix 5000 Task Monitor to assess your controller's performance, do note that once you have a Continuous Task the CPU usage will always be high (100%), or will seem high. This is because the processor never stops working i.e. it does not have a Null Task. When it is not processing other tasks (Periodic, Event, etc.) it is processing the Continuous Task. Because of this, simply reading the CPU usage is not an accurate indication of the actual load the user tasks are placing upon the CPU.

Instead, CPU usage should be based on the ratio of the Continuous Task rate to the desired execution time for this task. This desired execution time is best determined by using the Watchdog Timer preset. However, many do set this value a little to a lot higher than required for the task so as to avoid nuisance faults. Or they just leave it at the default (500ms). So the ratio from actual to desired may be somewhat exaggerated. Instead, though, we should try to provide the actual desired time for this task by monitoring its last and max. scan times for a period, and then setting the Watchdog Timer value to just above the max. observed. Anyway, I'm tangenting!...

As the Continuous Task will show 100% CPU usage, by monitoring the rate for the Continuous Task, we can better determine if the code we are adding, such as thousands of alarm instructions, is having a detrimental effect.

If no Continuous Task, then we can look directly at the CPU usage for the other user tasks. Without a Continuous Task we will have a Null Task i.e. a percentage period where no user tasks are being processed. If we subtract the Null Task percentage from 100 percent, the remainder will the be the CPU usage for the user tasks, comms, etc.

Example: Null Task = 80%
100 - 80 = 20% CPU Usage (for User Tasks)

If that were the other way around, then we would have to monitor carefully that precious 20% left of CPU usage, especially if adding thousands of alarm, or other instructions.

You might find that the controller CPU usage or Continuous Task rate is relatively acceptable, given the number of alarms you have per controller, but it is the communications modules you need to monitor more closely...

For L7x ControlLogix, where we have separate communications modules, these are also monitored separately for CPU usage and Connections in the Task Monitor. Not over burdening communications modules would be one of the key factors they look at in deciding the limit for alarms overall (3,000), and in burst mode (1,000). So check your comms module CPU usage and also check their webpage counters, errors, etc.

Sham said:
Its running on the same machine as the studio, not a dedicated machine for the alarm server. The machine has 24 gb ram and 200 gb hd. Good enough? what do you recommend?

It is fine to have the HMI, Data, Alarm Servers, View Studio, etc., all on one machine if suitably powerful enough and the application is not a large resource hog and if it only hosts a relatively small number of clients. However, it is important to be aware of the following - throwing a single machine with a multi-core CPU and a lot of RAM at these applications might give us a sense of boundless resources for which the application Servers may continue to consume at will. However, it does not work like that. The Windows Operating System limits the maximum amount of RAM utilized by a 32-bit application...

A Data Server (RSLinx Enterprise/FactoryTalk Linx) is limited to 3GB RAM but does work better with multiple cores/parallel processing. Its CPU utilization is typically 20% but this depends upon volume and scan rates, tag data types (optimized packets or not) and number of data clients (eg. View Studio, HMI Server).

Utilization for an Alarm Server will depend on which state it's in - steady or burst. When in steady, and again, depending upon how many of which type tags and at what rate; the CPU utilization will be typically 20% while always OS limited to 2GB RAM. No matter how many alarms you throw at a single Alarm Server, it's processing will not be permitted to consume more than this memory limit. But note, it's CPU utilization will steadily increase, possibly burdening the overall machine performance. Now throw frequent alarm bursts on top of this already heavily laden, but memory restricted Server machine, and you may end up here asking us these head scratching questions.

The final, but most important deciding factor in whether to host Servers on different machines, or not, is the HMI Server. On a relatively normal application, the Data and Alarm Servers will run at the reasonably low and typical 20% CPU utilization while also being RAM restricted. Whereas, the HMI Server is highly configurable and so is not as easily estimated up front. One of the key considerations here is the number of distributed clients for the application.

Some rules of thumb for the Hardware and Operating System...

The Hardware - For smaller systems (around 10 clients) a typical computer specification for an Application Server would be an Intel Core 2 Duo or an Intel Core i5 Standard Power processor with 4 GB RAM. For medium to large systems (10 - 20 or more clients) a Quad Core Intel Core Xeon (server chip) processor with 8 GB RAM is recommended. While more RAM never hurts, there are no general recommendations for large amounts of RAM, even for large Application Servers. As the Server machine should be dedicated i.e. not tasked with any other processes other than those of the network distributed system, 8 GB RAM should be more than adequate for the small number of servers it will typically be running.

The Operating System - Applications hosting 20 or more clients should really only be considering server class machines. So that is typically a Quad Core Intel Core Xeon processor with 8 GB RAM, or more if you so wish. For the Operating System, again we will look at the 10 client cut-off point. Just to point out, a "client" does not just mean an SE client running a graphical application. A client can be any one of the following -

FactoryTalk View SE Clients
FactoryTalk View Studio
FactoryTalk View SE Administration Console
FactoryTalk Historian SE Connector
FactoryTalk Transaction Manager Connector
FactoryTalk View SE Server (other)

So client counts can accumulate pretty quickly depending on the outlay.

For 10 or fewer client connections, the recommended operating systems are...

Windows 10 Enterprise
Windows 10 Professional
Windows 7 Ultimate with Service Pack 1
Windows 7 Enterprise with Service Pack 1
Windows 7 Professional with Service Pack 1

These are Workstation operating systems which have a 20 device connection and EULA limitation which may impede proper server functionality where many clients are required to connect concurrently.

For more than 10 client connections, the recommended operating systems are...

Windows Server 2016 Standard
Windows Server 2016 Datacenter
Windows Server 2012 R2 Standard
Windows Server 2012 R2 Datacenter
Windows Server 2012 Standard
Windows Server 2012 Datacenter
Windows Server 2008 R2 Standard with Service Pack 1
Windows Server 2008 R2 Enterprise with Service Pack 1

These are server class operating systems which have no concurrent device connection limitations and are ideal for larger client-based applications.

On a suitably specified server machine, you should only host one of each server type - SE Server (HMI Server), FactoryTalk Linx Data Server (Data Server), FactoryTalk Alarms and Events Server (Alarm Server) and FactoryTalk Network Directory. If a network application requires multiple HMI Servers, Data Servers, or Alarm Servers to distribute the load, it is recommended that you install the necessary software and run the Servers on multiple host computers.

So, for your 95,000 alarm (and growing) application...

You are advised (as rep also advised) - FTAE Servers that you create to distribute the alarms should be both limited to 10,000 alarms each and up to 10 FTAE Servers maximum for the application. This allows the maximum tested of 100,000 alarms per application for a ControlLogix L7x based system. This would mean, ideally, you need all 10 FTAE Servers with 9 using the maximum 10k alarms and the last using 5k alarms (with spare for your increasing number of alarms). Even if you close one eye to this, and look to place 16k alarms on each FTAE Server, you would only need 6 servers to allow you up to 96k alarms. If you go to 7 x 16k FTAE Servers you are allowing your application grow to 112k alarms. Also remember, each controller should only be polled by 3 of these FTAE Servers.

And to the crux of the issues you are facing...

Either way, with several FTAE Servers in play, you are continuing to ask for serious resource trouble by splitting this many alarms out, while still hosting everything on one "server" machine. It may, in fact, make things worse as you would now have several processes running on the one machine, with the same large number of alarms.

Add Servers...Add Hardware.

Regards,
George
 
Geo Server 2: Reached my 10k limit on that last one...

Your 200 GB HDD...

As a rule, you should specify a hard drive that has enough disk space to provide virtual memory that is at least twice the size of the physical RAM that is required. In other words, just reserve twice the disk space for the RAM they advise for a server class machine i.e. 16 GB disk space for 8 GB RAM. You would not need to reserve 48 GB of disk space just because you have 24 GB RAM. The server processes, if you will remember are OS memory limited, so they will not use most of the 24 GB RAM. This frees up more disk space, and RAM, for other tasks, such as Historian, data logging and trending, if required.

Sham said:
...by alarm summary, you are referring to the Alarm Banner right? It has a limit of displaying 2,000 alarms, right?...

They are different graphical objects, but at the alarm count and access level from FTAE Servers, the Alarm and Event Banner and the Alarm and Event Summary are one and the same. It's most likely that you have resource issues and the HMI client application is dropping connections to the FTAE Server which can take several minutes to be reestablished. That's just an educated guess, based on the little info you've provided us here. It's also possible you may have a corrupted Alarm and Event Banner and it may need to be deleted and recreated, as it does happen. Note, these graphical objects are not limited to accessing 2,000 alarms from the FTAE Server. You may have many more downloaded or device-based ALMA/ALMD alarms linked to them but should limit the amount of these that are active or in burst mode, at any one time, to 2,000 or less. There are methods to help you achieve this by grouping, prioritizing and creating dependencies between alarms.

Redundancy -

A Secondary Server's reliability is only as good as that of the Primary. Redundancy is implemented to fail over in the event of an unexpected failure of the Primary. You should not introduce Redundancy in an attempt to mitigate inherent Primary Server design flaws. Get the base design correct, and then, if you think you/they may need Redundancy, make the base the Primary and replicate its good design to the Secondary.

Virtualization -

The original CPR 9 release of FactoryTalk Alarms & Events did not support Redundancy so Rockwell partnered with a company called Marathon Technologies (Stratus owned) who provide server virtualization. This is sold as a high availability, fault-tolerant CoServer and branded as "everRun MX". This CoServer software runs on suitable server class hardware. The Rockwell/Marathon collaboration developed the tailored version "everRun FT". This provides hardware redundancy by combining two physical servers into one virtual server. In the event of a hardware failure, the duties of the failed hardware are seamlessly assumed by its redundant partner. This approach provides high availability, fault tolerance, and disaster protection for FTAE Servers. To implement an FTAE solution using the Marathon everRun FT technology, hardware and software will need to be specifically implemented to achieve the desired high availability. Again, typically requiring you use a standard off the shelf dual processor or hyper‐threaded, server class machine.

Using "standard" VM solutions, and also OEM releases of operating system software as opposed to Microsoft releases, is not advised, but alas, widely implemented. Job's got to get done, bills got to get paid, right?

But for SE, and whether using redundancy and/or virtualization, you've got to design these network distributed systems properly, at the architectural and recommended level.

Shortcut, shortcoming.

Regards,
George
 
Last edited:
Could l say Geospark, that was a very informative piece, l imagine you are a speed typist because if l could write that it would take me hours, let alone not having the info at hand.
I wonder if Sham releases how much effort you put into that to answer his question (but like most questions, the op could just RTFM) and felt his response to you effort a little bit underwhelming, but for all/anyone who has a similar issue your post should be on RA Knowledge Base, albeit then no one would be able to find it.
 
Last edited:
Apologies, it's been a hectic few days, and so says the human race...



You mean 1756-L75 :)

That is double or more than the recommended limit of 3,000 alarms per L7x controller. Depending on how many alarms are on scan at any one time, the controller's scan time and CPU utilization may be heavily impacted. To further reduce this impact - on top of trying to limit alarms to 3,000 per controller, it is recommended to limit the number of these alarms in "transition" at any one time to just 1000 per controller i.e. alarms being triggered, acknowledged, suppressed, reset, etc. This condition, of an accumulation of concurrent alarms in transition, is known as an alarm "burst".

Monitor L7 CPU performance...

While you can use the Logix 5000 Task Monitor to assess your controller's performance, do note that once you have a Continuous Task the CPU usage will always be high (100%), or will seem high. This is because the processor never stops working i.e. it does not have a Null Task. When it is not processing other tasks (Periodic, Event, etc.) it is processing the Continuous Task. Because of this, simply reading the CPU usage is not an accurate indication of the actual load the user tasks are placing upon the CPU.

Instead, CPU usage should be based on the ratio of the Continuous Task rate to the desired execution time for this task. This desired execution time is best determined by using the Watchdog Timer preset. However, many do set this value a little to a lot higher than required for the task so as to avoid nuisance faults. Or they just leave it at the default (500ms). So the ratio from actual to desired may be somewhat exaggerated. Instead, though, we should try to provide the actual desired time for this task by monitoring its last and max. scan times for a period, and then setting the Watchdog Timer value to just above the max. observed. Anyway, I'm tangenting!...

As the Continuous Task will show 100% CPU usage, by monitoring the rate for the Continuous Task, we can better determine if the code we are adding, such as thousands of alarm instructions, is having a detrimental effect.

If no Continuous Task, then we can look directly at the CPU usage for the other user tasks. Without a Continuous Task we will have a Null Task i.e. a percentage period where no user tasks are being processed. If we subtract the Null Task percentage from 100 percent, the remainder will the be the CPU usage for the user tasks, comms, etc.

Example: Null Task = 80%
100 - 80 = 20% CPU Usage (for User Tasks)

If that were the other way around, then we would have to monitor carefully that precious 20% left of CPU usage, especially if adding thousands of alarm, or other instructions.

You might find that the controller CPU usage or Continuous Task rate is relatively acceptable, given the number of alarms you have per controller, but it is the communications modules you need to monitor more closely...

For L7x ControlLogix, where we have separate communications modules, these are also monitored separately for CPU usage and Connections in the Task Monitor. Not over burdening communications modules would be one of the key factors they look at in deciding the limit for alarms overall (3,000), and in burst mode (1,000). So check your comms module CPU usage and also check their webpage counters, errors, etc.



It is fine to have the HMI, Data, Alarm Servers, View Studio, etc., all on one machine if suitably powerful enough and the application is not a large resource hog and if it only hosts a relatively small number of clients. However, it is important to be aware of the following - throwing a single machine with a multi-core CPU and a lot of RAM at these applications might give us a sense of boundless resources for which the application Servers may continue to consume at will. However, it does not work like that. The Windows Operating System limits the maximum amount of RAM utilized by a 32-bit application...

A Data Server (RSLinx Enterprise/FactoryTalk Linx) is limited to 3GB RAM but does work better with multiple cores/parallel processing. Its CPU utilization is typically 20% but this depends upon volume and scan rates, tag data types (optimized packets or not) and number of data clients (eg. View Studio, HMI Server).

Utilization for an Alarm Server will depend on which state it's in - steady or burst. When in steady, and again, depending upon how many of which type tags and at what rate; the CPU utilization will be typically 20% while always OS limited to 2GB RAM. No matter how many alarms you throw at a single Alarm Server, it's processing will not be permitted to consume more than this memory limit. But note, it's CPU utilization will steadily increase, possibly burdening the overall machine performance. Now throw frequent alarm bursts on top of this already heavily laden, but memory restricted Server machine, and you may end up here asking us these head scratching questions.

The final, but most important deciding factor in whether to host Servers on different machines, or not, is the HMI Server. On a relatively normal application, the Data and Alarm Servers will run at the reasonably low and typical 20% CPU utilization while also being RAM restricted. Whereas, the HMI Server is highly configurable and so is not as easily estimated up front. One of the key considerations here is the number of distributed clients for the application.

Some rules of thumb for the Hardware and Operating System...

The Hardware - For smaller systems (around 10 clients) a typical computer specification for an Application Server would be an Intel Core 2 Duo or an Intel Core i5 Standard Power processor with 4 GB RAM. For medium to large systems (10 - 20 or more clients) a Quad Core Intel Core Xeon (server chip) processor with 8 GB RAM is recommended. While more RAM never hurts, there are no general recommendations for large amounts of RAM, even for large Application Servers. As the Server machine should be dedicated i.e. not tasked with any other processes other than those of the network distributed system, 8 GB RAM should be more than adequate for the small number of servers it will typically be running.

The Operating System - Applications hosting 20 or more clients should really only be considering server class machines. So that is typically a Quad Core Intel Core Xeon processor with 8 GB RAM, or more if you so wish. For the Operating System, again we will look at the 10 client cut-off point. Just to point out, a "client" does not just mean an SE client running a graphical application. A client can be any one of the following -

FactoryTalk View SE Clients
FactoryTalk View Studio
FactoryTalk View SE Administration Console
FactoryTalk Historian SE Connector
FactoryTalk Transaction Manager Connector
FactoryTalk View SE Server (other)

So client counts can accumulate pretty quickly depending on the outlay.

For 10 or fewer client connections, the recommended operating systems are...

Windows 10 Enterprise
Windows 10 Professional
Windows 7 Ultimate with Service Pack 1
Windows 7 Enterprise with Service Pack 1
Windows 7 Professional with Service Pack 1

These are Workstation operating systems which have a 20 device connection and EULA limitation which may impede proper server functionality where many clients are required to connect concurrently.

For more than 10 client connections, the recommended operating systems are...

Windows Server 2016 Standard
Windows Server 2016 Datacenter
Windows Server 2012 R2 Standard
Windows Server 2012 R2 Datacenter
Windows Server 2012 Standard
Windows Server 2012 Datacenter
Windows Server 2008 R2 Standard with Service Pack 1
Windows Server 2008 R2 Enterprise with Service Pack 1

These are server class operating systems which have no concurrent device connection limitations and are ideal for larger client-based applications.

On a suitably specified server machine, you should only host one of each server type - SE Server (HMI Server), FactoryTalk Linx Data Server (Data Server), FactoryTalk Alarms and Events Server (Alarm Server) and FactoryTalk Network Directory. If a network application requires multiple HMI Servers, Data Servers, or Alarm Servers to distribute the load, it is recommended that you install the necessary software and run the Servers on multiple host computers.

So, for your 95,000 alarm (and growing) application...

You are advised (as rep also advised) - FTAE Servers that you create to distribute the alarms should be both limited to 10,000 alarms each and up to 10 FTAE Servers maximum for the application. This allows the maximum tested of 100,000 alarms per application for a ControlLogix L7x based system. This would mean, ideally, you need all 10 FTAE Servers with 9 using the maximum 10k alarms and the last using 5k alarms (with spare for your increasing number of alarms). Even if you close one eye to this, and look to place 16k alarms on each FTAE Server, you would only need 6 servers to allow you up to 96k alarms. If you go to 7 x 16k FTAE Servers you are allowing your application grow to 112k alarms. Also remember, each controller should only be polled by 3 of these FTAE Servers.

And to the crux of the issues you are facing...

Either way, with several FTAE Servers in play, you are continuing to ask for serious resource trouble by splitting this many alarms out, while still hosting everything on one "server" machine. It may, in fact, make things worse as you would now have several processes running on the one machine, with the same large number of alarms.

Add Servers...Add Hardware.

Regards,
George

I will take this all knowledge bit by bit to digest, but I guess you are really more helpful than our Rockwell guy, whose being paid heaps as well..
 
GeoSpark, I have started putting this all in a proposal. But there is one thing that I don't still understand. If the limit of displaying alarm is 2000 for the alarm banner, wont I always be having issues in the alarms being displayed? Because even after we divide the alarm server, there will be only one alarm server, and we will mostly be having more than 2000 alarms active at one time. What do you reckon?
 

Similar Topics

Hello, I made a change in alarm setup in factory view studio, where I changed a alarm message text. After that I made a run application and...
Replies
0
Views
70
I converted my PV 1000 to PV+6 1250. The main capabilities i wanted was remote access and a way to log alarms and retrieve them remotely. I was...
Replies
4
Views
2,889
Since we've upgraded to Factorytalk Studio v9.0, the font in our alarm history pages, and current alarm pages is huge once we do a download. I...
Replies
2
Views
2,970
Hi everyone! I have a FactoryTalk Alarm Editor question for you guys. I recently completed an install that involved adding two new assets to my...
Replies
8
Views
5,910
Hi all. I'm trying to use the AE_InAlmUnackedCount() function in FactoryTalk View Studio 8.0. If I enter the function itself into a numeric...
Replies
2
Views
5,090
Back
Top Bottom