Apologies, it's been a hectic few days, and so says the human race...
You mean 1756-L75
That is double or more than the recommended limit of 3,000 alarms per L7x controller. Depending on how many alarms are on scan at any one time, the controller's scan time and CPU utilization may be heavily impacted. To further reduce this impact - on top of trying to limit alarms to 3,000 per controller, it is recommended to limit the number of these alarms in "transition" at any one time to just 1000 per controller i.e. alarms being triggered, acknowledged, suppressed, reset, etc. This condition, of an accumulation of concurrent alarms in transition, is known as an alarm "burst".
Monitor L7 CPU performance...
While you can use the Logix 5000 Task Monitor to assess your controller's performance, do note that once you have a Continuous Task the CPU usage will always be high (100%), or will seem high. This is because the processor never stops working i.e. it does not have a Null Task. When it is not processing other tasks (Periodic, Event, etc.) it is processing the Continuous Task. Because of this, simply reading the CPU usage is not an accurate indication of the actual load the user tasks are placing upon the CPU.
Instead, CPU usage should be based on the ratio of the Continuous Task rate to the desired execution time for this task. This desired execution time is best determined by using the Watchdog Timer preset. However, many do set this value a little to a lot higher than required for the task so as to avoid nuisance faults. Or they just leave it at the default (500ms). So the ratio from actual to desired may be somewhat exaggerated. Instead, though, we should try to provide the actual desired time for this task by monitoring its last and max. scan times for a period, and then setting the Watchdog Timer value to just above the max. observed. Anyway, I'm tangenting!...
As the Continuous Task will show 100% CPU usage, by monitoring the rate for the Continuous Task, we can better determine if the code we are adding, such as thousands of alarm instructions, is having a detrimental effect.
If no Continuous Task, then we can look directly at the CPU usage for the other user tasks. Without a Continuous Task we will have a Null Task i.e. a percentage period where no user tasks are being processed. If we subtract the Null Task percentage from 100 percent, the remainder will the be the CPU usage for the user tasks, comms, etc.
Example: Null Task = 80%
100 - 80 = 20% CPU Usage (for User Tasks)
If that were the other way around, then we would have to monitor carefully that precious 20% left of CPU usage, especially if adding thousands of alarm, or other instructions.
You might find that the controller CPU usage or Continuous Task rate is relatively acceptable, given the number of alarms you have per controller, but it is the communications modules you need to monitor more closely...
For L7x ControlLogix, where we have separate communications modules, these are also monitored separately for CPU usage and Connections in the Task Monitor. Not over burdening communications modules would be one of the key factors they look at in deciding the limit for alarms overall (3,000), and in burst mode (1,000). So check your comms module CPU usage and also check their webpage counters, errors, etc.
It is fine to have the HMI, Data, Alarm Servers, View Studio, etc., all on one machine if suitably powerful enough and the application is not a large resource hog and if it only hosts a relatively small number of clients. However, it is important to be aware of the following - throwing a single machine with a multi-core CPU and a lot of RAM at these applications might give us a sense of boundless resources for which the application Servers may continue to consume at will. However, it does not work like that. The Windows Operating System limits the maximum amount of RAM utilized by a 32-bit application...
A Data Server (RSLinx Enterprise/FactoryTalk Linx) is limited to 3GB RAM but does work better with multiple cores/parallel processing. Its CPU utilization is typically 20% but this depends upon volume and scan rates, tag data types (optimized packets or not) and number of data clients (eg. View Studio, HMI Server).
Utilization for an Alarm Server will depend on which state it's in - steady or burst. When in steady, and again, depending upon how many of which type tags and at what rate; the CPU utilization will be typically 20% while always OS limited to 2GB RAM. No matter how many alarms you throw at a single Alarm Server, it's processing will not be permitted to consume more than this memory limit. But note, it's CPU utilization will steadily increase, possibly burdening the overall machine performance. Now throw frequent alarm bursts on top of this already heavily laden, but memory restricted Server machine, and you may end up here asking us these head scratching questions.
The final, but most important deciding factor in whether to host Servers on different machines, or not, is the HMI Server. On a relatively normal application, the Data and Alarm Servers will run at the reasonably low and typical 20% CPU utilization while also being RAM restricted. Whereas, the HMI Server is highly configurable and so is not as easily estimated up front. One of the key considerations here is the number of distributed clients for the application.
Some rules of thumb for the Hardware and Operating System...
The Hardware - For smaller systems (around 10 clients) a typical computer specification for an Application Server would be an Intel Core 2 Duo or an Intel Core i5 Standard Power processor with 4 GB RAM. For medium to large systems (10 - 20 or more clients) a Quad Core Intel Core Xeon (server chip) processor with 8 GB RAM is recommended. While more RAM never hurts, there are no general recommendations for large amounts of RAM, even for large Application Servers. As the Server machine should be dedicated i.e. not tasked with any other processes other than those of the network distributed system, 8 GB RAM should be more than adequate for the small number of servers it will typically be running.
The Operating System - Applications hosting 20 or more clients should really only be considering server class machines. So that is typically a Quad Core Intel Core Xeon processor with 8 GB RAM, or more if you so wish. For the Operating System, again we will look at the 10 client cut-off point. Just to point out, a "client" does not just mean an SE client running a graphical application. A client can be any one of the following -
FactoryTalk View SE Clients
FactoryTalk View Studio
FactoryTalk View SE Administration Console
FactoryTalk Historian SE Connector
FactoryTalk Transaction Manager Connector
FactoryTalk View SE Server (other)
So client counts can accumulate pretty quickly depending on the outlay.
For 10 or fewer client connections, the recommended operating systems are...
Windows 10 Enterprise
Windows 10 Professional
Windows 7 Ultimate with Service Pack 1
Windows 7 Enterprise with Service Pack 1
Windows 7 Professional with Service Pack 1
These are Workstation operating systems which have a 20 device connection and EULA limitation which may impede proper server functionality where many clients are required to connect concurrently.
For more than 10 client connections, the recommended operating systems are...
Windows Server 2016 Standard
Windows Server 2016 Datacenter
Windows Server 2012 R2 Standard
Windows Server 2012 R2 Datacenter
Windows Server 2012 Standard
Windows Server 2012 Datacenter
Windows Server 2008 R2 Standard with Service Pack 1
Windows Server 2008 R2 Enterprise with Service Pack 1
These are server class operating systems which have no concurrent device connection limitations and are ideal for larger client-based applications.
On a suitably specified server machine, you should only host one of each server type - SE Server (HMI Server), FactoryTalk Linx Data Server (Data Server), FactoryTalk Alarms and Events Server (Alarm Server) and FactoryTalk Network Directory. If a network application requires multiple HMI Servers, Data Servers, or
Alarm Servers to distribute the load,
it is recommended that you install the necessary software and run the Servers on multiple host computers.
So, for your 95,000 alarm (and growing) application...
You are advised (as rep also advised) - FTAE Servers that you create to distribute the alarms should be both limited to 10,000 alarms each and up to 10 FTAE Servers maximum for the application. This allows the maximum tested of 100,000 alarms per application for a ControlLogix L7x based system. This would mean, ideally, you need all 10 FTAE Servers with 9 using the maximum 10k alarms and the last using 5k alarms (with spare for your increasing number of alarms). Even if you close one eye to this, and look to place 16k alarms on each FTAE Server, you would only need 6 servers to allow you up to 96k alarms. If you go to 7 x 16k FTAE Servers you are allowing your application grow to 112k alarms. Also remember, each controller should only be polled by 3 of these FTAE Servers.
And to the crux of the issues you are facing...
Either way, with several FTAE Servers in play, you are continuing to ask for serious resource trouble by splitting this many alarms out, while still hosting everything on one "server" machine. It may, in fact, make things worse as you would now have several processes running on the one machine, with the same large number of alarms.
Add Servers...Add Hardware.
Regards,
George