CLX Redundancy/Scantime/ControlNet

I have a rather large application that runs on ControlLogix with redundancy. The program is about 1.3 million bytes and it runs on L55M14's. It is very heavy in tag's and indexed code called by for/next loops. The crossloading is pretty heavy on the redundant pair. It has two ControlNets one for IO and one for HMI.

The HMI's communicate via another (3rd)rack and processor that handles all the data collection/HMI interaction. All data in and out of the redundant pair is produced/consumed or messaged accross the ControlNet's

This rack receives all the system data via ControlNet. It is polled by 3 identical OPC topics with several thousand tags on the scan.(via ENBT card) The system is also polled via PLC/SLC mapping by 1 soon to be 2 different Energy Metrics servers. My scantime is excellent on the HMI procesor, even running a 50% overhead timeslice to support all the communications. If I have my decimal place right it is about 6ms.

My problem lies in the redundant pair. I have 2 issues.

Scantime causing missed inputs on counters. Scantime is about 425-460ms
Controlnet unscheduled bandwidth insufficient to allow for expansion or online editing in a timely fashion.

My questions are.

I have disabled several large subroutines for testing purposes. The scantime stayed more or less the same. Is crossloading the bottleneck on my scantime? I suspect it is, but don't know how to validate this?

I am getting ready to add another ControlNet to the system. This will give me another path into and out of my redundant pair. Will this impact my scantime? I am not adding tags or ladder code just re-directing some data. (My ControlNets are scheduled the way they need to be, so please no Reschedule suggestions)

I want to configure listen only connections to several IO modules on a busy ControlNet. If I add a controller to my ControlNet Configuration and set the RPI's exactly the same as the owners of the IO will this add too much workload to my CNB's or be fairly transparent?

My discrete IO is already in a scheduled task and the logic gets updated more frequently than my scantime. I don't think I can reduce the scheduled execution time anymore as it will impact the rest of the controller's workload?

I have two solutions for the missed discrete bits. We already use some HSC cards that work well. I could add several more. I think however the solution might be to use another controller to read the IO status and through ladder logic make my own HSC. These pulses are not really high speed just faster than my scantime. I could then produce/consume the data to my redundant pair. I have some data that is critical for control that I will use the HSC counters for, but data collection type data can go the other route.

Any thoughts?
Thanks in advance.

If your only problem is slow input updates:

The control logix processor has a shared memory area that the inputs are read into as they happen. The inputs are not updated at the start of the scan but each input is updated in the CPU as it changes state. Scan time has no effect on missing a quick changing input, unless the input changes state more than once in the scan of the program.
Question:
Are you buffering the inputs into an internal register in your code and then using this value. Check for a copy or copy synchronous command.

If the input pulses on and off with a short changeover time then look at setting the card to initiate an event task.

If you are using the change of state of an input to record the time an event happenned then a slow scan time will impact here as by the time the code is scanned then XXX msecs will have happenned.
Have a look at event based tasks and set the input card to send time stamped data to the processor. Accurate to a couple of microseconds.


Regards Alan Case

Yes, I am aware that IO updates asynchronous from the scantime. This makes no difference when the pulse comes and goes before my logic has a chance to see it. Yes, the IO is buffered in a scheduled task. It has to be for several reasons.

I am sure that my IO image sees these pulses, but I cannot get around the buffered data that is written into this code. My data is all written to large arrays and scanned in a for/next loop. I have to map/buffer it to an array.

Event tasks are a no no with redundancy, and my scantime is almost a half second so yes it changes states and back in one scan.

I wish the solution was that simple. I have to get these pulses into the data array for my system to work properly. I can't think of anyway other than HSC or another processor producing the data accross ControlNet.

Maybe I am missing something obvious?

I might be missing something myself.
I thought the reason that IO was buffered before a routine was so that the routine was executed with the data at the one state, ie not changing part way through. It sounds like you require the IO to be updated during the scan of the program. How can capturing the IO in another processor and sending it as produce/consume to the executing program be any different then not buffering the required IO.

Regards Alan Case

The IO has to be buffered to an array. There is no way around it. It is normally buffered for the reason you suggested, but in this case it is buffered for a different reason. I have to get inputs into an array of about [300]. I have a fairly high IO count about 400 I think.

My system as it sets is just is not capable of catching all the pulses. I have to find a way to do counter type functions and get the data to my system.

If I capture the data and calculate a pulse sum/per time unit I can have a valid rate calculation, or I can just use my pulse sum for a totalizer type function. My system is missing some pulses on certain inputs giving me invalid data.

Does that make any more sense?

I can't think of an answer. How long does the scheduled taske itself take to run?
Alan

Scheduled task takes about 40ms. It is set to execute every 150ms. I have serious doubts about lowering it enough to help. There is a LOT of code in the continous task that has to execute with regularity or I have a whole bunch of other problems.

Any thoughts on controlnet issues from above?

My assumptions:
a) The IO is buffered for other than the schedulled task if the task only takes 40ms to execute and the task misses an IO change.

B) When all you tasks/programs execute then the scan can be up to 650ms

C) You think the controlnet redundancy is pushing the the scan time out to this value?

D) You are looking for a way to reduce the scan time of this processor?

Regards Alan.

Sorry I have to go now. Has been a long day

PS A guy named Ken Roach will probably be along later who is much more knowledgeable about CLX than I will ever be.
Regards Alan

1.Even though the Task only takes 40ms, it only executes every 150ms. The change of states is during the 150ms causing the count to be off some of the time. Lowering the scheduled time will cause other problems in the continuous task.

2.I think the crossloading of the redundancy is the main reason for my scantime. If so there is no fixing that. I would like to prove or disprove this though. It is possible all the indexed code is slowing it down too? I can't change that though.

3.I don't think I can reduce the scantime of this system.

4.Can I add another ControlNet to it without hurting my scantime even more? This addresses another problem I am having.

5.Can you add listen only connections to an existing network at the same RPI as the owner's connections without creating a lot more overhead for the CNB modules? There processors run at about 35-45% now.(the CNB's)

6.This has been and still is a very reliable system. It does it's job well, with the exception of missing some counts.

7.It is important to note, some of the inputs and almost all of the outputs on this sytem are critical. That's why it's redundant. The inputs that are used to gather data and make less critical decisions are the one's I am considering outsourcing. Critical info will be gathered by standard input or HSC on the redundant systems local IO.

8. Downtime to play with this system is very hard to get.

That about sums it up.
Thanks ahead of time to all who reply.

It is very heavy in tag's and indexed code called by for/next loops

Click to expand...

I am not familar with controllogix, but I have used for/next loops in
a PLC5. I used them for alarming examining bits, not a big change in scan time. When I tried to use them examining analog
( indexing about 100 points, this doubled my scan time 60ms to 120ms
I got rid of it).

I may be missing something but why using of EVENT task is NO NO with redundancy system? I did not know this.

If PLC scan is an issue why not to try much faster L6x processors (yes, L6x can be used with redundancy v 13.5)

Why ControlNet used for HMI communications instead of Ethernet?

Are your produced/consumed organized in arrays about 500 bytes each?

I don't think crossloads will significantly influence a scan time as they done by SRM.

I would look at Redundancy and Design Considerations manuals, but site is down and I have no paper copy here.

There are lots of valid points raised so far...but the key one is that the SRM cross scanning does increase the scantime by a factor of about 5 to 8.

There is an AB document that goes into complete detail on the issue of scantimes in a redundant system. In essence it is vital to ensure that the standalone system is running at the fastest possible rates before adding the the SRM.


Sorry I cannot locate the document on my own hard drive right now...but it is definitely on the website and it is more or less required reading for any SRM system

And yes you can add another CNB module without adding to the scantime.

I think PhilipW refered to KB document A54732518 - Optimizing a ControlLogix Redundancy System, Programming Guidelines and Case Study

..., increases in program scan was considered normal. Typically, a user will see at minimum a 4 to 5 times increase in program scan. That is if the program scan was 8ms before redundancy was enabled, it will increase to a minimum of 32ms to 40ms. Depending on program and tag layout the increase can be much greater; increases as much as 30 times have been seen. Right now, the increase in program scan is the greatest issue to overcome with a ControlLogix Redundancy System.

Click to expand...

in response to a few questions from Contr_Conn

The release notes for version 13 firmware for redundancy restrict the use of an event task in redundant systems. I don't know why, they just do.

I really could not implement any code using one very well any way, that scan time still comes from somewhere whether it's event driven or periodic. It would still impact the rest of my system.

Yes, the produce/consume UDT's are "packed" for efficient transfer.

We do use Ethernet for HMI. This system also uses some Ethernet based IO that we have a 3rd processor own. We also use this 3rd L55 to handle all the HMI overhead, and Energy Metrix data collection. All of the data is produce/consumed or messaged between the redundant pair and this 3rd L55. When this system was designed having Ethernet in the redundant system was not possible. It would be a HUGE task to convert it to that type of interface. I also think the overhead would just slow it down even more.



I had given thought to the L6 series. Just a little expensive when you already own 2 L55's. I think it is the only thing I have heard that would have an impact on system performance.

But, The firmware release notes state:

"• Even though the ControlLogix5561, 5562, and 5563 controllers execute logic faster, they must
still crossload data.
• Given the same project and redundant system, a ControlLogix5561, 5562, or 5563 controller is
up to 30% faster than a ControlLogix5555 controller."

Crossloading slows it down too. Still 30% of my 400-460ms scantime would be a kick in the pants.

I am going to try and come up with a test stand to load this code into to determine just how much of this scantime is the SRM and how much is the code. I can't afford to play around with the actual system.

I am still thinking that short term solution might be to use another processor with a listen only connection to this IO. My RPI is plenty fast enough to catch my counts.

Long term I am leaning towards a processor upgrade. I am already buying one L6 for another part of this system. If I can come up with the cash for another identical one I could make a trade out. The L55's are more than fast enough for most any other application.

Can I add a listen only connection at the same RPI as the owner of the IO without overworking the CNB's? With multicast I would think so but have never tried before.

Thanks for the replies guys. I appreciate everyone's time.

RSL