CLX L61 scantimes suffering after V20 upgrade

Has anyone else noticed this issue?

We have several L61's that have been running v16.03 f/w for years with no real issues. Lots of motion, e-net comms, d-net. Most of the motion control stuff is done in timed tasks. All of the production, start/stop, and alarming are done in a continuous "main task".

Prior to upgrading to V20.11, these continuous tasks were executing at ~ 15ms, with a max of 21. Now we are seeing an avg of 30 with peaks of 45-50ms on the same task, same hardware.

Looks like quite a bit of code optimization can be done, but we're talking about a LOT of work, which was unnecessary before. Just wondering if anyone else is dealing with similar issues. We upgraded all the sercos and ENBT's obviously, just wondering what else we can do without re-writing a lot of complex code.

Thanks,
Dustin

bmacattack33 said:

We have several L61's that have been running v16.03 f/w for years with no real issues.

...

Prior to upgrading to V20.11, these continuous tasks were executing at ~ 15ms, with a max of 21. Now we are seeing an avg of 30 with peaks of 45-50ms on the same task, same hardware.

Click to expand...

I know this is no help, but ya'll broke two cardinal rules of maintenance!

1) If it ain't broke don't fix it.
2) Stay off the bleeding edge of software releases.

I wish I could help, but it will be some time down the road before we move past v17.

Sorry for rubbing it in.

Paul

Well, you could always flash back...

How have the scan times affected production? Is it causing real problems? Have you used the task monitor tool to better monitor whats going on in the processor?

What about your task priority settings? RPI? There is a lot that goes into scan times and communications with a processor so there are all sorts of variables to check. But if it's a problem, take that step back.

Re-writing code would be last resort, I'd be more interested in all of your task times, task priority, RPI settings, and what you're actually seeing using the task monitor tool. Maybe you have task overlaps, maybe you need change task time settings because one task is interrupting another.

While the version change may have brought on these problems, it's certainly a good "education" into how the processor handles various items while it scans and you can learn from it. Very well could be that the settings in the code are not ideal anyway, and this is bringing it to your attention.

OkiePC said:

I know this is no help, but ya'll broke two cardinal rules of maintenance!

1) If it ain't broke don't fix it.
2) Stay off the bleeding edge of software releases.

I wish I could help, but it will be some time down the road before we move past v17.

Sorry for rubbing it in.

Paul

Click to expand...

Preaching to the choir, Paul. I would counter with rule #3: Maintenance never makes decisions, only deals with the fallout.

The main push for v20 at this point was to facilitate the integration of FT AssetCentre into our plant network. Looks good on paper, the version control and protection, etc, but not at the expense of scan times.

Paully's5.0 said:

Well, you could always flash back...

How have the scan times affected production? Is it causing real problems? Have you used the task monitor tool to better monitor whats going on in the processor?

What about your task priority settings? RPI? There is a lot that goes into scan times and communications with a processor so there are all sorts of variables to check. But if it's a problem, take that step back.

Re-writing code would be last resort, I'd be more interested in all of your task times, task priority, RPI settings, and what you're actually seeing using the task monitor tool. Maybe you have task overlaps, maybe you need change task time settings because one task is interrupting another.

While the version change may have brought on these problems, it's certainly a good "education" into how the processor handles various items while it scans and you can learn from it. Very well could be that the settings in the code are not ideal anyway, and this is bringing it to your attention.

Click to expand...

On a few machines, we have seen some dip in machine efficiency. The main gripe at this point though is screwy production tracking values caused by timing errors. We have seen a serious hit on axis registration performance also caused by scan-time. I foresee quality time with the task monitor on this for the next few days at least.

Unfortunately, because of the corporate push for AssetCentre, flashing back is not an option. But the migration to v20 is happening slowly enough that we still have several v16 or v17 machines that we can study for comparison.

The issues that we've seen already have brought some poor programming to our attention, namely triggering bits in a timed task off of a tight axis position window, and then using these bits to trigger events within the continuous task. Now these things need to be corrected, and I'm sure there are similar cases of illogically prioritized tasks, RPI's etc, but we didn't expect upgrading firmware to expose any latent programming flaws. If anything, I would have expected the opposite. C'est la vie, I guess.

There are several things that jump out at me. The continuous task could probably stand to be scheduled relatively slowly, say ~50ms. There are also quite a few axis values that are being SSV'd every scan, which in most cases is unnecessary.

Doesn't look like there will be any easy fix to these issues, though. It will take careful analysis and planning, and there's no way around that. I'm just interested to hear how anyone else has dealt with a similar situation, or has seen this with v20.

Thanks again,
Dustin

Let us know what you find Dustin, and good luck.

Your post is very timely for me. I have been thinking of upgrading a bunch of our ControlLogix Systems from V15.01 so we can use AOIs.

Anybody have any recommendations as to which Firmware Version would be best to Upgrade to? (Obviously not V20 I think)

Right now we have V15 and V17.

Stu....

Stu,

We upgraded some of our packaging equipment from v16 to v17 about 1.5 years ago, access to AOI's being one of the chief reasons. Didn't see any performance issues, and we're running some pretty fast stuff on cartesian robots. I think v17 is pretty well a proven commodity.

bmacattack33 said:

Stu,

We upgraded some of our packaging equipment from v16 to v17 about 1.5 years ago, access to AOI's being one of the chief reasons. Didn't see any performance issues, and we're running some pretty fast stuff on cartesian robots. I think v17 is pretty well a proven commodity.

Click to expand...

Thanks Dustin, I appreciate your advice. We don't have any true Motion Control to date. The one ControlLogix V17 we have came that way from the OEM, and so far it seems fine. Stable with no issues to my knowledge. (One of the Systems I mentioned in JeffKiper's Rant).

Stu....

Since we've recently and painlessly upgraded some ten L61s from 16 to 19 (Motion, Positioning, SCADA, CNET-EtherNet/IP networking,...., the Works) I think your issues are related to the age of the L61 CLX family.
RA announced that V20 will be the highest firmware revision the L6xs will support; any further developments will be applied to the L7x family only.
My guess is that they've realized that L6x is pretty much a thing of the past and probably this is the reason L7x are more competitively priced than their older counterparts. This is Rockwell's way of saying "replace the hardware" and I think this is what you should be considering, especially when looking ahead.
The L6x family is more than ten years old; in the Automation world this is pretty much "close to collecting Social Security"...:nodi:

RA is moving from PLCs to PACs.
The L7x series has capabilities that support the PAC theory at which the L6x does not.

There's always a reason to change isn't there!

Do you have the ability to incrementally flash another machine? Say to V17 - V18 - V19 - V20. I'd be curious if this is specifically seen with V20, or in a previous version.

I have an L73 on my desk, and can perform a test of my current project to see if there is a difference in scan times between V19 and V20. My program is substantial enough that I should see a variance if one exists. Unfortunately I don't have an L63 laying around so I can't go into further firmware to help confirm what you are seeing.

Wouldn't hurt to call Rockwell if you haven't already.

Paully's5.0 said:

There's always a reason to change isn't there!

Do you have the ability to incrementally flash another machine? Say to V17 - V18 - V19 - V20. I'd be curious if this is specifically seen with V20, or in a previous version.

I have an L73 on my desk, and can perform a test of my current project to see if there is a difference in scan times between V19 and V20. My program is substantial enough that I should see a variance if one exists. Unfortunately I don't have an L63 laying around so I can't go into further firmware to help confirm what you are seeing.

Wouldn't hurt to call Rockwell if you haven't already.

Click to expand...

I spoke briefly with Rockwell yesterday, assuming that basically all I would get is a push to upgrade to the L7x series, and that would solve our problems. Surprisingly (and dishearteningly) enough, the tech told me that the issue with v20 is the speed of the motion instructions execution, and that there really isn't a significant improvement with the L7 series. Unfortunately, I didn't get to go too far in depth with him. I don't know if performance will be improved with later revisions to version 20, or with version 21.

I'd like to get an L61, and an L71, or L72 side by side (with identical axis module configs). Take an identical program and step up from v16, v17...v20 and clock the scans at each iteration. Probably be next week before I can do this, even if I can get ahold of enough spare hardware.

You can tell with version 20 that RA is adding a lot of tools to the bag. Probably part of the push away from PLC's toward PAC's. I'm wondering if all the extra "functionality" is making the program more cumbersome. I notice, for one, a lot more options for networking and tag access. I'm sure I'm just scratching the surface.

Which reminds me of another potential pitfall for anyone else who wants (or is forced to) surf the bleeding edge: Beware of upgrading to v20 if your program contains any AOI's or UDT's created with v17. I don't know if v20 is the first version to allow user selectable control of external read/write access, but I do know that this functionality did not exist in v17. Our programs had several elements of AOI add-on defined tags that for some reason defaulted to "read-only" after the upgrade to v20. Which turned out to be a real head-ache because one of these common elements was axis jog speeds, which we write to with our HMI's. Like I said, these tags were not created with any sort of protection, and most tags seem to default to "read/write" when upgraded, but not all.

It "seems" that only elements that are used in motion instructions default to read-only, but I can't say for sure. It's a fairly simple fix, but it does require an offline change to the AOI or UDT definition, and then a download. Now I can see why external access control is a nice feature to have, but I hate surprises, especially in a tight production environment.

@dmargineau,

I'm curious, you say painless, but have you checked your scan times before/after? From what I have read, it seems like v19 to v20 was more a "sea change" than previous revisions, as Oakley said. I'm betting that the differences are pretty minimal until v20.

What you're saying about Social Security, ain't it the truth? My company is investing millions in this equipment, and there are always a dozen projects going at once. A constant tug-of-war between production and engineering, you know how it goes. :handball:With all that money being spent, they are still watching every dime, and it's hard to get any project approved without a significant "cost savings" factor. We've got probably 200 L61 processors in the plant, and those numbers scare ME. I can't imagine the glassy stare and cold sweats they would induce in our accountants if we started talking about replacing all of them. But at some point we won't have a choice, but for quite awhile I'm sure we will have to find a way to make do with what we have.

Anyway, I digress. Anybody recently go from v19 or lower to v20?

Cheers,
Dustin

Being around mostly Rockwell automation for quite some time now, I have picked up a line of logic which, the longer it's been running, the more accurate it proves to be.
I will never upgrade firmware but to the (Latest-1) at the most and, when announcements like the L61 vs. Supported Firmware are being made, I promptly start informing the "upstairs" that soon enough we will have to be ready to upgrade hardware.
It was pretty painful initially, however, the more the logic line proved to be right, the less questions THEY were asking...
I see your point regarding 200 CPUs...We have less than 100 on this site and we are the largest...
Upgrading from 16 to 19 was actually a huge performance improvement!; I think it had mostly to to with the EtherNet/IP Unicast Connection feature carried by one of the "skipped" firmware levels (18 I believe). When running on V16 firmware we used to do quite a lot of managed EtherNet/IP switching and this was quite "labor intensive" especially when additions/modifications were being made.
Our Historian database systems' tags data collection rate throughput increased almost 300% within the same "querry frequency" setting; since the Historian tags are "compressed" (recorded only when changed values) the RSLinx Enterprise data server vas "noticing" almost three times more changed values within basicaly the same system.
Extrapolating to the individual systems, the CPUs "situational awareness and decision making" was most likely three times more accurate due to the efficiency of the "unicasted" connections.
Consequently I am planning to upgrade all L61s to 19 within the next months.
In your situation, I would first try to downgrade to 19 at least some of the most V20 troublesome L61s and see if the current issues are quenched. One thing to remember though, I would meke sure all the system's components are being firmware upgraded to the highest revision supported by the controller's V19 one; this includes all the peripheral's comms modules, comms bridges, motion/speciality modules, etc.
You might have to live with 19 until sufficient funds will be available for the hardware upgrade; I feel V19 will suffice for quite some time...If you live in a Rockwell world you might have to adjust accordingly...

dmargineau said:

Being around mostly Rockwell automation for quite some time now, I have picked up a line of logic which, the longer it's been running, the more accurate it proves to be.
I will never upgrade firmware but to the (Latest-1) at the most and, when announcements like the L61 vs. Supported Firmware are being made, I promptly start informing the "upstairs" that soon enough we will have to be ready to upgrade hardware.
It was pretty painful initially, however, the more the logic line proved to be right, the less questions THEY were asking...
I see your point regarding 200 CPUs...We have less than 100 on this site and we are the largest...
Upgrading from 16 to 19 was actually a huge performance improvement!; I think it had mostly to to with the EtherNet/IP Unicast Connection feature carried by one of the "skipped" firmware levels (18 I believe). When running on V16 firmware we used to do quite a lot of managed EtherNet/IP switching and this was quite "labor intensive" especially when additions/modifications were being made.
Our Historian database systems' tags data collection rate throughput increased almost 300% within the same "querry frequency" setting; since the Historian tags are "compressed" (recorded only when changed values) the RSLinx Enterprise data server vas "noticing" almost three times more changed values within basicaly the same system.
Extrapolating to the individual systems, the CPUs "situational awareness and decision making" was most likely three times more accurate due to the efficiency of the "unicasted" connections.
Consequently I am planning to upgrade all L61s to 19 within the next months.
In your situation, I would first try to downgrade to 19 at least some of the most V20 troublesome L61s and see if the current issues are quenched. One thing to remember though, I would meke sure all the system's components are being firmware upgraded to the highest revision supported by the controller's V19 one; this includes all the peripheral's comms modules, comms bridges, motion/speciality modules, etc.
You might have to live with 19 until sufficient funds will be available for the hardware upgrade; I feel V19 will suffice for quite some time...If you live in a Rockwell world you might have to adjust accordingly...

Click to expand...

Thanks for the suggestions.

Thought I would share this:
After some more digging, I found this extremely helpful document, which has an attached Excel Workbook that breaks down memory allocation and execution times for the Logix instruction set by Catalog # and Firmware revision. Good stuff :nodi: Just wish I'd consulted it sooner.

What I'm finding is confirming my suspicions: Most instructions' execution times show a slight decrease from v17 to 18 to 19, and then more of a drastic spike for v20 (for the L6x series cpu's). This is probably due to the increased instruction functionality. The Motion instructions show the greatest increase in memory and exec times from v17 to v20, which is where we're taking the biggest hit. However one can see that the performance hit for taking an L61 processor to v20 is more than made up for by upgrading to the L7x series. In fact, it appears that most instructions are at least 2x as fast for an L7x as an L6x with the same firmware!

Now of course, I'd prefer to just upgrade all 200 of our processors and leave the code alone. Don't think I can count on that in this lifetime though. My time in this case is much cheaper than the new hardware, so I'm still looking at code optimization.

Jumping in just to say that L7x series with V20 has backplane comms issues when any single I/O module is configured with a 0.2mS RPI (the fastest RPI).

The system works fine, it's just that when you go online via the backplane, you get intermittent RSLinx comms, and freezing of the RSLogix5000 display. Increasing the controllers SOTS has no effect.

From what I've seen, unless you absolutely must go to V20, I'd leave it a while, there are obviously issues they need to resolve in V20.

I second the views already given - "If it ain't broke - don't fix it". Upgrading existing, perfectly functioning systems just because there is a newer version is IMHO not a wise choice. My view is you should only use the "newest" version if you need its functionality.

My 2c

After some extensive testing and tinkering, I was able to achieve an approximately 30% scan-time reduction on a couple of machines that had been upgraded to v20. This was enough to get down below v16 times, and which I was pretty happy with.
However we were still seeing quite a bit of poor registration performance. Registration marks that were a fixed distance apart were showing quite a bit of variance on axis travel between the registration events.
Finally, we picked a machine that that was performing especially poorly and swapped out the L61 cpu with an L72. The results were staggering!
L61, v16: 25ms (continuous task)
L61, v20: 38ms
L61, v20 (optimized) : 23ms
L72, v20 (original code): 7ms
So basically an identical program run on an L72 showed over a 500% improvement!
Not only that, but the registration error we were seeing was nearly eliminated, so axis registration position seems to be more accurately captured.
I don't know if these are typical results, and they are certainly better than what I expected, but it has definitely been made me a believer in the value of upgrading hardware. I would advise anyone out there contemplating upgrading an L61 to v20 to let this thread be a cautionary tale. We know that v20 is the end of the line for the L6 cpu's, and and it may be that v19 is really the "performance ceiling" for this product line.

Cheers,
Dustin