Can a power outage initiate editing in a compact logix plc?

[Russ]

I was at a customer's site today, and went to do some trending, and pid monitoring. When I went to go online with the processor I realized that the most recent version of the program wasn't compatible with what was on the processor.

I figured that a recent power outage might have caused the program to load from non-volatile memory, and the most recent edits might have been lost. They weren't critical to the operation, and I still had all of the edits saved, so I wasn't too concerned.

I compared what I uploaded with my most recent version of the program, and for the most part I noticed that the most recent modifications were gone.... no big deal....

However..... then I found this (rung 22)... (see attached jpg).

What could have caused this? Is it possible that a power outage could do this? I've never seen anything like it before, and am a bit concerned....



Thanks in advance...


This rung is part of a specialized automated control routine that was worked up specifically for this site. When doing the initial controls work, before the implementation of the automated portion of the code, a competing engineering company repeatedly downloaded the software, and took extensive photographs of the physical modifications - without permission from the OEM nor from the site.

The rung in the picture is a critical portion of the routine.

When I last left site, back in August, I realized I hadn't secured the processor using the CPU Security Tool. At the time, it seemed as if the attempts to steal the process had come to an end, and so I wasn't too concerned.

This routine allows a thermal oxidizer to remain operational throughout a wide variety of Air Liquide membrane and pressure configurations, at numerous bio-methane facilities. It automatically adjusts to the changing waste stream composition, allowing the plant to optimize production, without risk of inadvertent shut downs.

There has been remote access in the past, and it's possible that someone accessed the network remotely.

 

[daba]

Your picture shows a rung of code that is not in the controller, it is in the computer.

Notice the lower-case "e" - that means the rung is being edited, entered, or is in error, and in that state it cannot be transferred into the controller.

The editing procedures do not occur in the controller, only in the programming software, and only verified code can be sent to the controller. Once there you can Test Edits, Assemble edits.

The only logical conclusion is that this rung edit either existed in the offline file that you opened, or was put there after opening. Possible cause include dropping something onto your keyboard/mouse/touchpad etc., or electrical interference... are you using a wireless mouse or keyboard ?

 

[Russ]

Your picture shows a rung of code that is not in the controller, it is in the computer.

Notice the lower-case "e" - that means the rung is being edited, entered, or is in error, and in that state it cannot be transferred into the controller.

The editing procedures do not occur in the controller, only in the programming software, and only verified code can be sent to the controller. Once there you can Test Edits, Assemble edits.

The only logical conclusion is that this rung edit either existed in the offline file that you opened, or was put there after opening. Possible cause include dropping something onto your keyboard/mouse/touchpad etc., or electrical interference... are you using a wireless mouse or keyboard ?

Exactly!

Until yesterday, the most recent revision of the code was dated 8/13/2013. There were three previous revisions that were also written in August: on 8/6, 8/5, 8/4.

When I went to connect with the controller, I had opened up the most recent version, 8/13/2013. The program had been left in its initial state; I hadn't expanded the main task, nor had I opened up any of the individual routines. Like many, I simply selected the most recent version of the code, selected who active in the communications menu. I then selected the processor in network. Unfortunately, I wasn't able to go online with the processor because program on the controller was different than the version I opened in logix.

This didn't surprise me. Due to production, there are times when the most recent version of the program can't be stored in the non-volatile memory. Since the most recent changes weren't critical, I assumed that a previous version stored in the non-volatile had been loaded into the processor due to a loss of power.

I didn't want to lose any changes in the most recent revision and, to be safe, saved the program with yesterday's date. I selected 8/5 revision for merging, and was able to go online after changes had been uploaded. The moment the upload was finished, I performed a 'save as' under a new, temporary file name DECEMBER UPLOAD.

During this entire time I hadn't expanded the main task, nor had I selected any routines. I was simply trying to get online with the processor. I wasn't surprised, only mildly irritated that it was going to take me longer to start the trend, because I was there to work on something else.

I ran the Logix 5000 Compare program to compare the most recent revision (8/13 saved as 12/23) against the temp file I had just saved. There were a total of 9 ladder changes spread over four routines. The routine containing rung 22 was the last routine in the main task.

I looked at the changes in the first routine and it appeared that my earlier assumption was correct; the most recent updates had been lost because they couldn't be saved to the non-volatile.

I opened up the 8/13 version in another instance of 5000, and began adding the most recent edits to the version that I was online with. I used the compare program as a guide.

Edits to the first three routines went smoothly. When I opened up the final routine in the compare program, the first two edits were similar to the previous few; the most recent version of the program had been lost. Then I scrolled down to rung 22 in the compare program and found it. It was first found in the Logix 5000 compare program.

I alt-tabbed to the online program and scrolled down to rung 22. Sure enough, there it was. The rung was in edit mode.

I looked at rung 22 in my my most recent revision, and it was fine. My first thought was the exact thought you’d voiced, that I’d somehow saved an offline version with the rung in edit mode, or had somehow fat fingered the rung while online. Except, I hadn’t expanded the main task in any version of the program or begun any edits until after I had run the compare program. I’d simply merged and performed a couple of save as.
I then looked at the rung in the previous August revisions: 8/13, 8/6, 8/5, and 8/4. In each instance the rung was normal. Since I did a save as on the 8/13 revision, renaming it 12/23 in preparation for using Logix Compare. The most recent program modifications for any of the programs in the August folder is 8/14. Note, I keep all programming mods in their individual yearly/monthly folders, using the date as a simple way of keeping track of revisions, hence the August folder. J lol….
The 12/23 version (which is a save as of 8/13) was fine, and so were all the August revisions.

It’s why I wrote this post in the first place. I’m stumped.
[FONT=&quot]They have dirty power, and I've seen power-related plc faults with a slc 5/04, but all of those had to do with the pids.. Output Max and Minimum values would switch, or the set point would be outside the Setpoint Min/Max range, etc.... [/FONT]
[FONT=&quot]The only compact logix issue I recall had to do with flexbus initialization when the processor was loading the program from non-volatile memory, and that was a fun one.

I was using a Dell latitude D830. I sometimes use a wireless mouse, but due to space considerations, I didn't use it yesterday. The wireless usb adapter is in one of the usb ports, but the mouse was turned off.


Note about variable names: The OEM and plant personnel were extremely irritated when a certain competing contractor downloaded the software without permission, and then proceeded to have their own engineers look over the equipment modifications, and take numerous photographs of the said modifications. All of this was done without the plant's or OEM's permission... we (plant/OEM/and myself) found great humor in the creative use of variable names - which also serve as a personal message to the culprits.
[/FONT]

 

[jdbrandt]

Wow.

Q: How do people keep track of stuff in a chaotic environment?
A: They don't.

This isn't even worse than "The man with two watches."

[A man with one watch always knows what time it is...a man with two is never quite sure.}

I would have thought that the proverbial "drawer full of floppies, all labeled <<LATEST VERSION>>" were long gone.

I guess not.

 

[daba]

despite your lengthy response, which will take several passes to understand, I can categorically state that a rung shown on your programming PC is not in the controller. Rungs that exist in the controller are shown by the software with an UPPER-CASE letter to the left, and by the way the editing interface works, it is impossible to see an upper-case E.

I recommend that you open your offline file (the latest, with up to date documentation), and UPLOAD what is in the controller.

That upload may well show rungs that are "pending edits", with I R and D rung markers, but I would be absolutely astounded if the upload included any rungs with lower-case letters to the left. The programming software does not allow this.

happy xmas

 

[JeffKiper]

We had some crazy power spikes on a L30 ver. 10. The customer had another SI working on the system and they started a case with RA. the official RA reaponce was that couldn't happen it must have been another programmer. So everyone's heads turned to me. I assured them I didn't do anything and couldn't if I wanted to because I didn't have v10.

This sparked my interest so I asked if I could work with them to solve the issue. We never solvrs the issue. We added a transformer befor a new UPS and another 24 VDC PS just for the plc power. The problem never returned after the nasty power was addressed.

 

[Russ]

We had some crazy power spikes on a L30 ver. 10. The customer had another SI working on the system and they started a case with RA. the official RA reaponce was that couldn't happen it must have been another programmer. So everyone's heads turned to me. I assured them I didn't do anything and couldn't if I wanted to because I didn't have v10.

This sparked my interest so I asked if I could work with them to solve the issue. We never solvrs the issue. We added a transformer befor a new UPS and another 24 VDC PS just for the plc power. The problem never returned after the nasty power was addressed.

This plant is notorious for having dirty power. When an OEM sent them a standard ups for a slc 5/05, the ups would actually reset the processor every time the plant would start up their 2nd compressor.

Without the ups, their 120vac would drop to 95VAC, but with the ups it would spike over 150VAC, and then drop to around 60VAC (which caused the reset).

I found a good voltage regulating ups, and all of the problems went away.

This issue stumps me. I've checked all of the most recent offline files, and none of them showed the rung in edit mode.

I can see daba's point about a wireless keyboard/mouse, or even a fat finger error..... however.. even if an invisible keyboard expanded the main task, and somehow selected the 14th branch, then double-clicked on it, then scrolled down twenty-two rungs, double-clicked again, and then deleted eight arguments from seven different branches..... I would have expected, at the very least, to see the ladder logic displayed, with the lower cased e's, on the right side of the screen.

But there was nothing....

I'm currently leaning towards it being a power issue, at this time, but I'm not 100% certain.

One interesting point to consider. Every argument that was deleted from the rung contained an element from the same array. The only two arguments that remained contained elements from a different array.

It is an interesting coincidence.

 

[JeffKiper]

Is the tag name in this array something odd?
Is that array still present when this fault happens?

Those are the only 2 questions I can really think of off the top of my head.

 

[Russ]

Is the tag name in this array something odd?
Is that array still present when this fault happens?

Those are the only 2 questions I can really think of off the top of my head.

I've only seen this fault occur one time, and the array was still present. The array is also in all of the offline versions I checked. It was created roughly 18 months ago.

I'm certain I flahed the non-volatile memory back in August. The problem with flashing the memory is that it requires shutting down the entire plant. Which requires hours of downtime at the cost of thousands of dollars. It's why the programming changes done on 8/13 weren't stored to the non-volatile - none of the changes were critical to plant operation at this time, and I didn't feel they were worth the downtime.

Next week the plant will be shutting down, and I'll take advantage of that time to store the latest version of the program to the non-volatile. When I do this I'll make sure the non-volatile is loaded every time power is cycled. It's one of the few things I can think of to try and minimize the possibility of this happening again.

Adding a voltage-regulating ups would also be nice.


One can dream.