possible none programmed change to program

Sometimes knowing too much can be a bad thing.
Ever since I read here about a month ago that it is possible for a program to change state through spikes or interference I cant get the possibility out of my mind.

Has anyone ever actually known a ladder program change but still work in an altered state without anyone changing the program?

Because now you have me thinking it might be happening to me.

A while back I did a concrete batching plant and odd things keep happening that I cant understand – such as too little water being added and alarms being triggered with no alarm etc etc.

This control room is supplied by a diesel generator, not the nice modern type in a box all running silent and everything but a very old ex-hospital generator.
A big tractor engine with old dials and knobs etc. and now I can’t help but think that the supply is a bit dodgy. (I have done no checks apart from a digital meter)

The short water problem for instance is controlled by a metering pump (1 ltr per pulse) and the batching cannot continue through the cycle until the set amount is reached – but yet I am told that sometimes the water is short by a large amount.
The metering pump is right at the point of going into the mixer – so leaks or diversion of water isn’t to blame.

I have reinstalled the program (with a complete memory wipe first) and I would never have done this before, had I not read here what I read.
Everything is perfect for a week or two then I get a call that odd things are happening again – any ideas or experience of ‘spike changes’

Makes the hair stand up on the back of my neck!

Goody,
I personally have not ever experienced anything like you describe. But is it possible? I am forced by experience to say that anything that can happen, probably will. Hey, that sounds familiar. I had a program/processor go completely berserk once. It was a new install and on start-up, it just did so many strange things that I shook my head in disbelief. I changed the processor, reloaded programs till my cable was getting hot and still the same thing. Then, in a stroke genius, or was it luck I can never be sure, some long lost bit of data surfaced in my RAM. The thought of during construction could one of my highly paid electricians have created a ground loop? Dash the thought! But, with nothing better to go on. I checked and sure enough, there it was. An instrument cable grounded on both ends. Can you imagine that? I found it hard to believe since the same crew had already built at least two of these systems and installed them.

Now, what this has to do with your troubles you are asking? I haven't a clue. But it is good to know that we all experience weird things in this business and sometime or another, the real culprit comes to light. I think your situation will have that resolution as well. Or at least I hope it does.

Just like randylud, I'm never going to say never, but I'd suspect that you're getting false input signals, probably due to electrical noise.

In the case of your metering pump, you keep pumping until you've accumulated a number of pulses equal to the preset number of liters you need. Electrical noise will show up as extra pulses, so you'll reach the preset too soon, a short fill.

My experience has been that anything that can scramble a PLC's memory also renders the program non-runnable, but electrical noise can make discrete input points appear to be on briefly. It can also cause analog signals to fluctuate wildly.

Also consider this. The chances of EMI altering the program, but not corrupting it enough to render it non-runnable are pretty slim. How much slimmer are the chances of a second random spike putting it back the way it was prior to the first incident?

I would have to conclude that your PLC is doing exactly what it is told to do and that your phantom gremlin is showing up on an input signal. You could try wiggling wires while the machine is running to see if you can generate the problem.

Alternatively, you could add a timer to the program and keep track of the minimum, maximum, and average time between pulses from the metering pump. Record the values for several successful batches. The next time you get a short fill, compare the values from that batch to what you recorded during a successful run.

Triple check the frequency

Goody I think its possible I know what is happening. Many years ago I worked for an amusement company (a carnival) with kiddie rides, bumper cars, double ferris wheel etc, we also used Generators. We had a ride called the Sea Dragon that looked like a Pirates boat that swings back and forth. This ride was made here in the US and used a very early model PLC (cant remember brand now), this was in late 80's and ride was 5 to 7 years old by then. At that time I didnt know what a PLC was.

Anyway, the ride used a hydraulic drive system to raise and drive 2 wheels to propel the boat in a pendulum fashion. I wasnt present when the problem started nor was I at that time in charge of the electrical dept, I returned and was told we had to change out the PLC. I did just that but when we started the ride it did the same thing, the drive wheels were raising and hitting the boat at the wrong time...ie it would start its pendulum motion but then act like it was being braked then swing then brake.

Note, this was my very first encounter with a machine using a PLC. I stayed up all night, read the manual, read about the plc, studied the hydraulics. Something I read was nagging me and I couldnt understand what it was, I experimented with things. Made alot of noise in the middle of the night.

In the process of experimentation I disconnected the power for the plc and connected it to "city" power...THE SEA DRAGON WORKED!!!!

WHY? At that time I didnt have a clue, so I studied the situation some more. When I was in charge of the Generators I always parallel them but the MASTER electrician that had the position at that time didn't. So I hooked up some paralleling lights to see how close the gensets were to being in sync...the 600KW genset the Sea Dragon was on was no where near the others...verified it with a Fluke 87 meter that has frequency on it. That genset was running close to 62 hertz...this doesnt hurt much EXCEPT where a PLC is involved.

If this genset is being started daily or having to be changed over at times then verify the process used and that it has an accurate cycle/frequency meter. The grief a half cycle can cause will amaze you.

The situations described above are exactly why I refuse to use a PLC as a "switch-handler".

Before there were PLC's things were "hard-wired" (remember that?). In many instances the design of the control was such that if a signal came ON or went OFF it must be TRUE! The affected circuit simply reacted - whether it was reasonable (logically so, in terms of the process) or not.

I program in terms of the process - what's happening, what should happen next, who should I listen to and who should I ignore.

Certainly, this does not eliminate all potential problems. But, it will most certainly minimize any detrimental effects. If any particular thing happens when it "shouldn't" be happening, then something is wrong!

"Shut'er Down Clancy! She's pumpin' mud!"

The program should run as if you decided to "BE THE COMPUTER"!

The unexpected can happen

To be fair I think for anyone to expect that the incoming cycles from a power source not be accurate is unlikely. That may not be the situation but if it is HOW would a computer of any kind know its timing base had changed because the incoming power frequency had changed?

It is impossible that noise or such things can alter the program memory so that the program starts to behave differently. The reason is that all the PLC brands (at least the ones I know of) have checksum test of the program memory. If the checksum fails, the CPU will stop.

What is possible (in theory at least) is that the data memory may be altered. There is no checksum of this (because it changes when the program executes). But I still find it unlikely to happen.

A possible explanation might be that the i/o is be affected by noise, as previously suggested. And there might many other causes.

The best way to locate the error is this: Investigate the PLC code so that you understand how it should work. Then go online and track the sequences according to your understanding of the PLC code. With determination you should be able to locate the cause of the error - wether inside or outside of the PLC.

The wacky world of PLC's

I have to agree with Jesper. After my tale of woe and reading the other responses, I realized that I had not added this part of the story. All the while that sneaky ground loop was making my life miserable, the PLC processor refused to operate that junk. The fault light did not go off till I corrected the problem. There were strange and even not allowed things that appeared in the program file and as the major players decided long ago, to operate a system on that kind of junk would certainly lead to disaster. I also agree with Terry on program strategy. To think like the process, the PLC processor and also like the worst and most inexperienced operator that can come down the line is worth the effort. In my early days, I was guilty of saying "no one would ever do that!" Well, the truth is they will and often do. Either by ignorance or accident or just plain mischief. The hardest part of any programming task is to figure out what to let your system do. I don't mean the obvious, I mean those other things that "no one" would ever do. Your program has to respond to the what if they do scenario.

Hi Goody...On the mobil Matting systems that I am building, which will be running off of rented 480vac gensets, we have found that installing a common, low cost, UPS on the control power line seems to eliminate the problems of miss counts and errors that you are describing...on the pulse misscount, can you change the pluse length, or maybe try adding an input timer/filter...Terry has described some of these. How about putting the pluse signal wire in a shielded cable? Good to hear from you again...now we know why you havn't been on here in a while....Work!!
beerchug
David

I certainly have to agree with the above comments. JesperMP is right that the program should not be able to change without the cpu noticing that the parity bits no longer match. It's not impossible that it could happen, just extremely long odds. Your chance of winning the lottery is better.

My first thought would be noise. I probably deal with a half dozen calls a day with noise related issues. Ground loops, coils with no surge suppression, floating neutrals, unshielded control loops or communications wire, etc. DC coils with no surge suppression are probably the top call for noise issues. A scope will help determine if this is the case or not.

goody
i face a problem like this. A few years ago i retrofit a packing machine using plc and leave working very well a few monthes later i receive a call telling me that the machine is not working at all.when i go to see the problem i found that the error led of the plc flashing indicating that there is a problem in the program.when i access the program i found that the "end instruction" of the program is missing but whole the program is ok.
when i ask about what happened they told me that the main power (city)
is cutted off and suddenly comes back in a very short time.for me this was unbelivable but there were no other reasons for that specially that they haven't access to my program because it was password protected.
the fact is the program is altered but the plc does'nt work and indicates error.
but in your case i agree with gurus that it may be noise or some thing like that .
thanks

I also think that noise is key factor of above mentioned problem. But its real science to deal with it. One manual is located at http://www.ab.com/manuals/gmc/GMC-RM001A-EN-P-JUL01.pdf
Also UPS is highly recomended.

Thanks chaps once again for your informed replies, I read them all with great interest; some I liked and gave great credence to, some, or at least one I thought was a little condescending.
(don’t all look at each other with a ‘does he mean me’ look on your face)

The original question centred on ‘had anybody actually experienced a corruption of a program that still ran’ I did think it was a long shot but as far as I can recall, I have never done a plc job before that was powered solely by a generator.

I had actually suggested to them, before that we install, a UPS, one of the reasons being that nobody watches or checks the generator apart from once a day during shift change. It cuts out regularly through running out of fuel or oil pressure.

I might have mentioned this customer before – I had difficulties with them from the start. Its political, they are an international company and once in a while they rotate management from site to site.
Each new boss has his favourite contractors from whence he came and everyone at this new manager’s site don’t do the job as good as the people he knows.
There are people waiting in the wings ready to ridicule and step in and save the day.

Back to the problem;
I might add that these faults and oddities never happen while I am there. They tell me what has happened but cannot replicate them. I am sure I don’t have any ground loop or wiring problems and I am as sure as I can be that the electrical side is sound.

My thoughts have always been centred on operator error or even operator malice.
The reason I mentioned the ‘adding water’ problem was because it is the most dramatic fault as they end up with a conveyor full of dry unusable concrete mix.

The conveyors are the life blood of this factory. No matter what happens – good concrete must be moving down these belts at all times or heads will roll. So, in the event of a dry mix, the finger must be pointed at someone.
We ‘the programmers’ are sitting ducks. For one, we designed the system and for two, we are not there.

My suspicions are that the operator (by intent or ignorance) sets the water too low then realising his mistake denies all.

Furthermore to the water adding system, they have 2 choices of how to add water.
1, set the litres to be added manually and adjust as necessary.
Or 2, use a system called an Hydronics unit.

The hydronics unit is a clever piece of kit that has a moisture sensor fitted to the mixer. They set a moisture level and the hydronics unit adds water and mixes until it reaches the required level.
However, it takes much longer to make a batch using this method as the hydronics mixes, then adds some water, then mixes again adding a bit more water until it is happy.

The water to be added can vary considerably, virtually from batch to batch. The ingredients to be mixed (aggregates) are outside in a massive pile. If it is a hot day a lot of water is needed, if it is raining then less water.

A batch can vary between no water to 160 litres – hence the hydronics unit. But it is slow. So they mainly use operator setting of the water – these are experienced men and can tell by feel if it is right or not.
But have you ever tried telling management that it is their operators that are messing it up when the operators have already beaten you to it by telling them that it is your system that is not right.

Of course, I have contemplated that I may be getting spurious counts from the metering pump but it is wired correctly and sheilded. I have put timers in to check the frequency of counts and they are always ok while I am there.

But isnt it good to share our experiences out there in the unforgiving world of automation

Goody, how about doing a bit of data logging to prove the fault?

Shouldn't be that hard to do, unless there is something I'm missing here. The only way I know to mix concrete or cement is by batching. At the end of the batch log the state of the water selector (Hydronics or Operator) and the reading from the hydronics sensor. If they really want to know where the problem originates, this will tell.

BTW, If I was the one who was condescending, or if this seems too elementary, I apologize. Just a thought.

Sorry about the condecending remark, it was Terry that nettled me with the 'why I refuse to use a PLC as a "switch-handler".' remark.

I am not going to make an issue over semantics so strike the paragraph from the records