It Stopped Running Because The 20 Year Old PLC Program Changed

Wow.

I apologize for distracting us from the main story of the thread, which seems to be that the code could not be fixed in the original SLC, because ... communications (am I getting warm)?

@I_Automation: am I understanding this right that the customer-supplied story suggests between-shift changes in the EEPROM (we know the code is coming from the EEPROM because it is powered on each day and S:5/8 is 1)? And as @Ken Roach mentioned, wouldn't there be a checksum or CRC tested when the code is loaded from the EEPROM, to catch such an event? Occam's razor and all that.

Is there a timestamp embedded in the data on the EEPROM?

There is so much more to this field than the easy part i.e. coding.

I am guessing somehow that the single bit that got flipped didn't trigger the checksum. I don't see how a random bit could flip the same time the perfectly needed to flip bit in the checksum flipped too.
Or since it wasn't a programmer change the checksum doesn't apply.

I never checked for a timestamp in the EEPROM, just downloaded the corrected program to it.

I_Automation said:

All palm buttons for 2 hand controls are 1N.O./1N.C. and both have to work each cycle of the buttons, and 2 hand controls the N.O. on both buttons have to come on within a set time (usually 1/4 to 1/3 of a second) Also, as I said the palm buttons are not wired to the PLC at all, the 2 hand control relay logic turns on a Running CR and the PLC gets a signal from that.




Also this press doesn't have a Pilz or Banner 2 hand control safety relay, just standard relays like have been used in 2 hand controls since the 1940's (maybe earlier)


This customer is good in that they have an archive on their network of every PLC, HMI and robot program on their property - even keeping machine files they have sold or scrapped out. Plus they demand a copy of any update, even if it's just an online save of current data.

Click to expand...

when a shop is that involved in their PLC hardware/software/programs.... anything is on the table.

For example, one time at a previous employer, I was looking into an issue with a machine. It was just failing a vision check once in a while, that was it... so while working at the issue, the machine suddenly stops and doesn't respond to anything. No idea what it is, I wasn't in the machine, it was just running and then bam, no worky. Some things were responding, I looked at the plc in the cabinet and some outputs were on, so of course I assume it's something with the process.

Long story short, I eventually get the laptop and connect to it and find that the program stored in the ML1200 is not the same as the one they had on file. They used their own internal versioning check out system, so the current program in the machine would be stored and locked for checkout. If someone was working on it, they would have to check out the program to get it, then return it for it to be available to someone else to check out. The system showed that there was no one who checked out the program in over a year.

But... The processor name said it was a singlepoint spotwelder, and this was not that machine. I put the program from the vault back into the processor and everything worked again... What happened was, one of the new engineers was in another area, connected to the network and was trying to download the program he was working on, to the machine he had just given a random IP for the network without checking. He just so happened to swap programs in another machine on the network (about 100 on the network for processors alone) and didn't realize what he did.






and for buttons... I've had multiple cases where someone used the wrong contact blocks on a palm button, then another person comes behind them the next shift, changes the program to work because they don't want to go through the hassle of getting their tools out, or doing a lockout/etc...

lots of options here, just seems very unlikely that it swapped on it's own.

I hate Nobody!!!!!
And his Brother Somebody

And their cousins, Anybody and Everybody.

Besides It always the last one out the door who always, always get the blame

Just reset the ID 10 T circuit and move on

That's it. I'm done.

John Morris said:

I hate Nobody!!!!!
And his Brother Somebody

And their cousins, Anybody and Everybody.

Click to expand...

I worked with a guy that said he was going to change his name to Nobody.
Then he would only respond if some paged "Nobody to xxxxxxxx" on the PA.

I was just going to tell the boss I was only going to respond when someone paged me by my original name from my home planet, which like Mr Spock is unpronounceable by the human mouth. (But always wondered why pure Vulcans, and even his father, only called him Spock)

Ken Roach said:

Possibility 1: A technician or contractor made changes to the controller and saved them in the EEPROM, and the customer is not telling you the whole story.

Possibility 2: Electromagnetic impulses or high-energy particles struck the EEPROM in precisely the correct places to change two related instructions to their opposite instruction codes and corrupt the memory checksum to the correct value to match.

The universe is big enough for Possibility 2, but I know which one I'd bet the rent on.

Click to expand...

This gets my vote for PLCS.net Top Reply of 2022 !!!

Ken Roach said:

Possibility 1: A technician or contractor made changes to the controller and saved them in the EEPROM, and the customer is not telling you the whole story.

Possibility 2: Electromagnetic impulses or high-energy particles struck the EEPROM in precisely the correct places to change two related instructions to their opposite instruction codes and corrupt the memory checksum to the correct value to match.

The universe is big enough for Possibility 2, but I know which one I'd bet the rent on.

Click to expand...

I am on possibility 2. I blame global warming and the last full moon.

I'm with that as well, as I posted, I had exactly that, there was no possibility of anyone connecting to the PLC, it was in Ram not EEProm, it happened 3 times, however, only the first time did it not fail on checksum (this is what really baffled me), the other two times it did fall over on checksum. Also had one that was in EEProm, all these were switched off every change of product & sent to hygiene for cleaning so perhaps 2 or 3 times a day, incidently these were portable machines on wheels (well I say portable) made in the US, the panel was part of the machine construction, this was Stainless with a thickness a tiger tank would be proud of, the main isolator would throw you on the ground when the spring was overcome & was as big as our 300A breakers on our distribution panels all this for a panel that drew less than 30A, they sure build them to be bullet proof in the states.

parky said:

they sure build them to be bullet proof in the states.

Click to expand...

"If more is better, then too much is just right!"

I spent 10 years in stamping plants. I can smell this post. The bottom of my shoes are now slippery with die lubeand oil dry.

JeffKiper said:

...and oil dry.

Click to expand...

Your stamping plant must have had a nice budget if they bought oil dry.


The one I worked at just told the operators to mop it up.