PLC/program maintenance

[JesperMP]

All CPUs are to be password protected.
Passwords are not generic. Must be unique per CPU.
Only trusted persons can know the passwords.
Before any program change, verify that the backup matches the program on the CPU.
Backup and document (*) every program change to a server.

The bigger PLC vendors have software solutions to handle the passwords, users, backups etc. across an entire plant.

Discrepancies between backups and CPU programs are to be investigated, documented, and ultimately end with consequences for persons that cannot be trusted.

Same goes for wiring changes, but here it is more difficult to enforce a strict regime. You cannot press a button and make the schematics test itself against the actual wiring.
That it is not easy is no excuse though.

*: In my opinion, the documentation part does not need to be very detailed. Just something like 'fix implemented due to program locking up in step xx..' is enough. To my experience no-one goes back and studies a minutely detailed log after the fix is implemented.

 

[JesperMP]

Do a backup before any changes start [..]

I disagree with that. Use the previous backup and verify it against the program in the CPU.
If the offline/online programs matches, then proceed with making the necessary changes and when done create a new backup.
If the offline/online programs do NOT match, then something has gone wrong and it must be investigated in detail. Only after knowing the full story can one decide what to do, i.e. keep or discard the online differences.

Regarding saving the online actual values, then that may be different from machine to machine.
On some machines you want to keep the initial values as that, initial values.
If the machine is properly programmed, there will be a function to save important machine parameters to a file that can be restored later.
If there is no such function, well then maybe there is no other way than to use a program backup to save the machine parameters.

 

[BobB]

Oh - this is interesting! It depends!


At BHP (steel manufacturing) online program changes were barred. I was kicked offsite because I did an online change on a simple generator backup system! I can certainly understand why they are very cautious with online changes on the manufacturing plant - tip a pot of molten steel over and the fire would destroy the whole place!


I never backup a program but always compare what is on the laptop and what is in the PLC. Just save the latest version - probably a backup I guess? The software calls up any differences that can then be checked.


It is funny - I do a lot of software for Canberra Deep Space Communications Centre - it is all online - cannot turn the PLC off for any reason at all. The tracking on the radio telescopes is tied in with the radio telescope in the US and another in Madrid. The power goes out - they loose a track that cannot be recovered.


For example when the satellite went past Pluto all the early pics came into Canberra. Lost - gone - how many millions of dollars would be wasted?


Canberra is the only radio telescope still tracking the original Voyager - yes - it is still going outside our solar system now.


They trust me totally - I write new software inside the PLC and do not connect it to the outside world until it has been tested and verified with their engineer - then turn on the outputs.


Not many would trust you to do that bay the way - the PLC controls the mains power - two generator power stations - a rotary converter - the site wide UPS - all the high and low voltage switchboards - can get a bit stressful at times. Always done in a rush as there are never enough windows where the telescopes are all offline or doing their own thing.


Interestingly very few PLCs I have programmed are password protected - even the NASA one in Canberra. If they have an issue the engineer rings me and we go through the program together and sort it out - he did a lot of PLC programming in a past life.


Amazon data centres are a different thing! Bloody hate them!

 

[JesperMP]

BobB.

Do I understand you correct that you remote in to a PLC to make online changes, and there is no password protection on the PLC ?
This sounds to me as a security breach waiting to happen.
Security was not a big thing until a few years ago. Now it is a HUGE thing. It is not only individual hackers, it is governments that actively trawl potential wictims, maybe not doing anything nefarious to begin with, but preparing to do so at the worst moment.

You probably have to login to the VPN to get access, but that is just the 1st layer of protection. All PLCs and HMIs must be protected.

 

[BobB]

Jesper MP - no remote log in - onsite only for this installation. They will not allow remote log in at all - the PLC is sitting on a remote I/O network and nothing else. If there is work to be done I have a 4 hour drive to go and do it.

 

[JesperMP]

Even a completely offline site may be vulnerable.
Hackers are very inventive. They can install a package on a device that then goes into action when the device goes online locally on the site, despite being offline to the internet.
Just saying that password protection should not be ignored on important stuff, or expensive stuff.

 

[TurpoUrpo]

What ever the way, make sure of having version control of some sort in place.

 

[Phil Buchanan]

FTAssetCentre or any similar revision control and disaster recovery software is an absolute must in any plant of significant size. This includes mandatory descriptive commenting and comment auditing. If you can get process engineering on-board, it makes MOC a lot simpler (not replacing MOC though) because if odd behavior is observed you can easily see exactly what changed, negating the need for taking screenshots of every rung you edited or saving the PLC with a new version number every time you make a tiny change.

This. Unless you have a bunch of guys with lots of time on their hands and need something to do you should not be wasting time doing backups and audits manually. Something like an FT Asset Centre will pay for itself in short order in even a medium-sized facility.

 

[Phil Buchanan]

Even a completely offline site may be vulnerable.
Hackers are very inventive. They can install a package on a device that then goes into action when the device goes online locally on the site, despite being offline to the internet.
Just saying that password protection should not be ignored on important stuff or expensive stuff.

Absolutely any disk to disk backup solution suffers from this problem. Disk to Disk backup and remote backup and replicate is good and it's fast but can be risky especially with malware and ****** lockers, etc.

That is why we employ tape backup onsite and offsite also that way there is a totally disconnected backup we can use to get back to a before the infection state.

This was made easier this year with the release of LTO 9 tape that is 18 TB standard and 45TB compressed and at that size will handle the entire backup on a single tape for most organizations which makes it easy to manage.

The tape drives run about 7K so we have been using them a lot recently for OT network backusp also.