Intentional "Bug" in System What to do.

Got a call tonight to visit a site urgently. They couldn't reach their usual Engineer. Production line was down. Lucky for them I had no alcohol consumed.

Arrived on site and went online to PLC. An integer register was being incremented each day and upon reaching 400 set a ROUTINE_INSPECTION bit which was used as a permissive for a lot of activations in logic.

What the hell? I've been asked to write a detailed report on what happened. Unforuntately for me I know the systems integrator company that commissioned this line.

Without seeing any functional designs specs etc I can only hope there was a reason for this. It appears nowhere on SCADA. Perhaps there is a genuine reason Preventive Maintenance etc but nothing obvious in code, comments etc. Theres only a runtime on SCADA so I can't dig into it.

What to do please? They have helped me in the past and I'm not out to hang someone.

That is strange. I once worked for a company which had a timed inhibit of this type but it was VERY WELL well documented. It depended on a specific inspection being done. If a bad condition existed it could destroy the machine. At that point they had not worked out a method for the PLC to directly detect the condition.
An inhibit like this with no supporting documentation is very suspicious.

What to do please?

Click to expand...

Simple. Write your report of the service call. Include why you were called in, what you found, and what you did to remedy the problem. That covers your rear end without pointing any fingers until you're sure that it truly was an intentional "feature". If they want a recommendation on what to do to prevent it from happening in the future, tell them it will require research on your part and thus a purchase order. That will give you time to consult with the original programmer.

From time to time we get asked on this site about programming in "time bombs" as insurance against a customer refusing to pay for services rendered. My advice to those posts is, if you're worried about customers not paying you, find a better class of customer. The time you spend writing a routine like that could be better spent marketing your services to customers who are willing to pay for value received.

Furthermore the consequences of failing to remove the time bomb from a system that has been paid for could be pretty drastic. About the best outcome you could hope for would be that you lose a customer. Worst case scenarios range from a costly lawsuit to criminal prosecution.

Just write your report as "Just the facts". Don't speculate about "Why?", just tell them "What, When, and Where".

Fatten up the report with technical detail. For example, describe the method you employed to find the problem and translate some of the offending ladder into readable English.

I would refrain from defending or criticizing the other integrator too much.

Perhaps it is a feature they use in other applications that was not correctly removed from this one? If it turns out to be a forgivable human error or an intentional time bomb, not being associated with them might be a good thing for you at this moment.

Also, make sure you did due diligence in searching for indirection and things like that to ensure this is truly a loose end and not some functional thing that has gone awry.

If you can grab screenshots, do so; it might be easier for somebody to recognize the data table or ladder logic in context.

If you had the chance, get revision numbers from the program and any timestamps or last-edits-performed information. Document where you got the offline program file from.

Otherwise I think you have the ethical principles well in hand: be factual and you don't have anything to worry about.

The only thing you need to cover is your own backside, so be sure to document when you were called, by whom, the purchase order number and date, and a brief summary of your direct involvement in the system (if any) prior to that call.

This is the sort of report you'll want to run by an editor, so that you can root out any speculative statements you might not realize you are making.

**Once got blamed for damaging a system I'd never seen: I had to provide a boarding pass to prove I wasn't in the country.