Programmers who go off script

theColonel26 · Mar 9, 2021

VAN said:
To add to this, I don't think anyone should have to look in the code.

If you can't look at an HMI and see exactly why something wont start (interlocks). If it starts and then stops it better have an alarm associated with why it stopped (unless someone pressed the stop button).

This is actually true, All HMIs should have a IO status page, and everything that could possibly go wrong because of external circumstances (Physical Machine conditions) should have an alarm with a description of the problem.

If it doesn't your program is not complete.

So really Controls Techs should never be in the code. If there is something that is not covered by alarms and IO status then it will be above their head anyways.

Paullys50 · Mar 9, 2021

theColonel26 said:
This is actually true, All HMIs should have a IO status page, and everything that could possibly go wrong because of external circumstances (Physical Machine conditions) should have an alarm with a description of the problem.

If it doesn't your program is not complete.

Must be nice to live in a world where the customer will pay for the added expense of all of those sensors and time required to identify all the things the production floor could possibly break in a system.

Case in point, recent project I commissioned had constant VFD faults occurring. The drive wasn't the problem, the motor wasn't the problem. A pair human eyes could easily observe operators dumping enough raw material into a manual addition bin to make a thick cement-like paste which quickly clogged the piping and pump. "VFD keeps faulting out, we don't know why". Should I recommend load cells? And automated feeder? Infer a clogged system based on some pump runtime vs active fault vs no change in level? Install a flow meter to monitor? Create some anti-clog algorithm with the limited control points available? What about when maintenance swaps airlines on valves (which the customer didn't pay for feedback) and now dead heads a pump because there isn't a way to detect this situation?

I_Automation · Mar 9, 2021

theColonel26 said:
This is actually true, All HMIs should have a IO status page, and everything that could possibly go wrong because of external circumstances (Physical Machine conditions) should have an alarm with a description of the problem.

If it doesn't your program is not complete.

I am working on a line a customer bought used that the PLC status screen on the HMI only shows the status of the card (comm's OK, faulted, etc.)

I have clicked on that I can't say how many times to see if an IO was on or not and there isn't one every time.

I would love to meet that programmer in a shop I have to wear steel-toe shoes.

janner_10 · Mar 9, 2021

Paullys50 said:
Must be nice to live in a world where the customer will pay for the added expense of all of those sensors and time required to identify all the things the production floor could possibly break in a system.

Case in point, recent project I commissioned had constant VFD faults occurring. The drive wasn't the problem, the motor wasn't the problem. A pair human eyes could easily observe operators dumping enough raw material into a manual addition bin to make a thick cement-like paste which quickly clogged the piping and pump. "VFD keeps faulting out, we don't know why". Should I recommend load cells? And automated feeder? Infer a clogged system based on some pump runtime vs active fault vs no change in level? Install a flow meter to monitor? Create some anti-clog algorithm with the limited control points available? What about when maintenance swaps airlines on valves (which the customer didn't pay for feedback) and now dead heads a pump because there isn't a way to detect this situation?

When dealing with stupidity on this level, access to your code and a laptop is the last thing these guys need.

theColonel26 · Mar 10, 2021

Paullys50 said:
Must be nice to live in a world where the customer will pay for the added expense of all of those sensors and time required to identify all the things the production floor could possibly break in a system.

Case in point, recent project I commissioned had constant VFD faults occurring. The drive wasn't the problem, the motor wasn't the problem. A pair human eyes could easily observe operators dumping enough raw material into a manual addition bin to make a thick cement-like paste which quickly clogged the piping and pump. "VFD keeps faulting out, we don't know why". Should I recommend load cells? And automated feeder? Infer a clogged system based on some pump runtime vs active fault vs no change in level? Install a flow meter to monitor? Create some anti-clog algorithm with the limited control points available? What about when maintenance swaps airlines on valves (which the customer didn't pay for feedback) and now dead heads a pump because there isn't a way to detect this situation?

Looking at the code would not help in any of these situations so, I do not see your point.

I guess I phrased that last part a little to broadly. My point was if there is a way to detect an issue based on the inputs available then there should be an alarm in the PLC. If you do that then there is not reason for a tech to ever look at the PLC code.

As to your specific example, I would point out that it would be a lot easier to figure out the problem if you pull an error code off the VFD and display on the HMI, that the Drive/Motor is over torqued. Regardless looking at the PLC code itself won't help.

janner_10 said:
When dealing with stupidity on this level, access to your code and a laptop is the last thing these guys need.

dschmiddr · Mar 10, 2021

As the programmer who goes off scipt all the time...

Yeah. I said it. I will attept to go full ST or SFC the moment anybody allows me to touch a PLC. Drives everyone nuts. They show me their sequencer and thier excel spreadsheets that are going to get lost and then I show the same operation with an SFC and the conversation devolves to 'Nobody knows that stuff.' Needless to say I stay busy on the SCADA side.

Saffa · Mar 10, 2021

theColonel26 said:
This is actually true, All HMIs should have a IO status page, and everything that could possibly go wrong because of external circumstances (Physical Machine conditions) should have an alarm with a description of the problem.

/QUOTE]

This might work for whatever you build, but when your process plant has 20,000 I/O that's a lot of screens...

To be fair, any good SCADA system should point the maintenence team in the right direction, without needing to break out a laptop. But... i have had plenty of need to hop online myself and figure out what is causing some weird dynamic we haven't experienced before, even on plants that have been running for 25+ years.

I do think that the laptop should not be a first resort to all troubleshooting, but making sweeping statements that the controls engineer should be able to foresee all possible scenarios and have an alarm for it... that just tells me you haven't yet come across any of the incredibly interesting and complex processes that PLCs get used in.

bill4807 · Mar 10, 2021

Paullys50 said:
Must be nice to live in a world where the customer will pay for the added expense of all of those sensors and time required to identify all the things the production floor could possibly break in a system.

Case in point, recent project I commissioned had constant VFD faults occurring. The drive wasn't the problem, the motor wasn't the problem. A pair human eyes could easily observe operators dumping enough raw material into a manual addition bin to make a thick cement-like paste which quickly clogged the piping and pump. "VFD keeps faulting out, we don't know why". Should I recommend load cells? And automated feeder? Infer a clogged system based on some pump runtime vs active fault vs no change in level? Install a flow meter to monitor? Create some anti-clog algorithm with the limited control points available? What about when maintenance swaps airlines on valves (which the customer didn't pay for feedback) and now dead heads a pump because there isn't a way to detect this situation?

Lol.
This is my life on 75% of projects. You know what happens as they find all these issues after commissioning, we are making several trips over the next 3 months adding all those features in. Juat because the manager at the time had to stay on budget. Now they are losing triple that in downtime.
I really shouldn't care after all its not my money. But what gets under my skin is im the one who has to make all the trips to integrate the fixes into panels that weren't sized to accomodate all these added features.
Again i shouldn't care its the customers machine they made the decisions.
But i can't let it go, it could have been perfect to begin with..

I_Automation · Mar 11, 2021

bill4807 said:
Lol.
This is my life on 75% of projects. You know what happens as they find all these issues after commissioning, we are making several trips over the next 3 months adding all those features in. Juat because the manager at the time had to stay on budget. Now they are losing triple that in downtime.

When I build panels and write the programs that I know features will be added later, I put them in the program and reserve outputs and CR's as needed, just label them Future Use.

That way when they have to be added all I have to do is enable the rungs or call the (odd named) subroutine and maybe add a couple things to the HMI (that may already be there but have a visibility bit that isn't on until the above rung(s) or subroutine is enabled.)

Adding it when doing the original program really doesn't take that much longer. Adding it later will take time and thought on how to integrate it.

JesperMP · Mar 11, 2021

Paullys50 said:
Case in point, recent project I commissioned had constant VFD faults occurring. The drive wasn't the problem, the motor wasn't the problem. A pair human eyes could easily observe operators dumping enough raw material into a manual addition bin to make a thick cement-like paste which quickly clogged the piping and pump. "VFD keeps faulting out, we don't know why". Should I recommend load cells? And automated feeder? Infer a clogged system based on some pump runtime vs active fault vs no change in level? Install a flow meter to monitor? Create some anti-clog algorithm with the limited control points available?

Some workers are malicious. They will knowingly overload a system if it makes their life easier, even if it makes some other workers life worse.
In that case, the remedy is some disciplinary action !

If this was a support case for a customer installation I would ask for them to record a video of the situation. That would reveal what is happening.

If your system must be idiot-proof and/or malicious-proof, then it will take some effort.
The bin could be dimensioned so that they can only fill in as much material as the pump can transport.
Or you could add some hardware to dynamically limit the flow of material or detect that an overloading is happening. A max sensor in the bin does not cost much. If the max sensor is activated for too long, trigger an alarm that calls in a supervisor.

bill4807 · Mar 11, 2021

I_Automation said:
When I build panels and write the programs that I know features will be added later, I put them in the program and reserve outputs and CR's as needed, just label them Future Use.

That way when they have to be added all I have to do is enable the rungs or call the (odd named) subroutine and maybe add a couple things to the HMI (that may already be there but have a visibility bit that isn't on until the above rung(s) or subroutine is enabled.)

Adding it when doing the original program really doesn't take that much longer. Adding it later will take time and thought on how to integrate it.

I agree.
I do this as well with the program as much as i can beforehand. I was more refering to the parts, wiring and mechanical items.

busarider29 · Mar 17, 2021

Fourpaw said:
Plus he talks to himself all day.

cardosocea · Mar 17, 2021

bill4807 said:
Lol.
This is my life on 75% of projects. You know what happens as they find all these issues after commissioning, we are making several trips over the next 3 months adding all those features in. Juat because the manager at the time had to stay on budget. Now they are losing triple that in downtime.

It's the age old debate of OPEX versus CAPEX... It's easier to hide things in OPEX than in CAPEX, but failing at CAPEX can severely reduce the ability of getting more investment. I agree that it's a pain, but there are corporate failings that cause this behavior.

JesperMP said:
If this was a support case for a customer installation I would ask for them to record a video of the situation. That would reveal what is happening.

There's also audit trails and alarm logs to record what has been done by the operator. A company I worked for developed a product on purpose for the customer to be able to trace back what the operators did. Essentially it recorded every message sent between PLCs on the network and whenever you wanted to review what happened you could download it onto a trend viewer or actually display what the operator was seen and the actions they were doing live on the control screens as a playback.
Needless to say it wasn't advertised much and left operators white as chalk when they realised such a thing existed.

Iner · Mar 17, 2021

theColonel26 said:
This is actually true, All HMIs should have a IO status page, and everything that could possibly go wrong because of external circumstances (Physical Machine conditions) should have an alarm with a description of the problem.

If it doesn't your program is not complete.

So really Controls Techs should never be in the code. If there is something that is not covered by alarms and IO status then it will be above their head anyways.

Control job tech in factories is partially to get into code to troubleshoot faster. It requires extremely good knowledge of a machine to troubleshoot it on sight and technicians can't have perfect knowledge of all their machines.

A good HMI will give great help to troubleshoot but let's be honest, it's nearly impossible to report all the possible cases unless you are only working on very basic machines or on standardized process with 200 pumps and 400 valves.

I have seen a few small factories without people able to "get into code" by lack of personnal or software/cable and it wasn't pretty. They have to call up external companies to help them restart their production, generally with a big delay.

I wouldn't advise any company operating a production plant to not have technicians able to troubleshoot by reading the code. Last time I went to a company that had a machine down for 4-5 days days since 1st January, they were quite desperate. Their maintenance technician and maintenance chief had spent days trying to find a safety faults, replacing I/O cards, shutting down and restarting the cabinet, checking the sensors, ES and the cables....

I went, connected my computer and found out that their S7-1200 CPU acting only as a safety controller, and out of reach in the cabinet, was in STOP mode... It was set up in the configuration to restart in the previous mode , and there was no way to turn them in RUN outside of TIA portal with these CPU. The machine started 5 minutes later. They told me that they wanted hire someone and wanted to get the soft.

My point is that you can't expect process designerd to think and test their machines and production lines with all the conditions that will happen during all the years.

At some point, people in factories will find some weakness and will want to correct them, or even add some minor functionality like a sensor, and they should be able to do it themselves and rather easily, integrate them into the logic in a smooth way.

JesperMP · Mar 17, 2021

I'm with Colonel26 on this one.
When we are talking about custom designs for a specific customer, I am all for that the source code should be handed over. In such cases, I even give training course for the end-customer how to understand our programming framework. Then the local tech has a realistic chance of troubleshooting without causing further problems.
On the other hand, when we are talking about standardized OEM machines, the local tech should not have to look into the code to troubleshoot.

Iner said:
A good HMI will give great help to troubleshoot but let's be honest, it's nearly impossible to report all the possible cases

I think that you CAN actually setup alarms for every conceivable hardware error. Higher level errors, i.e. that the sequence waits for certain conditions should also be reported. For standard machines, the programmers may have spent years on gathering every possible error situation.

Iner said:
[..] unless you are only working on very basic machines or on standardized process with 200 pumps and 400 valves.

Are you saying that on a very complex machine, a 3rd party programmer should be able to jump right in and troubleshoot the code ? That makes no sense to me.

Iner said:
[..]
I went, connected my computer and found out that their S7-1200 CPU acting only as a safety controller, and out of reach in the cabinet, was in STOP mode... It was set up in the configuration to restart in the previous mode , and there was no way to turn them in RUN outside of TIA portal with these CPU.
[..]
My point is that you can't expect process designerd to think and test their machines and production lines with all the conditions that will happen during all the years.

That is a blatant design error. Never heard that the S7-1200 has such a setting, but I am at a loss why anyone would want to chose such a setting. Can the S7-1200 change mode via its webserver ? In that case it could be the solution.

Programmers who go off script

theColonel26

Lifetime Supporting Member

Paullys50

Lifetime Supporting Member

I_Automation

Lifetime Supporting Member

janner_10

Lifetime Supporting Member

theColonel26

Lifetime Supporting Member

dschmiddr

Member

Saffa

Member

bill4807

Member

I_Automation

Lifetime Supporting Member

JesperMP

Lifetime Supporting Member + Moderator

bill4807

Member

busarider29

Lifetime Supporting Member

cardosocea

Member

Iner

Member

JesperMP

Lifetime Supporting Member + Moderator

Similar Topics