Downtime reduction

Is there are any PLC or HMI programming methods available to reduce downtime by rapid troubleshooting of any alarm in a minute or so? Any suggestion will be greatly appreciated.
Thank you.
Emil

A large piece of plc programming is diagnostics. Expressing to the operator or maintenance folk what is faulted and why when possible. The more information that can be provided the faster a machine or process can be repaired and resume normal operation.

For instance, a simple pneumatic cylinder with advance return sensors can easily have 7 failures I can think of off the top of my head and more if you measure and compare expected velocities.

PLC software is pretty ****ty in this regard. You got to do all the work yourself.

But usually this work starts at the mechanical and electrical side. You have to have a sensor for every movement of everything.

If you for instance have a pneumatic cylinder rotating something, you put the sensors on the thing it rotates and not the cylinder itself. That way you can detect both a mechanical problem, a pneumatic problem and an electrical problem.

Every circuit breaker, every contactor, every isolator switch etc needs a feedback signal so you know if it has failed and you can make an alarm for it.

On the HMI you build manual control for everything and show all the sensors that are involved in every function. So if you have a pneumatic cylinder and one sensor show an alarm, you want to be able to run it manually and you want to be able to see on screen the input signal from every sensor including air pressure, 24V power supply etc that is involved in that function.

So if it's a machine you have built alarms for everything and if you have an alarm on something you can run that something manually and see everything from the HMI screen you need to determine what the problem could be.

You also need to identify every sensor and show where it's placed so maintenance can find it quickly.

If you use sequences in your program you should show the state of every sequence as well as what it is waiting for at every step of the operation. So if it wait for some reason longer than expected you can show an alarm and show the operator why it stopped and exactly what it was waiting for.

If downtime is really not an option you use redundant control systems and you have to decide how far you want to go. This is very common in a lot of heavy industry for instance petrochemical industry. That industry commonly uses DCS systems which have more diagnostics and capability from the get go, compared to standard PLCs and HMIs, however same principles apply. You just have build more by yourself when working with PLCs.

The cost and complexity of what you are proposing would be enormous. That said, I have worked on some machines that included excellent troubleshooting guides for each fault, and they do indeed cut down on down time. To me, the most important tool for troubleshooting is a well trained tech. I don't remember who uses it as their signature here right off hand, but truer words have never been spoken. "Yes, training is expensive, but ignorance is where the real money is at"

Bubba.

I agree with Pete S approach, you can also consider a "Safety PLC" that has two CPUs. This way if one eeds to go down for a minor alarm, you can troubleshoot that issue, while the 2nd processor takes over the other operations of the machine. This allows you more time to troubleshoot one section of a large system/machine without interrupting the entire process. I know Siemens offers some very nice Safety PLCs, but are a little on the pricey side.

Still, is there any method known to smart programmers, which will simplify tracking of any alarm and detect which alarm starts first, second, and so on? Let's suppose you have 5k to 10k I/Os and you want to track them and make sure that they are real alarms and not alarm state after safety shut down. Is there any technique guys? Any general problem solving available?
Emil

"Source code is poetry!", don't you think?

emilsagal said:

Still, is there any method known to smart programmers, which will simplify tracking of any alarm and detect which alarm starts first, second, and so on? Let's suppose you have 5k to 10k I/Os and you want to track them and make sure that they are real alarms and not alarm state after safety shut down. Is there any technique guys? Any general problem solving available?
Emil

Click to expand...

First off, you need to inhibit alarms that are triggered by another event. So alarms that get raised because of power failure, safety shutdowns and what not, will actually not cause an alarm.

This can be done already at the PLC level if you want.

If you don't do this you get lots of false alams.

Next thing you need is a real alarm system for the operator. A HMI panel is usually to primitive for lots and lots of alarms but "SCADA" software that you run on a PC can handle more. DCS systems are the best, most capable though.

You need to see active alarms, acknowledged alarms and also be able to see alarm history. On larger systems alarms are often divided into different zones as well. You can often click on a alarm and see what has caused the alarm, at what time and also easily jump to the right process image where the object that caused the alarm is located.

Look at the image below and you'll see the some of his monitors showing lists of information. That is what typical alarm pages looks like. This is also what a typical classic control room looks like at a plant for each operator.

Is there any technique guys? Any general problem solving available?

Click to expand...

First-out...

http://www.plctalk.net/qanda/showthread.php?t=23974&highlight=first-out

http://www.plctalk.net/qanda/search.php?searchid=6597912

Is an operator making decision what to fix during alarm?

Is an operator making decision what to fix during alarm?
With a huge number of I/Os it is going to be very many alarms that are happening at the same time. How the operator can distinguish which is the main one? Here is another question: How long would it take for operator to analyze 3,000 I/Os, one I/O at the time? Is there any visual technique available these days?
Emil

Source code is poetry. It needs brain to write.

Alarms have to be prioritized, masking faults that are "downstream". For instance, if power (be it electrical or fluid) is lot to a section of mechanisms, the alarm logic would only indicate the power loss fault and mask out faults from every mechanism that relied on that power.

Pete.S. said:

PLC software is pretty ****ty in this regard. You got to do all the work yourself.

But usually this work starts at the mechanical and electrical side. You have to have a sensor for every movement of everything.

If you for instance have a pneumatic cylinder rotating something, you put the sensors on the thing it rotates and not the cylinder itself. That way you can detect both a mechanical problem, a pneumatic problem and an electrical problem.

Every circuit breaker, every contactor, every isolator switch etc needs a feedback signal so you know if it has failed and you can make an alarm for it.

On the HMI you build manual control for everything and show all the sensors that are involved in every function. So if you have a pneumatic cylinder and one sensor show an alarm, you want to be able to run it manually and you want to be able to see on screen the input signal from every sensor including air pressure, 24V power supply etc that is involved in that function.

So if it's a machine you have built alarms for everything and if you have an alarm on something you can run that something manually and see everything from the HMI screen you need to determine what the problem could be.

You also need to identify every sensor and show where it's placed so maintenance can find it quickly.

Click to expand...

We have in the past moved towards more sensors for easier troubleshooting. For machines operating in a "difficult" environment, we found we got more defects/errors and they were harder to track down. Quite the contrary of what we were aiming for. At some point, most of the errors you get are sensor defects (either the sensor itself or the cabling). Since then we have backed off, limiting sensors to what is critical for the process at hand. For our customers, most of the time less sensors is a more reliable machine. This depends very much on the environment and the process though.

You see a similar thing with cars in the last one or two decades. Cars with lots of electronic monitoring tend to have more defects. In the electronics, "ghost" warning signals while there is actually no defect. Some (mind you, not all) "cheap" lines or brands have proven to be significantly more reliable than upmarket models stuffed with electronics. If it is not in there, then it cannot break.

1. solve it in HR.

2. You need to give your operators half an hour to respond to something. So in your case you will need to preempt downtime events by 29 minutes.

3. Abnormal situation management consortium are a bunch of folk with some info you might be interested in.

4. PAS are a similar bunch.

willxfmr said:

The cost and complexity of what you are proposing would be enormous. That said, I have worked on some machines that included excellent troubleshooting guides for each fault, and they do indeed cut down on down time. To me, the most important tool for troubleshooting is a well trained tech. I don't remember who uses it as their signature here right off hand, but truer words have never been spoken. "Yes, training is expensive, but ignorance is where the real money is at"

Bubba.

Click to expand...

That quote is Ron Beaufort's

Toine said:

You see a similar thing with cars in the last one or two decades. Cars with lots of electronic monitoring tend to have more defects. In the electronics, "ghost" warning signals while there is actually no defect. Some (mind you, not all) "cheap" lines or brands have proven to be significantly more reliable than upmarket models stuffed with electronics. If it is not in there, then it cannot break.

Click to expand...

Same with boilers (endless water heaters), when shopping for one, the decision came down to most people's opinion that brand X fails more often but not the actual boiler itself... it's generally just a monitoring sensor that will inhibit the boiler.

This being said, I've seen the extreme of this in a company that decided not to put fuses in a machine because they caused too many false alarms...
I asked them to explain further their theory when I had a PSU, instruments and PLC input cards burnt because a fuse wasn't in place when I would be more than happy to replace a fuse.

I offered to calculate the fuses for them in the upcoming upgrade of that system... they chose not to pursue the discussion further.

cardosocea said:

...a company that decided not to put fuses in a machine ...

Click to expand...

Bahaha, I love it.
"We keep blowing fuses when equipment fails, let's get rid of the fuses."
"I keep cracking my helmet when i fall off my bike, I'll get rid of my helmet."
"I keep having broken Kevlar vests.."
"The climbing ropes are only rated for one fall of 5m or more.."
"The brakes on my car keep wearing out..."