Operators that don't know their systems

[tomalbright]

Speaking of not knowing their systems: In about 2004 I was working offshore on an old GE 90/70 PLC system that had replaced an ungodly amount of relays. It was basically a simple open-close, start-stop application. Wonderware was in use on the system, and was really the only way to do any controls.

They brought in a new guy - basically, in his late 50s with over 30 years of offshore experience. Told him this was a PLC system here's Wonderware, pretty simple, blah blah... Which was absolutely true.

I'll never forget the man picking up the mouse and staring at it, like he had never seen one before. He didn't come back the next week.

 

[dploof23]

I enjoy funny stories. I could tell a hundred about similar situations. After many years of reflection, I think that 95% of the problems found in this type of owner-programmer-operator interaction story are caused, one way or another, by poor communications.

1. The owner did not write clear specifications that said exactly what he expected the machine to do, so the programmer worked in the dark until near the end when he was able to start the machine and see the problems first hand.

2. The PLC programmer did not develop an outline of his proposed program to give to the owner, with step-by-step descriptions of how he proposed to write the program. No one at the customer company knew what was in the black box until it was delivered.

3. The HMI programmer did not use clear labels on his controls. Some operators could figure out how to run the machine, others did not understand some of the steps. There were buttons labeled "CLOSE". Close What? Stop What? Start What? The process consisted of steps, but no where on the control panel did the operator know what step the machine was performing. This type of mis-communication leads to maintenance problems.

4. There were no overall graphic schematic of the process, so most operators did not have a clear idea of the whole process, but tried to limp by with inadequate knowledge. Training was poor to none, with old guys quitting and new guys coming in with a 5-minute instruction session followed by "you will learn on the job in no time at all".

5. Changes and revisions were made from design start up until the machine was started up. No one had a clear idea of all the changes, as the operators were trained based on the original program that had been changed a hundred times.

6. The machine did not have a good user manual, with clear instructions and directions. There were very few drawings and diagrams.

Lack of commmunication, poor commuication, lack of understanding of the existing instructions, poor training, and such can be summed up as 95% of the owner-programmer-operator problems. The other 5% are really the easy part to solve: the machine malfunctions. Solving the big 95% means that a bunch of people are going to have to learn to talk to each other.

Why did the other operators not know about the reject delay timer? You got it: poor communication. For some reason, the people who knew about that timer did not think it important that all operators be instructed about its function. Ideally, the instructions would be included in the HMI panel program.

Lancie I love your message here, information sharing and communication is HUGE when it comes to the success (or lack there of), of a machine. I work for a company that blow molds plastic containers. Where we differ from most is we build our own blow molders and auxillary equipment in house. My job is the electrical controls, PLC, HMI programming etc. On the newest machine we built I designed the HMI to include a trainig section with pictures and diagrams, fault help, a ladder monitor, motion control troubleshooting....I even have my electrical schematics right on the HMI. Long story short I had been programming this HMI for the better portion of a week when one of the electricians working in the plant asked me why it was taking so long. I told him what I was doing and that there was 100+ screens on the HMI. He said I was wasting my time, no one would use it, etc.

The operators love it. When there is a problem it "leads them to water" for lack of a better expression. Also the newly hired techs are required to use it as a training tool as well.
In general the entire technical staff benifits from it and they have told me as much. Plus, if all that time spent designing the screen saves me a call at 3am then it is priceless as far as I am concerned.

Dave

 

[OkiePC]

7. The operators sees the alarm as an opportunity to test his fire button speed on the reset button. Then they think if they clear the history it will "stay gone"
7a. They keep building better idiots.

We have a robot cell with a reset button with contacts for the cage and light curtain, and another to the PLC. It's a big green extended head lighted button.

After closing the cell door, you press it once, the safety circuit resets. This takes 1/4 second, then the PLC sees the safety is happy for a full second then sends a fault reset signal to the robot. The robot does its thing and gets ready this takes a couple seconds. Press the button a 2nd time and the robot starts running.

Or you can press the button 8 times per second for three and a half seconds (which I measured repeatedly with online edits and ladder code).

I had to walk him through it. "Press the button and wait. Take a breath. See that big green icon on the robot screen? Okay now hit the button again..."

 

[tomalbright]

You know this jogged my memory all the way back to yesterday.

I was doing some Wonderware work (trying to fit a rectangular monitor into a square monitor hole. Our PCs blow, BTW) and had my hands ON THE KEYBOARD. This is #3 of the control PCs in the control room, so there were two others. The alarm sounded, and like Okie says, he jumped up, reached over to my keyboard, and slapped the space bar. The alarm, which beeps 8 times, was on #7, and hit #8 when he pressed the space bar. I had absolutely nothing running on the PC.

I said "why'd you do that?"
Him: "I silenced the alarm"
Me: "you see wonderware running over here?"
Him: "no, but it worked!"
Me, waiting for the next alarm while watching his boss behind him hovering over the keyboard. Got it in three beeps.
I don't think we finished the conversation. I think his eyes glazed over.

 

[Lancie1]

I don't think we finished the conversation. I think his eyes glazed over.

I hope that sometime later, the boss or someone sat down and tried to teach this operator how the plant worked. Either do that or fire him for inability to learn by magic.

 

[bulletin blues]

First and foremost, I must admit that as I get older I become more humble and realize that the operators often have absolutely no clue "how" the equipment electrically/mechanically works, nor, I guess, should they, as they make a few shillings per hour less than I. With that being said, I work on many pieces of equipment that give sure signs of when it is operating. Start the pump, you'll see hundreds of gallons of water pouring over the side of the tank. Start the motor, you will hear a screeching and whining noise as it ramps up. Start that press, the ground shakes violently beneath your feet. Yet I still get numerous calls telling me the equipment isn't starting. I ask them to turn it on. They hit the start button. I stand there dumbfounded, with an equally dumbfounded look on my face, as I feel the ground shake, push my earplugs in further do dampen the deafening sound of the belts squealing, and stand back from the water dumping on my clothes, and they just look at me like "where do they get these maintenance guys, I told him the equipment isn't on because the green light on the push button isn't lit and he looks confused?"

 

[Muffler Bearing]

I don't fault operators for knowing every mechanical & electrical detail of their equipment. But, there is one story I must share because it still kind of makes me mad.

There was an operator of 35 years who was an "expert" on a certain process. When there was something wrong, he could usually troubleshoot it and direct maintenance staff to fix his machine.

This day, he determined that a chiller was not properly cooling a chemical solution.
The conversation went like this :
Him : "This thing isn't cooling at all! It probably needs some refrigerant!"
Me: "How do you know what the right temperature is? There's not a temperature gauge. "
Him : "It just ain't right, and I'm not working until it is."
Me : "Once again, how can you tell? Can you give me any idea how cool the solution is supposed to be?"

At that point he basically stormed off and went to my supervisor. Sure enough, five minutes later, I was called into my supervisor's office. I said the liquid felt cool, there's not a temperature gauge anywhere, and he can't give me a reason why it's out of spec. Needless to say, the elite operator wins the argument, and it was agreed that it was time to call in our HVAC contractor.

The HVAC contractor had to cut a hole in the side of the chiller in order to get access to the refrigerant lines, and they checked the refrigerant pressure. No problem noted with anything on the chiller.

$600 service charge, and now the chiller has a big ugly metal hole cut in the side, which really irritated the higher ups because this was in a "show" area where we typically took customers. HVAC contractor tech tells the operator "There's nothing wrong with this chiller. What is it supposed to be set at ? It's reading 50 degrees F right now"

While the contractor had been working, I went and got the manual to the machine. I looked it up.... sure enough, 50 degrees F was the cooling loop set point. 6 hours and $600, and all the operator could offer up was "Huh. It just seemed warm to me."

 

[Ned_Flanders]

Another simple story: I worked for a confectionary manufacturer and one of the rather attractive ladies who ran a line had a problem.
It was something simple that required adjustment and I took 5 minutes to explain what the problem had been and how the adjustment would cure the problem.
HER eyes glazed over so I said "You don't believe me?"

She replied "I don't know what the f*** you're talking about!"

 

[Goody]

My problem was not the operative, it was the owner of the company.
It was my first time there but I had heard so many stories of what a firebrand he was.
Especially with outside contractors.

He had permanently banned so many for seemingly minor things - ie walking onto the site with your hands in your pockets - having a cup of coffee while working etc.

He was at me immediately, telling me how much money he was losing for every hour the line was down. He even got irritated because my laptop took so long to boot up. Leave it in hibernation mode or boot it up before you come I was advised.

The control panels for this machine were about 50 meters long with about 50 doors all of which were open. While I was studying the program he was in and out of each door pressing things and calling me to come down and 'check this'
Each time I tested whatever he thought could be the problem I had to stop what I was doing, walk there (I think he expected me to run - I could see impatience in his eyes) then get back to the program, losing my thread every time.

Eventually, he stood with me at the PLC and laptop actually blocking my view of the laptop screen whilst bombarding me with questions 'What does this line do? What does not equal mean? Why are some of these blue and not others?

In a moment of utter frustration and in a raised voice I said 'will you just f*** off and let me do my job'
There was an absolute stunned silence.

I had just enough time to think to myself ' That's you banned Goody'
When he said ' You are right. I am only hindering you'

He left and I did my job.

A while later I was watching the machine running when the manager came up to me and asked if it was true that I had sworn at the owner.
I laughed telling him it was true.

I have been back many times and I am held in high esteem, especially by the workers who always ask me to re-tell the tale.

 

[NetNathan]

Classic...
When I arrived the operator had already dismantled a large push button pendant on a vertical mill.

Why...you ask?
He stated the problem must be there..... because nothing happened when he pushed "this button".

It was the milling head "UP" button...... to a milling head that was ALREADY on its up limit....

I asked him why he did not try the "Down" button...he said he wanted to go up not down.

It took me 2 hours to put the push button pendant back together.

 

[controlconcepts]

Not a PLC but a high level alarm control panel...

Speaking of cleaver operator's that know a little to much.

Manager " The alarm light flashes but the alarm buzzer will not sound it's probably no good"

Our Tech " Ok we will take a look at it"

Open panel simulate tank high level lamp lights no buzzer.
ok test buzzer output relay 24 VDC, check connections to wire, ok trace wire to terminal blocks, ok on the door panel where buzzer is check voltage 0 VDC !!!!! tug on all the connections start to un screw wires off terminals check and inspect all good.

check connection at the buzzer pop wire off "looks ok" test bare wire for voltage O VDC HUH wait what is this clear scotch tape on the bare copper. Yep
The operator did not want his nap interrupted during a tank fill with that awful buzzing to tell him to stop the pump and not over fill the tank { A petro Tank at that }
Manager asks what we found showed him the tape and no repercussions for the operator.
Like someone already said
"you can design a fool proof system" "but then they will design a better idiot"

 

[OkiePC]

It took me 2 hours to put the push button pendant back together.

http://www.01521.com/hp/2009-11-27-milking-stool.jpg

 

[Lancie1]

In general the entire technical staff benifits from it and they have told me as much. Plus, if all that time spent designing the screen saves me a call at 3am then it is priceless as far as I am concerned.

Dave

A job well done. But if all operator controls were like this, there would be fewer of those "dumb operator errors" to tell funny stories about.

 

[Donnchadh]

Hi

Just to add another one

It was the first time that I had been given the jobork write a full program and HMi for a machine, up to that point I had added a few lines of code, trouble shoot etc. the company I worked for had actual got a programmer in to write this program and I felt I should have been let at it. Anyway the program had to be rewritten as it was not working as it should and off I went and spent a whole bank holiday weekend at it just to prove to my boss that I could.
To be fair he was very happy with what I had done and we shipped it to site that week. I was full if it and sure enough I got a call less that a week later , one off the heads had been cut if by the indexer. I was so sure that this could not happy with the code as I had written until I got to site. There was an alarm up on the screen showing that head 3 was not up and that it had stopped the indexer. I said to the operator how did this happen and he told me that the alarm had come up over and over so he just taped the reset button down and it worked away until the crash.

I learned that day what a one shot but was thanks to that operator


Donnchadh

 

[tomalbright]

Another one snuck out of my repressed memory:

After a hurricane, our platform system was unable to pump oil, but we manned them anyway, and did repairs on what we could. A few days in, the operators were performing an ESD test. I went down to the panel after they complained it wasn't working. After a little digging, I found somebody had actually stripped and spliced the ESD wiring, and covered it back up where you couldn't see it - we couldn't shut down in case of any emergency, like a FIRE. Only thing running was the generator, but I fixed the wiring, reported it to Upper Control, and didn't sleep well at all that night. It was actually terrifying that an operator would do that. We never found out officially who did it, but I had a good suspicion of which operator it was.

Disabling your primary shutdown? Not allowed, and could result in jail time. Simply to not be bothered by an alarm.