TConnolly
Lifetime Supporting Member
Yesterday Otis came up to my office asking for help with a small assembly press that he had been trying to troubleshoot for a couple of hours.
This assembly press is about the size of small refrigerator and is part of a larger flexible manufacturing cell. Because the cell is flexible and reconfigurable the press is powered from 120V and just plugs into a wall outlet. Hydraulic fluid comes from a large 480V multi-pump HPU through a distribution header that rings the room and a pair of quick disconnect hoses. The HPU feeds multiple flexible cells and no others were reporting any problems. It interfaces with the HPU through two pairs of relays keeping the power systems separated.
The press was cutting out mid cycle. Nothing in the program would cause that except a shut down of the HPU - which wasn't happening - or the operator pushing the stop button or breaking a light curtain beam.
Otis and I headed down to the manufacturing floor. No problems with the stop button or light curtain so I thought it could be a relay problem making the PLC think the HPU was off. I swapped out the ice cubes. Nothing. I forced on the pump running input to force the PLC to believe that the pump was running. Cycle would still terminate. It was then that I noticed a flicker in the indicator lights. That led me to discover that for a few milliseonds power was dropping out to the PLC IO and was almost immediately restored. Engaging the retract always has a big thump to it as the hydraulics is just a bang/bang DCV valve and piston, everything else on the press is pneumatic. I'm thinking something has to be loose and the thump of engaging the retract is making it move. There are a number of hardware safety interlocks (several shields, light curtain, anti-tiedown, estop) ahead of the PLC outputs so one by one I worked upstream to eliminate them.
Otis wondered if it was the retract hydraulic valve. I didn't see how it could be but to a non-electrical guy like Otis the retract valve was a logical common denominator. While I continued to check all the connections Otis crawled behind the machine to get a part number off the valve. About the time I had traced the signal back to the branch circuit fuse block all the power dropped off the machine. I asked Otis what he did and he said "Nothing." Then the power came back. A few seconds later it went back off. I said "Are you bumping the plug?" and he said no, he was moving a hydraulic hose and nowhere near the plug. Then it hit me what was happening. I leaned over the top to peer behind the machine where the hose connects to the header and started laughing when I saw the problem. I told Otis "I know what's going on and we're gonna look really stupid for having spent so much time on this."
Earlier that day, in fact, right before the problem surfaced, the operator had swept behind the machine. She decided to keep the power cord up and out of her way by wrapping the cord tightly around the hydraulic hose - the retract pressure hose in this case because it passed in front of the wall receptacle before connecting to the header. On retract, the hose flexed, barely unplugging the machine. With the cycle terminated, the hose flexed back, pushing the plug back into the socket. It happened so fast that the PLC itself didn't shut down and the HMI never rebooted, all being able to ride through the milliseconds long bump.
Since the cell is set up for flexible manufacturing, the assembly press is on a mobile base. At one point I almost asked Otis to unlock the casters and tow it out of the cell to have more room to work. Had I done that, when I pushed it back in the hose would have been in a slightly different place and the problem suddenly vanished. That would have left behind an unsolved mystery, a true ghost in the machine.
This assembly press is about the size of small refrigerator and is part of a larger flexible manufacturing cell. Because the cell is flexible and reconfigurable the press is powered from 120V and just plugs into a wall outlet. Hydraulic fluid comes from a large 480V multi-pump HPU through a distribution header that rings the room and a pair of quick disconnect hoses. The HPU feeds multiple flexible cells and no others were reporting any problems. It interfaces with the HPU through two pairs of relays keeping the power systems separated.
The press was cutting out mid cycle. Nothing in the program would cause that except a shut down of the HPU - which wasn't happening - or the operator pushing the stop button or breaking a light curtain beam.
Otis and I headed down to the manufacturing floor. No problems with the stop button or light curtain so I thought it could be a relay problem making the PLC think the HPU was off. I swapped out the ice cubes. Nothing. I forced on the pump running input to force the PLC to believe that the pump was running. Cycle would still terminate. It was then that I noticed a flicker in the indicator lights. That led me to discover that for a few milliseonds power was dropping out to the PLC IO and was almost immediately restored. Engaging the retract always has a big thump to it as the hydraulics is just a bang/bang DCV valve and piston, everything else on the press is pneumatic. I'm thinking something has to be loose and the thump of engaging the retract is making it move. There are a number of hardware safety interlocks (several shields, light curtain, anti-tiedown, estop) ahead of the PLC outputs so one by one I worked upstream to eliminate them.
Otis wondered if it was the retract hydraulic valve. I didn't see how it could be but to a non-electrical guy like Otis the retract valve was a logical common denominator. While I continued to check all the connections Otis crawled behind the machine to get a part number off the valve. About the time I had traced the signal back to the branch circuit fuse block all the power dropped off the machine. I asked Otis what he did and he said "Nothing." Then the power came back. A few seconds later it went back off. I said "Are you bumping the plug?" and he said no, he was moving a hydraulic hose and nowhere near the plug. Then it hit me what was happening. I leaned over the top to peer behind the machine where the hose connects to the header and started laughing when I saw the problem. I told Otis "I know what's going on and we're gonna look really stupid for having spent so much time on this."
Earlier that day, in fact, right before the problem surfaced, the operator had swept behind the machine. She decided to keep the power cord up and out of her way by wrapping the cord tightly around the hydraulic hose - the retract pressure hose in this case because it passed in front of the wall receptacle before connecting to the header. On retract, the hose flexed, barely unplugging the machine. With the cycle terminated, the hose flexed back, pushing the plug back into the socket. It happened so fast that the PLC itself didn't shut down and the HMI never rebooted, all being able to ride through the milliseconds long bump.
Since the cell is set up for flexible manufacturing, the assembly press is on a mobile base. At one point I almost asked Otis to unlock the casters and tow it out of the cell to have more room to work. Had I done that, when I pushed it back in the hose would have been in a slightly different place and the problem suddenly vanished. That would have left behind an unsolved mystery, a true ghost in the machine.
