OT: "We didn't do ANYTHING"

Inspired by Biomorf's thread...

What's the best instances of the customer/end user reporting a problem to you saying "we didn't touch it" or some variation on that, yet eventually the full story unfolded and proved somewhat different?

I used to work for a company that sold machine vision systems to inspect LCDs used in mobile phones. There was apparently a high "false reject" rate. Without telling the customer we added a feature to record the images of failed displays to hard disk. The next time the customer complained of false rejects we showed them a raft of images with fingers and other "foreign bodies" inserted in between the camera and the LCD.... QED.

Or, at another company, the case of a paper manufacturer whose machine "suddenly" started winding creases into the paper. Eventually it turned out the customer had moved the machine themselves (by about 6 feet!!!)and had left a misalignment of about an inch between the unwind and rewind sections of the machine.

I worked as a service engineer and we had a customer who complained of a plc fault on a machine. The fastest thing to do was send them a pre programmed Mitsubishi plc overnight. The next day they said "That doesn't work either".
I had to go to site and the first thing I noticed the wiring to the PLUG AND SOCKET I/O terminals was like a plate of spaghetti. "you didn't disconnect these wires individually?" said I. "Oh no, we didn't do anything".

Lets just say the plc didn't like the mains fed back up it!!!

Oh, yes. The famous 'out of box failure'. When you get it back, you can smell the fried electronics as soon as you open the packing box. You open up the unit, and find half the tracks blown off the PCB and a copper-like sheen on the inside of the case where the vaporized metal has deposited itself. Clearly, someone has connected the 24V input to the mains, but no-one is saying anything, and since we love our customers, we just nod sagely and replace it anyway. Such is life...

I have a story with a slightly different slant on the "I didn't do anything" alibi.

A customer of mine incorrectly connected a servo amplifier and damaged the power section. Given the design of the power connectors and the vagueness of the documentation, this was fairly easy to do with this particular unit. Nevertheless, we returned the damaged unit to the factory for repair. It came back with a note that said the factory tech had found no problems and returned it with no repairs made. It still showed the same behavior, so we double checked all the rest of the connections which looked correct. Then we swapped a known good amplifier and the system worked correctly. We re-submitted the damaged unit for repair, this time with a detailed description of what had been done wrong as well as a detailed description of the behavior that led us to conclude that there was a problem in the first place. We acknowledged that given a cursory check the axis might appear to be OK, but the motor could not develop any significnt amount of torque.

The unit came back a second time with a note from the same factory tech saying he had found no problems and performed no repairs, and suggesting that the people involved might benefit from attending some factory training so they could properly install and troubleshoot these units, avoid unnecessary expenses, stop wasting people's time, yada, yada....

This time, however, the axis worked perfectly when we installed the original amplifier. When I took the cover off, it was obvious that the factory tech had replaced the power module, since the original power module had been wave soldered in place and now it showed signs of having been attacked with a hand-held soldering iron. It appeared that the factory tech had played a little game of CYA so as not to have to admit that he missed the problem the first time.

"This industrial PC you sold us came with a virus !"

I won't say it was an impossible thing, since Sony had done just that a few months before. But zero other customers had made that complaint, and we build the hard drives from an image file. But I remained skeptical.

"The virus had to be on your installation CDs because this system has never been connected to the Internet."

I hadn't thought of that possibility. Are you certain the operators can't reach the Internet from that machine ?

"Your service tech must have introduced the virus into our network because our firewalls are tested and impenetrable."

What's your public IP ? [An hour later, connected to the WiFi at a coffeeshop] Here's a copy of your PLC program, your drives configuration, and the addresses where I could load data to override the chlorine dosing pump.

The IT manager at that site later won recognition for his work securing the plant networks. I read his report: "We detected a virus infection....."

We had a furnace where the front door had jammed a few times and it was left unattended. The heat caused the door to warp. The customers complained that there was no alarm sounded. On our field service trip there, we found a wire stuffed into the alarm silence button to keep it pushed in.

My most interesting one was a blower that just "self destructed". No one did anything, of course. After some serious investigation it was determined that an operator had disconnected the high temperature alarm because he was annoyed by all of the "nuiscance" alarms.

Some years ago I was called out to one of our plants at 2.00am because they could not get a machine to start. After a 200 Mile drive I arrived and checked the system to make sure everything was in order. I then questioned the operator about his start up procedure. The discussion went something like:-
Have you pressed the start button?
What start button?
The green button with the label which says Start next to it.
I didn't know I had to do that.
I duly pressed the button and drove back home.

Got called to a machine one night because it "just quit working".

I opened the control panel to find half of the terminal blocks ripped off their din rail or bare wire just hanging at the bottom of the cabinet.

After a fairly heated conversation with the operator and shift supervisor the operated finally admitted to snagging a 2" conduit with the overhead crane and ripping it about 3' in the air.

That was a very long night.

Manglemender will remember this incident (or certainly one like it, there's been a few.... )

A customer with a S5 CPU called up to complain that the CPU had lost its program for no reason.

Eventually it transpired that it'd gone to STOP for some reason, e.g. card failure. The shift tech couldn't work out why, so had wiggled the switch on the CPU from run to stop and back again a few times in quick succession......

Another variation on this... Way back in the day, I was at a customer's site and he was complaining that some software we'd provided to him wouldn't talk to an industrial HVAC controller. He said that he'd tried it on every PC in the office, but with no joy. I opened up the DB9 connector on the cable coming from the controller to check the wiring, and just about jumped across the room when I received a 240V shock. (This was in the UK -- none of those wussy 110V jolts over there! When I was growing up, if you got a shock you ached for a week.) Turned out the earthing on the HVAC controller was all screwed up. Worse, since he'd tried it on 'every PC in the office', he taken out the serial ports on about a dozen motherboards...

Here's a good one, I had only started my latest job on the monday (as site software engineer), on the saturday one of our plants "stopped working" (S7 with over 2000 I/O + ifix) & as the engineers were used to re-boot the scada & the plc one guy re-boote the plc by the key yes run to stop then reset over & over again yep lost the program, as no one had my telephone number they called out a local systems supplier (very large & well known company), they proceeded to spend the whole weekend on the problem as they had a few weeks before uploaded an image of the processor it was asumed hey could fix it, monday morning I arrived & these guys said it was one of the profi valve islands, we jumped into the car & travelled down to the supplier, They promptly "repaired" it & we rove back, fitted the island, still bus error?, in the meantime I noticed an e-stop in on the plant, pulled it out & hey presto no bus fault (24v to module via e-stop circuit for safety). however still problems, in the meantime my boss had contacted the original supplier of the system & he turned up & downloaded their copy however still no joy, I was standing in the background watching (new to job & didn't want to interfere), however noticed the ether link to plc power light going dim, I jumped in the car again down to local supplier for a £20.00 link 9 pin to rj45 & a spare cable, hey presto all working again.
All suppliers leave site & I go home, however I got a call later on that evening saying the agitators not working, found that there were no analog signals to the VSD's it appeared that the supplier had put in an out of date copy of the software (agitators were upgraded to variable speed at some time by them).
The only thing I could do was up the min speed to get them by as I had no code or pg at that time.
It cost over £9,000 for the callouts for the two systems houses + te "Repair" to the valve island & in the end cost £30.00 to repair it by me 3 days later.

While most of my work is with irrigation pumps, a couple of my customers are tomato packing plants with their own "maintenance" personnel, So I've seen the GAMUT of "We didn't touch it" scenarios. Of course the more memorable ones are the times it took a long time to find the problem. while the plant was down, employees were standing around...... and then it became obvious that not only had someone CAUSED the problem, but all concerned had given a whack at trying to FIX it before calling me.

I am really more amused by (and am writing to share) an amusing response that seems to just be human nature:

After I've evaluated a problem, discovered the source, called to get a price on repairing or replacing it, and presented ALL the info to the customer....... a large percentage of them will look me straight in the eye, dead serious, and say: " No... That can't be it, because it was working just fine YESTERDAY."

Another, less perplexing but still amusing idiom, is when the customer needs a quote on a piece of gear or an upgrade on existing equipment: A common qualifier is "I DON'T need anything 'ELABORATE', now......"

I suppose "elaborate" stuff is perceived to COST more.



Stationmaster

This is my most recent one. We built an ice conveyor using 30hp blowers to transport ice to each processing line in the plant. If the system faults anywhere it will tell you where the fault is. It's been up and running for about 8 months without a problem. I got a call at about 2 am, that both blowers are coming on at the same time (one is a back up blower) and keep blowing the piping apart. since the blowers are electrically wired to where if one is one the other cannot, and the program will not allow this, and the gate valves have sensors on each position to only allow the correctly valved blower to operate, I at first said that's pretty much impossible. So I drove down to the plant, and hooked up the laptop, and watched the system run. It cycled normally...until it turned on the screw to raise the ice out of the holding area and drop it into the pipe to blow into the plant. At this time the second blower started. I could not see the plc calling for the other blower, and was wondering why the pressure switch was not shutting down the system. I shut it down, and checked the wiring.
This is going to be the long part, The interlocks on the blower starters were bypassed, the switches on blower B (the backup blower) were bypassed to tell the plc the valve was in the closed position even though it was open, the pressure switch was bypassed, and for the win the output wire to the back up blower had been moved 8 terminals down on the panel and put into the same one as the ice elevator. Of course nobody had done any wiring on the system. The actual problem after putting all of the wiring in the correct place, was a faulty air pressure switch, shutting the system down and throwing an error. The responce I got was....the system should have been built so this could not happen.

Not sure if this is actually a "We didn't do anything" but I found it quite humorous.

We have a customer on an island in the Caribian Sea. Said customer buys several conveyors from us. Several of those conveyors can not reset with material in them. In the stop sequence we have delays to prevent this from happening. Well the customer kept breaking the conveyors and requested new parts under our warentee. We supplied the parts and even sent a service tech to the plant to see what the issue was. The Service Tech can find nothing mechanically wrong with the machine. Since the machine was built before I started woorking for the company I was told look at the program something must be wrong there. I look and I look,everything looks good to me so I tell my boss, let me create a data logger in the program to see if I can catch what is happening. Boss gives me the ok so I create the program changes. I talk to our customer's engineer and he informs me he is not comfortable with changing the program. So I book a flight and head to the plant. I install my new program and check to see it is logging what I want. Since I am out of town I decide to stay at the plant for awhile and watch the machine to see if I can see anything.

Promptly at shift change I find the problem. The operators instead of using the stop button to stop the system use the E-stop. Reason being is when you use the stop button the system runs for another 30 seconds to clear itself out before shutting down. That was to long for the operators to wait.

I showed this to the plant engineer and also showed him my data log had caught this event. After a few disgusted words from the engineer about how he never thought of that I left.

Awhile later I went back to the same plant and the E-stop in question now had a plastic cover over it with a note saying use the stop button to shut this system down.

I know that covering the e-stop is illegal here but I did not question anyone on it since we have not had to supply any warentee parts since my previous visit.