Just in case you needed a laugh

Just in case you're having a rough day and need some consolation that you're not actually that bad at your job...and plus, I just need to tell somebody who will understand how ridiculous this is

I have been liaising with a large multinational company (who shall remain nameless) who makes a certain type of machine. Said company has installed two of these machines in one of my client's site. There has been much pain and suffering trying to deal with this company already.

Anyway, we asked if we could have a copy of the program so that we could set up communications with the central control system. No, their program is proprietary and they are unable to release it to us. Okay, that's fair enough I guess. I mean, odd, because it's not like it's a really whizzbang high tech cutting edge machine, what it does is quite basic. But no problem. As it turns out, despite all this secrecy, they haven't actually put any security on it at all, so we can quite happily upload it and marvel at how utterly ridiculous this code is. They must use identical code for every single machine they ever build and just disable the bits they don't need. There are about 75 programs in this thing, of which 5-10 are inhibited completely. There are 718 AFI instructions. The program takes up to 15 minutes to download. Again, not a complicated machine. But anyway, that's all beside the point.

We write up a spec and ask the company to make the changes for us. Unfortunately there are only 5 people in the world who are allowed to touch the code for their PLC's, only one of which does any work for Australian clients. Oh, and he's in Sweden and is a contractor. So we will need 4 month's notice for him to log in remotely and make these changes. Okaaaay...

4 months later he logs in, ignores the spec we wrote, makes about 5% of the changes and pats himself on the back for a job well done. After many phone calls to a grumpy programmer in Sweden who's been forced to get out of bed and finish the job we have it working more or less the way we want it.

A couple of months later we're ready to do the other machine. This time he's not available at all, so he's going to make the changes offline and send their local service tech in to download the program.

Local service tech shows up. Alarm bells start to ring when he spends two or three minutes trying to work out which card is the processor and which is the ethernet card. I just steer him to the server room, patch him into the network, give him all the IP settings and away he goes. Well, I mean, I leave him alone for 2.5 hours and by that time he's managed to download a new PLC and HMI program. We test it, all OK. He goes home.

3 weeks later, the customer turns off the machine. When it powers back up, the HMI doesn't look right. Our friendly local service tech has forgotten to make the new project the default one to boot to. I'm now interstate, so over the phone I talk the client through the process of getting into configuration mode via one of the many back door methods, working out which of the 11 runtime files left in various places on the HMI is the right one, selecting that file to be run at bootup, and restarting the terminal. It boots up, and it's got all the new stuff on it now, but nothing works. I dig through all the RSLinx Enterprise settings with the client and confirm that's all OK, then I jump online remotely and upload their top secret program. As it so happens, the PLC is in remote program mode. Okaaaaay...it also has a minor fault to do with loading from the CF card. Okaaaaay...so I switch it to run mode, and everything on the HMI starts working...except all the new stuff. Of course, our friendly local service tech has forgotten to write the program to the CF card, and so at power up it's overwritten the new program with the old.

So we ring the company who puts someone on a plane and sends them to site (because God forbid they just email me a copy of the top secret program to download myself). They arrive a few hours later and open up the cabinet. 5 minutes later I get a call from the client.

"uh yeah, this guy wants to know where to plug his laptop into to download to the PLC."

After a stunned silence, I replied "if he doesn't know where to connect his laptop to, I would suggest you take it off him, close the cabinet, and get him offsite before he hurts himself".

Oh - where is the like button!!!! LOL

We have had cases like this where the machine builder got a third party to produce the software and neither kept or even understood it themselves. On one occasion a fairly well known supplier of refrigeration compressors was unable to fulfill their obligation to integrate their systems with a plant SCADA system because the only person who had any knowledge of their proprietary comm's protocol had left the company and it was not documented anywhere.

How about a company sending a tech from Germany to Detroit to update the programs/parameters for PLC, HMI & servo controllers - but he doesn't bring a laptop, programmer for the servo controllers, any printout or flash drive with copies of the files; never makes 1 change to the machine & charges $9,500.00 for the service call.

Aabeck said:

How about a company sending a tech from Germany to Detroit to update the programs/parameters for PLC, HMI & servo controllers - but he doesn't bring a laptop, programmer for the servo controllers, any printout or flash drive with copies of the files; never makes 1 change to the machine & charges $9,500.00 for the service call.

Click to expand...

I would say that sounds like a company that does not get paid.

I used to work for an integrator that did turnkey machines for OEMs. We, of course, would give the OEM the source code for everything and make sure they had a copy of the programming software. We had a surprising number of customers that either didn't have a controls engineer on staff. They just made the same machine over and over with no alteration or improvement year after year until a competitor put them out of business or they payed us do a new version of the machine for them. Invariably, I would end up having to make service calls for the dumbest stuff, like change an input from NC to NO four years later because the sensor they used to use went obsolete.

The craziest part was that some OEMs has who they thought was a controls guy, but they really just debugged panel wiring and loaded a program they didn't write. These guys couldn't make a program change to save their lives. We tried to stay political and not rat these guys out, but if we were working with the OEM, it was because the OEM was finally developing something new and that always rocked the boat. Sometimes the faux controls guys would make up damaging ******** about why they couldn't make simple changes themselves and jeopardize our business and we'd be forced to expose them. Once, I was called into an OEM that was threatening to stop doing business with us because their guys couldn't figure out our "overly complex" program. We arranged for me to make a simple change using the control guy's laptop and walk him through it so he could at least do something. The president of the company decided to supervise the process from over my shoulder. When he saw that the programming software was not installed and the plastic wrap was still on the DVD, the president turned red. When I made the super obvious change in about 20 seconds (swap two states in a state machine to change order of operation), he turned purple and fired his guy on the spot. Similar firings happened a few other times, but never in such dramatic way.

Thanks for that story, makes me feel superior if for no reason than in letting me recognize that my own foibles are less dramatic.

I did a project about 12 years ago that, for lack of a long tedious story, required my program to be VERY complex with regard to interlocking of machinery and conveyor systems, because everything about the process was closed loop with recirculating product going back through the machinery for out-of-spec reprocessing, with loops feeding other loops and only one final output. The upshot was that if ANY portion of ANY of the loops failed, the entire process had to go into a precise orderly shutdown sequence, otherwise the mess that was created was stupendous (there were no surge capacity points within the individual processing systems). I didn't design the mechanical portion, and my suggestions were ignored, so I just had to make it all work.

After a solid month of programming and testing, I had it all working perfectly. Then the owner steps in and DEMANDS that I provide a manual override for EACH individual motor and machine in the system. I explained to him how incredibly complicated it was going to be to get the desired product flow and production rate out of this manually and how risky any error was. I was told they ONLY wanted it for testing and troubleshooting, that they would NEVER attempt to run it manually. I agreed to add the manual overrides, but convinced them that I should make it so that each of the 48 of the 30mm red illuminated push-pull buttons on the control console used to accomplish this would flash when in Manual Override as a warning to operators not to leave it that way. So having used relay outputs (because they wanted 120V control), I just made the relays flash on and off.

Fast forward 6 months, I get a call because the SLC 500 PLC is "failing". 5 minutes of troubleshooting on site and I discovered that over half of the relay output cards were failed, including welded contacts, something I had never experienced before. I replaced all the cards, turned it back on and ALL 48 of the red pilot lights were flashing at me in unison. Had I been afflicted with epilepsy, I would have been in a grand mal seizure! Turned out that the operators, in their infinite wisdom, decided that my program was NOT better than their ability to manually control this system. But in order to avoid the pitfalls that I had spent so much time working around, they operated it piecemeal, one system at a time, very slowly, resulting in less than 10% throughput capacity. Management never noticed...

And the 48 flashing red lights? No kidding, the operators wore dark sunglasses when at the console! 6 months of flashing the output relays once per second however, used up the rated life of the relays on the output cards.

Close the damn door!

I had a call to a plant I had installed a simple setup of an air supply fan to a compressor room and an exhaust fan. The supply air fan speed was dictated by a differential air pressure switch. As the pressure in the room decreased due to air compressors chewing up the air, and started to run close to a vacuum, the supply fan would speed up to try and normalize the pressure. I believe this to be a pretty standard set up for compressor rooms.
I got called in by the chief power engineer to look at the "failed" supply fan drive. I went and tested it in every way I could and all seemed good. I gave the shift engineer (not the chief, after hours) an invoice and off I went. The next day I get a call from a very distraught chief engineer who was very upset that his supply fan drive was outputting 0 HZ and he had to pay a service invoice. I went back in to meet with him. The first thing we did was enter the room and open the panel. Drive at 0 HZ. He said SEE! I reached back behind him and closed the door, almost immediately the supply fan started to ramp up. I explained to him about the differential pressure in the room and how when the door opened there was no need to run the fan.
He thought this was a stupid way to control the system, I said it outsmarted you didn't it?
He was not happy but I was laughing on the inside!

thank you ASF for bringing this topic up for discussion ... the complaints in your original post are being mentioned on a regular basis by the plant maintenance managers that I've been talking with lately ... here's what they've been telling me ...

basically, over the years our company has always relied on calling in outside vendors whenever we've needed help troubleshooting our PLC-controlled systems ... lately though, instead of an "older guy" who knew his way around – we're starting to get more and more "young guys" showing up - who obviously don't know what the heck they're doing ... they seem to spend all of their troubleshooting time just talking back-and-forth on their fancy cell phones with "tech support" - who's trying to walk them through even the most basic procedures ...

the "outside vendor" idea worked OK in the past – but now we're finding ourselves between a rock and a hard place every single time something PLC-related needs to be done ... we've GOT to get our own maintenance people up to speed on this type of technology ...

Click to expand...

underlying this is another sore spot – the JIT (Just in Time) operation model ...

back in the old days, a plant would make a "lot" of their product - and then store it in a warehouse ... the finished products were then pulled – and shipped – whenever a customer called in and placed an order ...

but then ...

the financial consultants came up with a grand new idea ... they pointed out that all of those products being stored on warehouse shelves represented a LOT of money – money that was "tied up" and therefore wasn't available for investments in research, expansion, dividends, and so on ...

JIT was born ...

using the "Just In Time" model, the plant only produces products once it already has customers placing orders for them ... this cuts down on the amount of money tied up in material being stored in the plant's warehouse ...

sounds great ... and it works great – until ...

now think about what happens whenever the plant's production equipment goes on the blink – and suddenly "Just In Time" turns out to be "Just Too Late" ...

keep in mind that the plant's downstream customers are probably working on a "Just In Time" model too – and if your plant can't meet YOUR customers' demands – then they won't be able to satisfy THEIR downstream customers' demands either ...

in many cases, we're not talking about losing just an "order" – we're talking about losing CUSTOMERS – who are suddenly being forced to look for a more reliable source of supply ...

so – two points:

(1) relying on outside vendors for emergency technical support isn't as dependable it used to be ... THINK: basically the "old guys" are retiring – or dying off ...

and ...

(2) emergencies seem to be a lot more common – and a lot more critical - these days ...

THINK: the "Just In Time" model means that you no longer have any "Just In Case" slack available in the system to save your hide while you're waiting for someone qualified to get the machinery back in operation again ...

those two points – considered together – go a long way towards explaining the sense of "desperation" that I'm hearing from more and more plant maintenance managers these days ... I'm sure there's a sermon in there somewhere ...

Totally agree Ron. I think the problem is mostly that the paths of the accountant and the paths of the service/maintenance engineers never cross. The accountant has a much better understanding of the financials and can see massive benefits in the JIT method that you or I probably don't fully comprehend. And likewise, the accountant doesn't have the technical understanding to fully realise the fragility of running a factory that way.

Oh, and an update on the original debacle. My suggestion that perhaps this guy should be shown the door before he hurt himself was unfortunately (but understandably) overruled, so I talked him through the process of unplugging the uplink to the network, so that he at least could only break his own company's equipment, and plugging into the PLC's ethernet card. 45 minutes later I got another phone call - he's just managed to get the PLC to show up in RSLinx, but isn't really sure how to get the program into the PLC, or onto the CF card...and seems to have only a vague understanding of the differences between "going online" and "downloading".

I'm not sure how long they were there for that night, but he did eventually manage to make it "work". And by "work", I mean that every time there's a power cut in this area renowned for spectacular electrical storms, the PLC will power back up in program mode, and all of the recipe data will be lost. So as long as their operators can open an electrical cabinet to switch the PLC to "run", and manually re-enter that day's recipe, no problem!