Updating a previous post

I had previously asked if anyone had any advice or warnings for me that would let me know what to expect with a new computer controlled production line that was being installed where I work. The first post was titled "Warnings and/or Advice Needed" Aug 23, 2007 1:09pm, in the text area it is number 31152. I sometimes go back and re-read it myself. I would like to once again thank everyone for all the replies I received. Here's the update. The line went into production the weekend after the 4th of July this year. We (the maintenance dept) were allowed to assist in the installation as far as bolting the line together, we were NOT allowed to assist in installing any I/O or wiring or program development. There were no changes to the line allowed during the installation even when we found oversights or mistakes. The control program, from what I've seen, is in the 20,000 line range written in Visual C++. The computer controlling the line crashes almost daily, production has been down from 5 minutes to 5 hours trying to get it going again. 3 different people have done programming on the system. The information displayed on the operators screens is more often wrong than right. On average we are now able to achieve 65% daily production over what we used to be able to do. The company that installed the line left us with no hard copy of the control program, no electrical diagrams, no documentation of any kind and no training given on the system. The wrong units routinely go down the wrong final assembly lines having to be hand carried back to were they belong, won't move to the next station, sometimes get run off the line onto the floor which ruins the product. We lose ethernet conectivity to any of the 5 or so control cabinets bringing part or all of the production line down. Twice we have been unable to get in contact with the company to troubleshoot the system when down for hours at a time. The advanced diagnostics that were suppose to be included in the system to pin-point where the problem is is non existent. A generic fault message is all we get 99% of the time. The I/O modules used in the installation were designed and built by the company that did the installation and we are now locked in to them for replacement parts. The operators are not allowed to turn the line off at lunch or breaks or in between shifts for fear that it won't re-start when needed. The company that did the installation said we could'nt do what we wanted to do with a PLC so we had to go with a computer. I have counted roughly 450 digital I/O. No analog, no batch mixing, no PID, just simple photoeyes, limit switches, prox sensors, solenoid valves, ect. The plant engineer in charge of the project continues to make excuses for why the line doesn't work right. This new line has not made it's daily rate even ONE DAY since starting up. I know there is always a startup period for any project to get the bugs worked out but they are not fixing anything. In fact, if you look at the percentage of daily production achieved it has actually been getting worse as time goes on. This production line is a bad joke. I tried to be open minded about a computer controlled system, I am all for change and new technology. We found out during the installation the the company doing the install had never built a production line before. At least now when the line crashes the production people don't call maintenance anymore because they know there is little we can do for them. Friday I heard the start up horn at least a half dozen times because the lines shut down for no reason and no error message, just shut off. Just thought some of you would like to know how it's been going, Thanks again.

A few words but im in the uk so bear with me

I guess the white coats may be to blame here ?

I presume you have not paid in complete for the system that doesnt work yet ?

My customers would not have paid me for this and i would be up a gum tree!

However i only do jobs that work

Now you know what happens when maintenance does not do contractor oversight. Good contractors know this and in a way want maintenance around.

Dan Bentler

A 17 slot rack could have handled up to 512 I/O points (32 points per module) using ControlLogix with the processor in slot 0.

Just curious, how many quotes did you get for this job?

As far as maintanance, the more they get involved the less phone calls I will get later.

I would try to get as much I/O drawings as I could and consider having someone rewrite it that knows whats they are doing. It may not be cheap but considering what you lose each day, it may work out to be a bargain.

PLC companies like Rockwell Siemens and all the rest have 100's if not 1000,s of people working on their control systems to get it right. Smaller companies like Delta would have nowhere near that amount of people but they concentrate on a niche market and I presume that the R&D that they did before releasing a product would have been huge. It amazes me that some companies think they can do the same thing with a PC and a 3 people writing code for a 1 time job.
Regards Alan

This sounds like a lost jobs and law suits waiting to happen

vettedrivr, are these programming errors, Mechanical breakdowns, or I/O failures? Your complaint is rather broad and not specific. However, it does seem like the system error reporting is lacking and that is one of the most important part of the system. A good system will consist of 50% debug code. Our product has much higher percentage than that. I was told by a manager back in 1982 that anybody can make a control system but quality system have good diagnostics and can recover of faults quickly.

There is nothing wrong with C++ it is just a tool. We use C++ and C. However, if is not the right tool if it is overkill and the maintenance people will not eventually take ownership of the system. Some systems can't be turned over because they are just too complex but the control system at vettedrivr's plant sounds trivial by his descriptions. I think 504bloke is right about the management guys picking the system integrators without first checking if they are capable.

512 digital I/O should be easy and on the verge of trivial.
Why make a trivial project difficult with C++?
Making custom I/O cards is silly because there are so many I/O cards that can be bought off the shelf. I see no hint of a need for special custom I/O. We use CompactLogix in our systems for normal I/O. We are perfectly capable of making our own I/O. We do for our own product. Just because you can doesn't mean you should.

Smaller companies like Delta would have nowhere near that amount of people but they concentrate on a niche market and I presume that the R&D that they did before releasing a product would have been huge.

Click to expand...

When you are relatively small you must at least be able to win at a niche that you choose if you want to stay in business. Later there are options to expand the niche but we have tried to remain focused. There is no way a small company can even try to compete with Rockwell or Siemens at every thing they do.

We have been making motion controllers for 25 years now and our personnel turn over is very low. Think about that. I call that tribal knowledge. There is big and there is experienced. We have lots of experience.

Making something without a lot of R&D scares me. I worry about the mechanical parts and wear over time. Yes, Delta Computer Systems really makes industrial computer systems and they require mechanical parts. One doesn't have the time to do mechanical tests for years and years on a system. Everyone wants to go faster and faster and just because you can doesn't mean you should.

All I can say is, Wow! That's exactly why you should leverage specialized hardware (PLCs) and off the shelf software (HMI/SCADA packages). Unless you're Walmart or similar custom apps like that written from the ground up are doomed for failure. Perhaps you have legal recourse?

You don't have a control problem you have a management problem.

Even the best system will not work well if not implemented correctly. You don't need a crystal ball to see what will happen on a project like this. Maybe you should look for new management some where else.

450 IO? That's nothing (although IO count is not really indicative of complexity). My guess is that the contract came after a lot of nice lunches and dinners. In fact, that's how it usually goes.

Unfortunately, you're in a no-win situation. The project manager (or whoever is responsibe for this system) doesn't have a lot of power without making himself look bad. So, he'll downplay the problems unless he's holding a lot of money back.

If I were you, and I really wanted to get it fixed, I'd probably set up my own machine monitoring system. If I didn't have time to sit next to the machine and record downtime, I'd toss a small plc out there and tie into various signals via some opto-isolator relays and at least detect when the machine was running and stopped. Trust me, someone will pay attention when they see that they are losing money.

This isn't a PLC vs C++ issue. It's a matter of a bunch of guys that couldn't write code. Before there were PLCs, there were relays, and before there were relays there were belts, gears and cams. Anyone can make anything work with any tool if they know what they are doing.

You're absolutely right - it's a management issue and an incompetency issue, not a C++ vrs PLC issue. However, I stand by my argument that contractors delivering a finished one time use project like this with compiled custom code is a ridiculous idea. Even with the best programmers, the usability of the application will die with the support contract. It's not that I have anything against C++ or VB inherently - it's that I've seen such applications in industry. Enough to make you cringe!

S7Guy said:

450 IO? That's nothing (although IO count is not really indicative of complexity). My guess is that the contract came after a lot of nice lunches and dinners. In fact, that's how it usually goes.

Unfortunately, you're in a no-win situation. The project manager (or whoever is responsibe for this system) doesn't have a lot of power without making himself look bad. So, he'll downplay the problems unless he's holding a lot of money back.

If I were you, and I really wanted to get it fixed, I'd probably set up my own machine monitoring system. If I didn't have time to sit next to the machine and record downtime, I'd toss a small plc out there and tie into various signals via some opto-isolator relays and at least detect when the machine was running and stopped. Trust me, someone will pay attention when they see that they are losing money.

This isn't a PLC vs C++ issue. It's a matter of a bunch of guys that couldn't write code. Before there were PLCs, there were relays, and before there were relays there were belts, gears and cams. Anyone can make anything work with any tool if they know what they are doing.

Click to expand...

Vettedrivr,
Thanks for the update. I would say that you were right back in 2007: it turned out worse than the Denver Airport baggage handling system.

We found out during the installation the the company doing the install had never built a production line before.

Click to expand...

Oops! Somebody in management and procurement really screwed up, or took a bribe, or was related to the company owner.

To answer some questions,


504bloke - I don't think the final payment has been made yet after questioning one of the junior engineers no longer on the project.

Mark Buskell - How many quotes did we get? 1

Peter Nachtwey - There has not been a mechanical failure or I/O failure yet that I know of. The problems appear to be 1) the programmers didn't take into account the interaction of operators into the system. The contractors had only built sequencer machines to date, box goes in one end, finished product comes out the other end, no operator interaction. 2) the programmers don't fix the problems with the program, they only get into the system and do the equilivent of forcing I/O to get past the problem and then call the incident a "fluke" occurance, the same problems pop up over and over again. I don't have any idea what brand of photoeyes, solenoid valves, prox's they are using, there are no brand name labels on any of them.

S7Guy - I agree it's not a PLC vs C++ issue, but as I stated in my first post, no one here knows visual C++ or even how to access the program to look at it. There is no plans to have anyone trained on the visual C++ language.

CharlesM / Lancie1 - We found out recently that the engineer in charge of the project had a production line installed at a previous employer and it was a mess (still is a mess). It was part of the reason for his dismissal. Some here think he is looking for vindication for that project with this one which is why he is giving this project every opportunity to start working correctly.

I have been watching the production totals since the system started. The percentage of completed units/day has actually decreased as time goes on. It seems that as the programmers add code to gather data, they cause other problems with the line functionallity.

All the propeller heads will ultimately take the blame

This is a very sad example of why control engineers get dumped on. I agree with others...there was payola involved, somewhere. Sadly, reputable firms lose business to clowns like the ones that 'sort of' did this job, all the time. Sadly, its not until the project is a total wreck that anyone finally believes (and even then, may not believe) the other bidders.

Vance Van Doren did an article years ago" "The 10 things you need to know before an automation project.", and it sounds like you broke all 10 of his rules.

CharlesM said:

Even the best system will not work well if not implemented correctly. You don't need a crystal ball to see what will happen on a project like this. Maybe you should look for new management some where else.

Click to expand...

vettedrivr said:

S7Guy - I agree it's not a PLC vs C++ issue, but as I stated in my first post, no one here knows visual C++ or even how to access the program to look at it. There is no plans to have anyone trained on the visual C++ language.

Click to expand...

Ok, let me play devil's advocate for a minute. Let's say they programmed it with the architecture of your choice, but the program was still faulty and caused tons of downtime. Would your plant apply the resources necessary to fix it? Or would they have a "hands off" attitude until the project was accepted and paid for? If I was your manager, I would be afraid that if you guys made changes to the program, the oem would consider themselves in the clear.

So, based on what I've see in other plants, the architecture of choice isn't the problem right now- it's the code. If it was running like a swiss watch, it wouldn't matter if it was controlled by C++ or a bunch of hamsters-driven belts; everyone would be happy.

S7Guy said:

or a bunch of hamsters-driven belts

Click to expand...

Nice......... I'm gonna use that one day