ControlLogix L55 and Ignorance Problem

Good morning Gentlemen,

This is my first post and I need some serious advice.

Firstly I want to say I am not a PLC programmer nor do I wish to ever be one, I am a process Engineer that works for a small Systems Integration company in Ireland.

I got assigned to commissioning a water treament system in a Pharma plant and I've managed to make a really big mistake.

As part of a validation test I was required to shut down the PLC to demonstrate a system shutdown. THe test worked but what I didnt realise was that when the PLC was in "Program" mode it retained its outputs. I somehow left a steam valve opened over night which managed to cause a 15,000L tank to crack resulting in an employee inflicting serious scaulding. Now my boss says its my fault and I better have a good excuse.

The guy who wrote this program has since left the company and I cannot contact him.I asked a guy who works in Automation to have a look at his program and he said its awful, no structure to it and the entire program written in a single routine with no comments and over 100 warnings. Is this my fault? Was the PLC faulty or was it the steam valve? Where can i find documentation on this PLC?

There are quite a number of unfortunate events that are contributing to the results:

1) You are admittedly outside of your experience and skill set. Unfortunately, that's your boss's fault -- you should not have been assigned this commissioning task. Your skills are probably required in this commissioning, but not for this particular task.

2) The person who should have been involved in this particular task is no longer with the company.

3) The program was poorly done. Without a copy of the program, none of us can really comment on this beyond the comments of your friend. However, there's nothing technically wrong with putting all of your code in a single routine, with no comments. Properly written, though undocumented, code will work properly. I like to explain it that breaking your program into multiple routines just gives "chapters" to your "book," or it "catalogs" your program. But a single routine does NOT make your program work better or worse than multiple routines.


That there are over 100 warnings ~may~ be cause for some alarm. Without knowing those warnings, we really cannot comment any further on them. Certainly, those warning may provide clues.

I doubt the PLC itself was faulty. It did exactly as it had been told in the program.

Was the valve faulty? I don't know... Is it the right valve for the application??

For your validation test, did you turn the processor OFF (i.e., remove power) or just put it into program mode? Was the processor in program mode over night while the steam valve was open? Are there any other manual controls that could have cause the valve to be open?

I'm NOT aware that a ControlLogix processor retains its outputs while in program mode. Whenever I've put one in program mode, the outputs have always gone to their "OFF" states. Again, we would have to know some more about your hardware configuration and program to be able to comment on that. It seems like there ~may~ be some things that can be set to "retain last state", but that's NOT the default configuration and I've never used it...

As far as documentation, that can be found here. You may have to register on the website to get some of what you're seeking, but registration is free. Likewise, you should be able to contact your local Allen-Bradley supplier. They would be able to point you to better contacts if they can't answer your questions directly.

I somehow left a steam valve opened over night which managed to cause a 15,000L tank to crack resulting in an employee inflicting serious scaulding. Now my boss says its my fault and I better have a good excuse.

Click to expand...

Regardless of the programming practices implemented, I'd question the design of the safety system to allow something like this to happen.

If in fact this controller is using 1756 ControlLogix IO, then for Analog outputs, there is a configuration tab that defines how the output is to operate when in program mode and when the controller faults.

I dont think you can be blamed at all.

Without knowing this particular field of application, I would guess that there should be an overpressure relief system and/or the pressurised vessel shall be dimensioned to withstand the pressure with a considerable margin. The valve could have failed on or off for a number of reason, regardles of which reason it should not cause a dangerous situation.

We use pressure tanks in several applications, and there is always a certificate with the tank stating that the tank may be used up to a certain pressure.

I would call you an "operator" in the respect that someone else told you to do what you did (shutdown of the PLC), and that you did not have the insight into what would happen.
An "operator" cannot be blamed, unless he knowingly ignores safety instructions.

There shall be a socalled "technical file" for the plant, and part of this is the socalled "risk analysis".
There shall also be an instructions for use wherein all important safety recommendations shall be listed.

I dont know what happens in Ireland in such cases, but if the accidence is serious enough, in most countries the police will show up. And they will routinely ask for access to the technical file and the safety instructions.

So the questions are these:
Is there a technical file with a risk analysis ?
If so, what does it say about the affected system that failed ?
Is there an instruction for use ?
If so, what does it say about the affected system that failed ?

If there is no risk analysis, or there is no instructions warning against what you did, then the original designer neglected this important part of his job.
It doesnt matter that the guy does not work there any more. Depending on the country, the person that designed the system is to blame, or the company that was responsible for making the system is to blame.

This is a very unfortunate incident and I really feel like a fool. A person has sustained horrific injuries and I dont have any answers. I called my boss and he called me a **** a-hole.

I am working as a process engineer for this company now 2 years. I have a BSc in mechanical engnieering and since I have been working for this company I have been doing nothing more than documentation.

As a collegue of mine left I was asked to commission these water treatment skids. I have no knowledge of PLCs or what they are even about. Now I have been asked to change parameters, timers etc and I dont even have the software that is required to download the program and SCADA to the PLC. When I open iFIX to see the PLC program it fails.

Is it poissible to modify the PLC program without iFIX?

I think you should keep it in separate threads. One for the safety issues, and the other for how to use the PLC and SCADA programs.

Just to answer your last questions.

iFix is a SCADA program, not a PLC programming software.

About "I have been asked to change parameters, timers etc":
With iFix you can modify values in the PLC, depending on that the values have been programmed into the iFix interface. So yes, you can modify parameters and timers etc. if they are available in the SCADA already. If not, a PLC/SCADA programmer is needed. Dont fumble about this yourself if you are not proficient in either.

About "When I open iFIX to see the PLC program it fails.":
Are you saying that the IFix SCADA does not connect to the PLC, or what ?

Anyway, your boss sounds like a psycopath or just incompetent. I would not be bullied like that if I were you.

Hi Jesper

I was download I was able to download to the PLC via iFIX. Sorry if that seems stupid. My boss is a physco and if I tell him I cant do it I fear he will sack me or even hit me. Its happened before. How long would it take me to figure out PLC programming? Is their days involved? Weeks even?

The best thing to do is get someone in there who knows about PLCs to straighten this out then look at learning about PLCs another day

After thinking about this a bit, you have a professional obligation to only operate in your area of expertise. Since your boss has asked you to do something that you have no training or knowledge of, then you should decline citing those deficiency in knowledge. He should respect this and find another subject matter expert to execute this testing.

Your boss is on the hook and attempting to assign or pass blame. You should seriously consider your future with this company.

Another thing - there should be some type of validation documentation for this system. What are the requirements and design. How did the programmer know how to program this? What are the design constraints of the system? Should there have been a mechanical protection system (i.e. rupture disks on the tank, etc)? Was the system designed by a professional engineer at an engineering firm? Who is the responsible engineer? Was the design approved?

This whole scenario is a classic example of a poorly designed system. I would guess that there is going to be litigation over this.

CountyLouthCowboys said:

My boss is a physco and if I tell him I cant do it I fear he will sack me or even hit me. Its happened before.

Click to expand...

I would go see the boss, crack him over the head with some sort of large implement and staple my resignation to his forehead! Then find a decent employer.

Hi "County.."

It sounds so bad that I would look for another job. Seriously. Come to Denmark, we need engineers ! And we have beer too

Anyway. Are you member of a union ? They may help you with advice in cases like this.
Is there a safety-group in your plant ? Someone must be appointed to coordinate safety tasks. At least that is how it is over here.

As for learning PLCs, I would estimate it like this:
For the most basic knowledge that would let you poke around with a small home project - 2 to 4 weeks.
For a "real" project with a small simple machine - 2 to 3 months.
For a "real" project with a complex machine, but without advanced functions - 1 year.
For more advanced functions, and for safety related functions, 2 to 3 years.

Guys,

I have called my boss and he told me its my problem and I need to deal with it. He said there is no one else available to do the job because all our staff have left.

Now what is need to actually a program called ControlLogix 500 or 5000 and not iFIX. Is this available for download? Is it possible for me to look at the program in MS Word or Excel until I get the software somewhere?

first of all, this MIGHT be on the up-and-up ... but before we go any further, let’s consider these selected quotes from CountyLouthCowboys:

Jeebs and Paulus. I know where you live. When I get you im going to rape your mother...Then im going to dig up your Grandfathers and make love to their skull.. Anyone else any suggestions on my problem????



If you guys dont reply to this soon I am going to get you all thrown off this site and then I'm going to call the cops and have you arrested...



Thats it. I have called the cops and this site is being terminated...

Click to expand...

all of this and more can be found in this thread - for anyone who is interested ...



anyway ... let’s move on and try to at least turn this into a valid learning exercise:



one thing that has NOT been mentioned yet, is a topic that I try to hammer into the noggins of all of my ControlLogix students (both programmers AND technicians) and it's related to the following screen shot ...

with ControlLogix it IS possible (for better or for worse) to configure a discrete/digital output to “act weird” when the processor either “goes into program mode” or when it “faults” ... since this does not usually happen with other Allen-Bradley systems, I consider it EXTREMELY important to make people aware of it ...


I’ve heard rumors of situations where these settings have been “messed with” by inexperienced programmers who just wanted to “see what they do” ... a few innocent mouse clicks here and there - but they then forgot about changing the settings back ... the processor was eventually placed in “program mode” to turn the machinery OFF - with some surprising results when some of the outputs actually came ON instead ...

Wow

Reading this thread that is the first thing that comes to mind. WOW!!!!!!!!!!

First: State of a PLC should not ever,ever, did I say ever cause a pressure vessle to crack. All pressure vessles have to have a pressure relief valve on them. Heck the hot water heater in your house has to have one. Safety is almost always about redundancy. If system A fails then System B keeps things from going boom. If system B fails then system A better be able to keep things from going boom. If a really big boom is possible then system A Sytem B and even SYtem D, E, and F may be required.
Second: A mechnical engineer should never comision a PLC. Nothing personal but you are not qualified. You should have worked with a Programer to confirm the valves you are trying to control are actually being controled as specified. IE: This should have been tested BEFORE pressure was introduced.
Third: Maybe it is a difference between here and over there but my boss cusing me is not allowed period. If for no other reason he will not have any teeth left in his mouth after he is done. Simple rule I am not allowed to cuss him he is not allowed to cuss me. Heck he is a freakin idiot for sending you on this comissioning anyway. Where the heck was he at when you were changing modes on the PLC?

My suggestion: Document everything that happened to as much detail as you can. You may need this when the lawsuits start. Then get the heck out of there. If something as fundamentaly obvious as a pressure relief valve being missed is what happens on equipment this company comissions I would be a nervous wreck thinking of the what else could have been over looked.

This sounds like something out of the 1950's. Heck even then they knew about pressure relief valves.