Tips for newbie to industry?

nyanpasu

Member
Join Date
Aug 2018
Location
Canada
Posts
79
I am currently a second year student (out of 3) in a technical college program called "industrial electronics". The curriculum includes what you'd expect : programming PLCs, power, electricity and electronics, instrumentation and controls, math, physics, CAD, etc.

I am also employed in a coop fashion by a local systems integrator. I work about 25-30 hours during my studies, and 40-60 during breaks. On weekends, I provide support for a client (two day shifts) with the help of my boss (basically, I give a problem a shot, and if I am completely stumped, I can call my boss at any time during my shift for help. This way I am introduced to a whole slew of issues and tasks and I am developing my autonomy). The client, of course, knows about the details of this arrangement.

I have been doing this support for around a month, and while my autonomy has improved, I feel like I'm learning only through prior exposure; I can now do common tasks (change a valve and teach it, fix the random PLC/HMI comms problem, etc) but when I am exposed to situations I haven't seen before, I never know where to start. For instance, today, one process was stuck in a loop. I found the cause (temperature transmitter was reporting a value a few degrees below what it needed to clear the step), but because similar problems in the past had been caused by faulty valves (or closed valves that shouldn't be closed), I looked in the direction of the steam valves, up until an operator showed up and gave a knock on the transmitter itself (apparently this operator had had this problem before), which made it transmit the actual (correct) value (and obviously this means I am changing the probe and transmitter tomorrow, when the process is free). Given that there were other TTs farther down the line who showed the right temperature, in retrospect, maybe I should have clued in, but I was too attached to my prior experiences to do so.

Maybe this is normal, given that I am so fresh, but I'm a bit embarrassed at how bad I am at troubleshooting. I'm relatively handy with a laptop and a PLC so I can nearly always find out why the process isn't moving, but I struggle with finding the root cause and the solution. There was one time where I fixed a problem without any help and I walked away almost high from the pride. I guess I wanna chase that high.

Perhaps some of you would like to share your knowledge. I do well with methods and I think my problem is that I don't know the right questions to ask (a kind of quis, quid, ubi, quibus auxiliis situation I guess) when I come across a problem and that I have no method to apply beyond "look at what's causing the hold up on the PLC". So perhaps a push in that direction would help a lot. Troubleshooting is a skill and it gotta be able to be learned. If it makes any difference, I "get" electronics troubleshooting. I built a PI controller on a breadboard with a dozen op-amps and I had to troubleshoot my share of problems. But I know how op-amps work so it was just a matter of following each input and output from the beginning and poking around when a part didn't do its job correctly. I don't know the processes at the client's plant very well yet so I can't apply the same logic. But I don't think it's optimal to have to know every process like the back of your hand to troubleshoot well.

Another thing I am not so great at, and I found this out at school, is tuning a PID. I've had quite a few labs on the subject but my results seldom came out as what was expected. The tuning constants I get are always way too aggressive. Doing Ziegler-Nichols methods, even when choosing a Ku that isn't even ultimate (oscillations decay slightly over time) because I've had it with getting oscillatory processes and overworked valves, I still get like 2:1 decay. Maybe I take the textbook too literally, but I was expecting to come away with good tuning after applying the "recipe", not results I need to modify afterwards, else what use is the method when I could just go by feel (and in fact, that's how I've been most successful)?

Thanks for any help :)
 
Last edited:
Here are a few tips. Troubleshooting is an acquired skill as you can see. Try starting at the source, or the beginning of the process. Verify that everything is working correctly. Once you reach a point that something has failed repair that then move on to the next step. Take the time needed no matter how urgent that fixing seems to be and take no shortcuts. Stay safe and out of harms way when repairing as things may behave unpredictably. Continue to study every day and READ the manuals.
Most failures can be traced thru following the power flow. At the point of power interruption, most problems can be found and solved.
 
Remember that the longer a machine or process has been running successfully, the lesser the likelihood that any problems are due to errors in the program and the greater the likelihood that they are due to field devices.
When you talk to the operators about the problems they are having, make sure they are describing the true symptoms and not their interpretation of what they are seeing. A problem that shows up at one particular work station may be the result of a malfunction further upstream.
When you think you have figured out the source of the problem, ask yourself if that failure explains all of the symptoms.
If for no other reason than the entertainment value take a look at this thread at MrPLC.com
http://forums.mrplc.com/index.php?/topic/9522-plc-law/
 
You are off to an excellent start in your troubleshooting!

Adding to the advice above,

Getting the information that you require from operations and maintenance staff is *BY FAR* the most difficult part of troubleshooting. Sifting through the opinions and rumors to determine what actually happened is sometimes not worth it. If you can co-opt someone to assist you in reproducing the problem, you will get much more information from seeing it yourself.

If you can't reproduce the problem, you have a much more difficult task.

Don't be too hard on yourself, and your method. There is a reason that you can't buy 'experience'. You just have to work through it, to practice. Listen to everyone, and their opinions, but examine the information critically. Some of it will be wrong. Some of it will have bad assumptions. Observations are normally the most valuable.

If you can work from the beginning to the end and determine if things are working at each step, that is definitely the way to go. Electricians, mechanics, techs ... will all give you a hand with the troubleshooting if you are humble and ask for their opinion. If you ask them to solve your problem, or do your work, the results won't be positive. If you can state the problem that you were told, what your observations are, and what you are stuck with you should get help from the people most familiar with things. Something like 'Operations says that this system normally does this. I figured out that the the PLC is waiting for the pressure over there to change, but that does not seem related to what operations is saying is broken'

It's great to come in and solve problems, but it takes a team to do most things worth doing. It does not need to be a formal team. Talking to several people, getting input, and giving credit when the problem is solved will get you a long way toward honing your troubleshooting skills
 
The thing that helps me most night in and night out is to approach every problem believing that it's all lies. If you ask the operator if they changed anything before it stopped working, they are going to lie and say no. If you ask if anything unusual happened before it stopped working and you are told no, it's probably a lie. If the temperature is supposed to be X and the transmitter shows Y, then there is a fair chance the transmitter is lying or maybe the transmitter is being honest and the cable has gone bad or gotten damaged, so the PLC itself is not telling you the truth. The trick is to figure out who or what is lying to you this time.
On well established system that have been running for years, then you will most likely find that a field device or the operator is the cause of your trouble. With that being the case what I do is take things one step at a time. Check the last place things are good, and then move forward one step and check again. In your example, the TT showed X, but you expected Y, and you assumed that the TT was not lying, but until you prove the TT is being honest then you shouldn't have moved on. Of course even if you find the device that is lying, you are not out of the woods yet. Just wait until you change a proven bad device only to find out later the the new "honest" device was bad brand new out of the box. That little wrinkle will drive you all kinds of crazy.
The most promising thing is that you recognise and acknowledge your shortcoming , and you learned from it. That is experience in action right there, and you shouldn't beat yourself up over your lack of it. After 30 years I'm still learning new things everyday, and I still have days that I can't see the tree in the forest. Just keep doing what you are doing, and if you are like most people, after 20+ years you will still feel like there is more that you don't know than you do. That is what makes controls and automation fun, the process is always getter smarter at a faster rate than you are, so you will never run out of things to learn.

Bubba.
 
The thing that helps me most night in and night out is to approach every problem believing that it's all lies. If you ask the operator if they changed anything before it stopped working, they are going to lie and say no. If you ask if anything unusual happened before it stopped working and you are told no, it's probably a lie. If the temperature is supposed to be X and the transmitter shows Y, then there is a fair chance the transmitter is lying or maybe the transmitter is being honest and the cable has gone bad or gotten damaged, so the PLC itself is not telling you the truth. The trick is to figure out who or what is lying to you this time.
On well established system that have been running for years, then you will most likely find that a field device or the operator is the cause of your trouble. With that being the case what I do is take things one step at a time. Check the last place things are good, and then move forward one step and check again. In your example, the TT showed X, but you expected Y, and you assumed that the TT was not lying, but until you prove the TT is being honest then you shouldn't have moved on. Of course even if you find the device that is lying, you are not out of the woods yet. Just wait until you change a proven bad device only to find out later the the new "honest" device was bad brand new out of the box. That little wrinkle will drive you all kinds of crazy.
The most promising thing is that you recognise and acknowledge your shortcoming , and you learned from it. That is experience in action right there, and you shouldn't beat yourself up over your lack of it. After 30 years I'm still learning new things everyday, and I still have days that I can't see the tree in the forest. Just keep doing what you are doing, and if you are like most people, after 20+ years you will still feel like there is more that you don't know than you do. That is what makes controls and automation fun, the process is always getter smarter at a faster rate than you are, so you will never run out of things to learn.

Bubba.

This really talks to me.

I was support during a power outage, and everything had failed one way or another, so I was running around trying to reinitialize/restart everything. One operator calls me to tell me one of her pumps isn't working. So I go there, ask her to point me to the failed pump (I knew next to nothing about the plant at this point), and she points me to that one pump, I follow the wires to the drive and the drive seems...functional? I'm a bit perplexed at this point, so I ask her to try again so I can see what it does exactly, and sure enough, that drive starts up but the process shuts down due to a drive failure to run.

The number on the alarm is not the number of the pump she pointed me to, but for some reason, I kept looking at that pump. My rationale was that I was some young newbie who didn't know much about the plant and even less about this part, that she had been operating this process for a while, possibly years and that the numbering system on this particular process is a bit askew, so I took her at her word and trusted her. It took 30 more minutes and I actually called my boss and had him come to help me out because I was so confused. All he did was ask her if she was "really sure" that this was the right pump and instantly, I saw the doubt in her eyes. We had a look at the P&ID and sure enough, it was another pump and the drive was located away from the eyes in another room (so I didn't walk on a random faulted drive). My boss stayed the morning to teach me the meaning of "trust but verify" and share related war stories. I have a feeling he knew all along but he came anyway to teach me a lesson. Well learned, LOL.

I remembered your post today. I had a "broken" photoswitch. Turns out it was working but probably the machine running for a few weeks moved the part on which it was sitting/what it was "looking" at by an order of millimetres and that was far enough away to keep it from detecting what was in front of it. I was going on the assumption that it was broken, but I checked whether it was lying or telling the truth before changing the photoswitch. And well, sure enough. Just had to move the bracket forward a bit.

You are off to an excellent start in your troubleshooting!

Adding to the advice above,

Getting the information that you require from operations and maintenance staff is *BY FAR* the most difficult part of troubleshooting. Sifting through the opinions and rumors to determine what actually happened is sometimes not worth it. If you can co-opt someone to assist you in reproducing the problem, you will get much more information from seeing it yourself.

If you can't reproduce the problem, you have a much more difficult task.

Don't be too hard on yourself, and your method. There is a reason that you can't buy 'experience'. You just have to work through it, to practice. Listen to everyone, and their opinions, but examine the information critically. Some of it will be wrong. Some of it will have bad assumptions. Observations are normally the most valuable.

If you can work from the beginning to the end and determine if things are working at each step, that is definitely the way to go. Electricians, mechanics, techs ... will all give you a hand with the troubleshooting if you are humble and ask for their opinion. If you ask them to solve your problem, or do your work, the results won't be positive. If you can state the problem that you were told, what your observations are, and what you are stuck with you should get help from the people most familiar with things. Something like 'Operations says that this system normally does this. I figured out that the the PLC is waiting for the pressure over there to change, but that does not seem related to what operations is saying is broken'

It's great to come in and solve problems, but it takes a team to do most things worth doing. It does not need to be a formal team. Talking to several people, getting input, and giving credit when the problem is solved will get you a long way toward honing your troubleshooting skills

Thanks :)

I have had mixed experiences with the operators. Sometimes I feel like some of them just tell me **** to give me the run around and watch me lose my sanity. I'm not the most socially aware/straight up social guy either so I tend to take people at their word and I also tend to assume that they are competent. When I started that gave me a huge deal of trouble. I'm a bit better now, got that "trust but verify" mindset.

Speaking of replication, there's something a bit nutty that keeps happening to me : whenever I ask the operator to do what he/she was doing again in order to replicate the problem, things end up working. It's gotten to the point that people joke my mere presence scares the processes straight. I'm not sure what to think of that, because there is this one particular problem that has been occurring for two weekends straight, but whenever I want to see it happen, it just...doesn't, and all I get from the operator is "xyz isn't working".
 
Last edited:
Coming from prior experience, you have the right mindset! About PID tuning... It's more of an art than an engineering topic. A lot of loops you just have to use trial and error.

For troubleshooting? When following root cause analysis (RCA), you have to know the process to begin with. In your spare time, I would recommend taking to an operator and getting walked through the process. While getting walked through, examine any sensors that are used to activate the next sequence. 80% of your problems will be mechanical failures. The rest will be divided among electrical, operator error, parameter settings, and so on. So always ask yourself where the machine is at in the sequence it's supposed to be in, what are the next steps, and what conditions need to be met before the next step can be executed? You'll never master troubleshooting. You'll only become more proficient. And once you do, it'll be a huge skill that employers look for.

You have to be careful, as some issues go deeper, even though they may seem obvious. One age old scenario that I've seen happen in real life is you have a hydraulic pump. The breaker to the pump keeps tripping. It'll run a few minutes and trip again. While the motor looks like it's bad, or a bearing is causing friction and high current draw, the real answer is just a dirty filter causing cavitation and drawing too much current on the load.
 
I would second some of the comments above...

we used Kepner Tregoe many years ago as a formal training course to solving problems, with the link below giving you ideas as how to go about it:

https://www.kepner-tregoe.com/tasks/render/file/?fileID=1BB3A74A-9F9F-386B-32BCC39F1FF51555

It has stood me in good stead over the years, and although I still have the book at home, the procedures have become second nature. You do not always get it right, but if you follow a logical process, you will get there. Some problems defy logic, however................

Never ever believe fully what the operator tells you. Prove it or see it for yourself, if you can. 5 shift teams may well give you 5 different stories.

PID loops are never easy. Trial and error, with that one, unfortunately. PID tuning software may assist, but not always. Change one parameter at a time, and monitor/test.

If you are such a scary influence that when you get there, things work, then you need to stay there longer....so that the operators do not do the thing they were really doing, and break the plant again...
 
It's gotten to the point that people joke my mere presence scares the processes straight. I'm not sure what to think of that, because there is this one particular problem that has been occurring for two weekends straight, but whenever I want to see it happen, it just...doesn't, and all I get from the operator is "xyz isn't working".

30 years in the trade and I have had this happen several times. A couple of the times it was the operator, operating the machine correctly whilst I was standing there, but when I left they tried to take shortcuts, creating problems.

Not much else to add to the above great advice, except when troubleshooting, change one thing at a time, check for a resolution if not move on. Too many times I've had too many fingers changing stuff (and not saying sometimes), that when a solution occurs you are not sure what fixed it and production don't want to shut back down to figure it out.
 
Troubleshooting gets easier with experience. Generally speaking though, prove the entire system out, don't make assumptions. Given time you'll get enough experience to make some assumptions, but even then you'll get bit in the butt sometimes. Wring out the whole loop.

As far as tuning goes, you are taking the text books too literally. The text books are written for systems operating perfectly, in a vacuum, with no outside influence. You have to remember you are tuning a valve that's been in operation for years on end that have seen many operation cycles, seen many pressure and temperature swings, etc. Valve seats get tight or sticky, pneumatic operators gets worn, sometimes a 4-20 output to the operator isn't truly linear so you'll get bulges or dips midrange. Sometimes the wiring between the PLC and the device are very long and you'll have loss (IE the PLC outputs 12 milliamps, but the operator only gets 11). Text books are written for perfect conditions however the world is a very imperfect place. Tuning loops is as much an art as it is a science. Just as with troubleshooting, given time you'll get the "feel".
 
As others have said, troubleshooting is a learned art.

Personally, I do these things:

First thing I ask is whether the machine has been running correctly up to the point of diagnosis or if it was doing anything different before it started to fail. If it was running ok prior to you diagnosing, you can assume there is no need for any programming change.

The worst thing I see new people do is start changing any code. That should always be a last resort.

Electrically speaking, always check the power sources. I'd say 90% or more of the problems have to do with a power issue in my experience.

Make sure your main source of power is coming in. This is usually pretty easy to tell. If you have safety zones, verify that you can start/stop the different zones or see an indicator that they are on / working. Verify that sensors, drives, valves, etc are being powered on and have some sort of indicator light.

When you can verify all power sources, then you move on to checking to see if things are working. See if a switch toggles on/off like it is supposed to. Check values of analog inputs.

As an electrical person, I can't tell you how many times I get called out for a program issue that just turns out to be something physical (like the operator leaving the coffee mug in the light curtain or a sensor reading correctly that a valve is plugged up and no flow is present, etc)
 
Something you should know is that troubleshooting like any skill takes time to learn. After I finished school I would say it took me about two years to be a good tech. I have also improved my skills since then and always will. Don't beat yourself up when you haven't finished school and have a month of work related experience. The fact that you desire to learn is enough to make you great at your job in the future. The tips I have for you is when solving problems don't think of the big picture(the whole machine) just think of the problem at handle, you do not need to know or learn the whole machine to fix it, just the circuit or part that is retaining to your problem. If you have a E-stop problem you don't need to know anything beside what effects that circuit. I know that sounds simple but I watch young techs get overwhelmed because of that type of thinking. I was the same way. The other thing I would recommend is when troubleshooting think KISS- Kept It Stupid Simple. If a machine doesn't run start with the simple things first. Do I have Power, Is there a tripped Breaker, is the E-Stop circuit made, is the start or stop button(s) making.

Best of Luck,
 
1. Assumptions kill you.

Not literally, but when you assume that X is functional, it might well not really be the case.

2. Write stuff down.

I can't believe how many times I've been stuck but when reviewed my written notes, I realized I'd skipped something or made an assumption that I hadn't tested.

It also amazes me how many people have issues with analog stuff, post a thread, but fail to ever mention actual values, like "it's low 15% at 24psi." Sometimes the numbers point to one cause or another. Without numbers, what's "low"?

3. Bill Mostia authors a book, Troubleshooting: A Technician's Guide, 2nd Edition, publisher is ISA. (Barnes & Noble $24. I get nothing, darn it).

It's not something you read in the middle of a crisis, it's a text you digest over time that provides strategies and tactics for troubleshooting. He mentions root cause analysis.
 
I would say that you should try to be the best operator in the plant... time and time again, knowing the available functions of the machine and what are reasonable (or correct) settings is a life saver.

I've worked in an industry where training was light and when it existed was very, very poorly timed (as in train operators, but only let them use equipment half a year or a year later), and most of the time knowing how a machine work lets you immediately understand if some change to the settings was done to it, and then understand roughly what sensor or actuator may be giving you a headache.
 

Similar Topics

Imagine a heat exchanger cooling down city water to make ice water and send it all around the factory. On the loop are multiple loads that may or...
Replies
3
Views
1,566
I’m currently starting a brand new design with a compactLogix controller and a panel view 700. I’ve completed similar projects from start to...
Replies
7
Views
1,214
As I'm getting up to speed on some of the latest versions of TIA Portal, I noticed there is lots of security features. I'm wondering if anyone...
Replies
2
Views
1,051
and go! I'll start. Always comment the Boolean instruction for their TRUE state. For example. It is much easier to read a normally closed contact...
Replies
65
Views
21,402
I have been programming plcs and hmi's since the 90's. I would like to think that I have mastered my field/trade, but I know that I have not even...
Replies
0
Views
841
Back
Top Bottom