Logix5561 HYD02 953D faulting after 45 mins

First, up front apologies for my lack of specific PLC knowledge! I hope I can provide the right detail to get the right assistance!
I am responsible(!) for a new machine that has the above components providing control for the movement of 2 hydraulic cylinders - and quite a bit more.
We have a fault that only occurs after 30 minutes to one hour of operation of the 2 cylinders. Up to that time the machine works perfectly.
The first fault shows up as the transducer position 'flickering' to a position equal to maximum extension. The HYD02 card 'faults' and can be reset. The machine will then operate for a short time until the same 'maximum extension' fault occurs. This will now stay in this state for say 20 seconds (it does vary) and then return to the actual value. We can then restart but fault occurs randomly but more frequently and for longer.
If we stop the hydraulic motion for a few hours we can again get 45 minutes operation before a fault.
We tried to isolate VSD's and other 'noise creators', separated cables from LVDTs, common grounds for separate Power supplies etc. We cannot find anything that appears to start up at the same time as the fault appears.
We have had all sorts of specialists looking at this. The last AB serviceperson said 'replace the HYD02 card'. We did this, fault is still there with same frequency!
Any ideas?

The symptoms you describe, would send me looking for a temperature related problem, that has a close proximity to the oil.

As time passes the hydraulic oil warms, the oil thins, the cylinder moves more freely, end points shift, the sensor on the cylinder has a heat sensitive component, solder joint expands,

I don't know the component you are referring to so I am just throwing up my thoughts for what I would have looked at with a side ways glance.

Get a Scope. Without a scope we are just guessing/

SimpleCAD said:

We have a fault that only occurs after 30 minutes to one hour of operation of the 2 cylinders. Up to that time the machine works perfectly.
The first fault shows up as the transducer position 'flickering' to a position equal to maximum extension. The HYD02 card 'faults' and can be reset.

Click to expand...

You must look at the signal coming back from the Temposonic or Balluff rod. Make sure you have a regulated power supply for the MDT rods.
The HYD02 counts the time between the start and stop pulses coming from the MDT rod. If something happens to them then the HYD02 reports an error but it is not the cause of the error.

The machine will then operate for a short time until the same 'maximum extension' fault occurs. This will now stay in this state for say 20 seconds (it does vary) and then return to the actual value. We can then restart but fault occurs randomly but more frequently and for longer.
If we stop the hydraulic motion for a few hours we can again get 45 minutes operation before a fault.

Click to expand...

This does seem temperature related but again you must get a scope. The problem is that there are a couple different types of errors that you can get. One is that the magnet on the end of the piston is extended too far past the effective range of the MDT rod. In this case the HYD02 will not get return pulses and fault. The other kind of error is from noise. In this case the noise looks like a stop pulse and the HYD02 card stops counting. From one you describe the former is most likely.

We have had all sorts of specialists looking at this. The last AB serviceperson said 'replace the HYD02 card'. We did this, fault is still there with same frequency!
Any ideas?

Click to expand...

Yes, get a scope. Replacing the HYD02 card will not fix a feed back problem. Have you swapped out a Balluff rod for a Temposonic rod or the other way around. The magnets are not always compatible and sometimes they will not return a good return pulse when the rod is extended.

Have a second MDT rod ready to connect to the HYD02 when the systems faults. I bet the second MDT rod works just fine.

Thanks for all feedback.
I have added some more detail that may present a 'clearer picture' of our problem

Peter Nachtwey said:

You must look at the signal coming back from the Temposonic or Balluff rod. Make sure you have a regulated power supply for the MDT rods.
The HYD02 counts the time between the start and stop pulses coming from the MDT rod. If something happens to them then the HYD02 reports an error but it is not the cause of the error.

Click to expand...

We have arranged for a technician with scope for tomorrow morning.

To add more detail on what we have done to date;
We have removed LVDT from cylinder and checked magnet position and also manually moved a magnet up and down the shaft - everything works perfectly. (We could not recreate the 'null position' fault by moving magnet up and down. Although we could not operate cylinder with encoder removed)

We have 2 encoders and cylinders, and we have had the error on both, although it happens much more frequently (about 1 in 5) on one encoder.
The Gemco LVDT has a 'green LED' to indicate correct operation (magnet in position and communication with card). This LED is constantly on - even when we have the fault.

Peter Nachtwey said:

This does seem temperature related but again you must get a scope. The problem is that there are a couple different types of errors that you can get. One is that the magnet on the end of the piston is extended too far past the effective range of the MDT rod. In this case the HYD02 will not get return pulses and fault.

Click to expand...

We have checked encoder shaft length and this 'null position' is at a point on the shaft that the hydraulic piston cannot physically reach unless disassembled from the machine. The 'errors' occur anywhere along the cylinder travel and the readout we get is always the same ('null position' - maximum value for encoder)

Peter Nachtwey said:

The other kind of error is from noise. In this case the noise looks like a stop pulse and the HYD02 card stops counting. From one you describe the former is most likely.

Click to expand...

With the extra information I have provided, is the latter now more likely - or do we need to look at getting another encoder?

Peter Nachtwey said:

Yes, get a scope. Replacing the HYD02 card will not fix a feed back problem. Have you swapped out a Balluff rod for a Temposonic rod or the other way around. The magnets are not always compatible and sometimes they will not return a good return pulse when the rod is extended.
Have a second MDT rod ready to connect to the HYD02 when the systems faults. I bet the second MDT rod works just fine.

Click to expand...

As noted above we have the same problem with both rods - although more frequent with one.
The rods are 50" long and there are no spares in Australia that we know of!

Our feeling here is that we have a 'noise problem' but one that is somehow connected with temperature? I could write a book on what we have tried to date (some points follow but please ask if more detail would help.)

Fitted dedicated, regulated power supply for encoders (all 24V supplies have common grounds) (Note: the problem 'may' be less frequent since we did this - there did appear to be a correlation between a specific 24V solenoid operation and the fault, this has now gone. This solenoid 'typically' operates about 30 min after start up.)
Replaced cable to encoder, (this runs into a junction box - only thermocouple and low voltage in box) made new connections and new cables outside cabinet (away from interference) direct to card. We ran the 24V supply around the other side of the machine to the signals.
We have 2 VSD's in system - we turned these off and were able to create the fault with them off.
We have 5, 3 ph heaters and pumps/fans (Max Total 24kW - generally much less - all wiring well away from encoders and card) running at the same time as the metering units. We have checked contactor on/off for each of these units and see no correlation between these and faults. (These units run virtually continuously and contactors have come on/off many times before the fault appears.

We look forward to checking with scope - any ideas on what noise could 'appear' after 45 mins?
(I hope you don't think we need to get new encoders!)

Again, appreciate the assistance!

Gil47 said:

The symptoms you describe, would send me looking for a temperature related problem, that has a close proximity to the oil.

As time passes the hydraulic oil warms, the oil thins, the cylinder moves more freely, end points shift, the sensor on the cylinder has a heat sensitive component, solder joint expands,

I don't know the component you are referring to so I am just throwing up my thoughts for what I would have looked at with a side ways glance.

Click to expand...

Thanks Gil,
The encoder is on top of a 50" cylinder in a well ventilated area. It doesn't appear to change in temperature very much whether we are operating cylinders or not - we have checked this by placing our hand on the housing. I should say that there is oil heating nearby so the encoder is always 'warm' - say 25-30 deg Celsius. Their rated max. operating temp is 85 deg Celsius. The cylinders only move slowly - say 4" in 20 seconds - and the hydraulic oil temperature is about 35-40 deg C (The servo valve gets quite warm but this is isolated from the encoder.)

We have 2 encoders in the system and they both have provided the same fault - so I hope it is not a cracked PCB or I have 2 50" long units to replace and they are not in stock in Australia!

We are speculating until tomorrow when the scope comes.

SimpleCAD said:

We have arranged for a technician with scope for tomorrow morning.

Click to expand...

We are just speculating till then but I can tell you what to look for.

Assuming you have the MDT rods set for start/stop. There should be three pulses for every feedback update. There will be a an interrogation pulses from the HYD02 to start the process. That is the pulse you want to sync to scope to. There should be two return pulses. One is an immediate echo that follows shortly after the interrogation pulses. The second return pulse is the stop pulse. The HYD02 starts its counter when it receives the interrogation echo and stops the counter when it receives the stop pulse. If the magnet isn't moving it should be very easy to sync up and see all three pulses. Once you do that you should be able to move the magnet back and forth and see the the stop pulse change position back and forth on the scope. The interrogation echo should be stationary since its delay is just a function of the length of wire between the HYD02 and the MDT. BTW, this are MDT rods which stands for magnetostrictive displacement transducer. Rockwell calls them LDTs rod for linear displacement transducer and all though the rods are linear it doesn't describe the interface.

From what you have described the rod seems to think it is receiving and sending the pulses OK. The question is whether the HYD02 is receiving the stop pulse. The stop pulse must be of an amplitude that can trip a threshold and stop the counters. If not the counters will keep counting and the state machine in the FPGA will generate an error if the second pulse does come before it needs to send the interrogation pulse for the next update.

I doubt this is caused by noise.
BTW, Balluff and Temposonic makes MDT rods too.
I would look for one of them but the problem is the magnets. Some have the north poles pointed towards the rod and other have the south pole pointed towards the rod. I know the magnets are not always compatible between the different rods and that is is not fun to replace the magnets.

My suspicion, the electronic on the Gemco rod are getting warm and are not sending a strong enough stop pulse. Check the amplitude of the return pulse over time. You are entering summer time there and it must be getting hot. The oil gets hot when running. The electronics must be very conservatively designed when A M B I E N T (The forum thinks that is a nasty word for some reason ) temperature is 110-130 degrees F ( oil temperature and the air temperature is 100 degrees F. The electronics inside will be hotter yet.

We have cans of 'cold' here where we can spray them on the circuit boards to thermally shock them. Do you have anything like that there so you can spray the MDT electronics and cool them off? It would be an interesting experiment.

Scope says electronics don't like heat

Thanks Gil and Peter,
Heat is certainly the issue!
I made the mistake of believing that the Gemco Specified 'Operating Temperature' (-40 to 85 deg C) was the 'Working Range'. We now know that when the Gemco electronics case reaches about 35 deg C we can get a fault.
Peter we are using a 953D-VP with a PWM signal. When we connected the scope we saw one 'strong' pulse and another 'faint' pulse that was 'shorter'. The strong pulse would vary in length depending upon the encoder magnet position, the short pulse appeared to be the same length all the time. When we started our tests the encoder case was at 25 deg C. As we operated the cylinders the servo valves get hot and the heat is transferred to the encoder cards. After about 45 minutes the case has reached a temperature of about 35 deg C. As the case gets hotter the 'faint' signal becomes much stronger and we then get the incorrect encoder position and drive fault. At this time we then applied the cold spray to the encoder electronics case and we could see the 'short' (originally faint) pulse become weaker. When we cooled this case to less than 15 deg C the short pulse disappeared.
As soon as this signal became faint the encoder reading would return to actual.
Just to prove a point we lightly applied heat to the enclosure case and could recreate the problem.

So the problem is quite clear, our 'simple solution' is to apply some air cooling to the encoder cards (we are planning to drill 2 small holes and feed in some (dry) compressed air whist we run. (I should say that it is not very hot at the moment in Melbourne so these encoders haven't seen any 'real heat' as yet!)

We were wondering if there is any way of filtering out this other signal, or if there is a reason the electronics are so sensitive to heat? (We have 2 of these encoders and they both have the same problem.)

Certainly appreciate the assistance - we can finally get on to the challenges of production.

That is terrible.

There are other companies that make MDT rods. Companies like Temposonic and Balluff make rods that will operate in much higher temperatures.

Using start stop instead of PWM will help. It takes extra electronics to convert a start stop signal to a PWM signal. Perhaps this is the electronics you are referring to.