Ethernet/IP communication errors

[kjmason528]

Hi,

I am dealing with a problem with a vendor that has gone unresolved for a long time. We have a slc 5/05 processor communicating over ethernet with a wonderware intouch app that starts and stops feeders on a compounder. The start and stop is done through relay outputs on the PLC to discrete inputs on the feeder. When we start the feeder from the HMI, feeders stop running when they are supposed to be running. The vendor says the problem is with our network corrupting packets and before they can even begin to troubleshoot it we need to fix our network. So my question is this, a packet has a format that should include a checksum. If a packet arrives at the PLC that doesn't match the checksum doesn't it get rejected? Therefore the output state wouldn't be updated until a packet is received that matches the checksum? I expect at least a little bit of error checking. A system that arbitrarily accepts and acts on bad packets would be dangerous.

 

[Dravik]

Wait, what?

There must be PLC logic that activates the Relays.

Unless the packets are crashing the PLC/comms somehow, it is very unlikely that a packet to write 'start or 1' to a 'start bit' in the PLC would instead write '1' to a 'stop bit' unless something is written in a VERY strange fashion.

IPv4 has a checksum function right in the header. This isn't unique to the ethernet i/p comms.

 

[kjmason528]

Exactly my thinking, but this has gone on for over two years. IT says the vendor is nuts. We've had meetings with the vendor and IT and I am at the point where I am going to offer to step in an figure out where the flaw in the logic is and fix it. I am just trying to make sure I'm not wrong. I told my boss last summer that it wasn't a comms issue.

 

[Dravik]

Boy, relay outputs, common power supply?
Does it 'always' happen or only when x # of feeders are running already?

I dunno, I'd check the simple things first(as the vendor) before I jumped to network packet corruption.

 

[kjmason528]

That's a great question. It seems to be random, but it probably isn't. As it happens in most of my experience, the people reporting the problem aren't capable of keeping track of when it happened or what they were doing. We just end up with scrap.

It is a common power supply though. The have an ice cube relay between the Feeder and the PLC the actual input power comes from the feeder controller power supply. I really just think there is something really stupid happening, but nobody will look at it. It could even be the way they set up the start button on the hmi. I would set it up as a latch in the PLC so if the intouch application does something janky, you would have to hit the stop button to get it to quit.

I think you've confirmed what I'm thinking though.

 

[Nova5]

HOW does your HMI Control the PLC Logic? It should just PULSE the start bit, standard latched run should hold it until you hit your stop command. HMI should Hide/Show motor status icons that will pulse the start or stop bits. Does it Set and hold a 1 into the bit?

You are going to need to start trending the things that stop the motor, including anything the operators can do and when the motor running status changes to off that should be the event that stops your trend. THat way the next time it happens you can then go check the trend and see without having to scroll back

 

[Bit_Bucket_07]

Hi,

I am dealing with a problem with a vendor that has gone unresolved for a long time. We have a slc 5/05 processor communicating over ethernet with a wonderware intouch app that starts and stops feeders on a compounder. The start and stop is done through relay outputs on the PLC to discrete inputs on the feeder. When we start the feeder from the HMI, feeders stop running when they are supposed to be running. The vendor says the problem is with our network corrupting packets and before they can even begin to troubleshoot it we need to fix our network. So my question is this, a packet has a format that should include a checksum. If a packet arrives at the PLC that doesn't match the checksum doesn't it get rejected? Therefore the output state wouldn't be updated until a packet is received that matches the checksum? I expect at least a little bit of error checking. A system that arbitrarily accepts and acts on bad packets would be dangerous.

It doesn't sound to me as if your drives are controlled over Ethernet, but by discrete PLC outputs. Why are they interested in the network? I'd be looking for electrical noise and/or voltage drop on the relay output wiring first. You should verify whether the PLC output remains on when the feeders stop. If your relay wiring is a long run, you might want to consider using interposing relays nearby the drives rather than wiring from the drives all the way back to the PLC cabinet.

As for comms, I have seen HMIs (including InTouch) fail to reset momentary pushbuttons sometimes on heavily loaded networks. When start/stop pushbutton bits get stuck, strange behavior can occur. That can be overcome by resetting the PB bits in the PLC program, if it is occurring.

 

[Maxkling]

Is the start and stop strictly controlled by the HMI? Is there a “watchdog” or safe guard to shut the process down if comm loss happens between the HMI and PLC?

 

[Peter Nachtwey]

The Ethernet packets have a checksum. Either you get the whole packet correctly or you don't get it at all.
https://en.wikipedia.org/wiki/Frame_check_sequence

The vendor is full of ....
It is time for a log. That is what extra memory is good for although it is a little hard on a SLC because memory is broken up into chunks of 256. Still that may be enough.

 

[AustralIan]

Just a start and stop right?
I have seen InTouch write an entire byte (or word?) when you only want to write a bit.
This is fine normally because it reads the contents of the byte before writing.
But... in the example of momentary push button, it can occur that the adjacent bits of the pushbutton are changed by the program in that brief time between reading and writing. For example if you have B/0 Start, B/1 Stop, B/2 RunCommand
My solution was to have ONLY write commands from SCADA in the one 32bit memory space.
The supplier has probably seen this problem before and solved it with upgrading the network (reducing time between reading the byte and overwriting the byte)