Generic Ethernet Timeout

MattFromGA

Member
Join Date
Dec 2012
Location
Georgia
Posts
7
I'm having problems with a connection timeout after changing the device connection properties in RSLogix. I have to power cycle the device to get them to connect again. The problem is very inconsistent. Sometimes I can change connection properties for 5 or 10 minutes without it timing out. Sceencap vid - https://www.youtube.com/watch?v=JkiqSPgazws

Device - Avery Weigh Tronix ZM615
PLC - L18ER
Network - Laptop, PLC, ZM615, another generic ethernet device for comparison.

I've found a few threads discussing having to physically unplug the ethernet and reconnect to reestablish a connection, but no obvious solution.
 
When you change the RPI as you illustrated, the ControlLogix should close the connection then re-open it with a new connection request with a new RPI value request.

Maybe the Weightronix takes longer to tear down its CIP connection, or maybe they wrote their driver to accept a connection with a specific RPI just once after reboot, and they don't handle a different RPI request gracefully, or maybe the criteria that cause its connection re-establishment to fail are more complex.

If you can set it up once, reboot the Weightronix, and then correctly survive power cycles and downloads, doesn't that solve the issue in practice ?

To provide Weightronix with useful information to investigate the problem, start by capturing the data to that device with Wireshark.
 
Thanks Ken for your insight. I definetly think you're onto something regarding the way the CIP connection is made.

The customer is experiencing the WeighTronix connection dropping out in the middle of a production run. I am only able to reproduce the problem by inhibiting, toggling unicast, or changing RPI. I don't think the problem is specific to any of those things. I think changing one of those values triggers the PLC to reset the connection like you mentioned. The odd thing is, 5 minutes after making that video I could change any of those settings and the connection wouldn't drop out. Wait 5 minutes, toggle inhibit, and the WeighTronix wont reconnect until it's power cycled.

The customer did do a Wireshark trace and sent it to WeighTronix. We were able to see where after connecting the PLC bombards the WeighTronix with identity requests then the connection times out.

Would the EDS file being incomplete cause this? When going to the device properties in RSLogix and clicking on the "module info" tab, RSlogix times out and says the information cannot be found because Avery hasn't populated those fields in their EDS they've been using since 2012. Could this be why the PLC is bombarding the WeighTronix for identity requests?

Video 1 - Able to reproduce the problem by inhibiting the device https://youtu.be/v0MmDqd18wc


Video 2 - Same setup, 45 mintutes later unable to make it fault at all https://youtu.be/CHt3YIw_--c


Video 3 - 5 minutes after video 2, able to reproduce the problem on the first try https://www.youtube.com/watch?v=JkiqSPgazws
 
is it possible that you have an addressing conflict with something else in the plant? we had a display with the same ip address as a hand scanner. the display was only powered up every 5 minutes or so, don't know why it was that way.
james
 
We were able to see where after connecting the PLC bombards the WeighTronix with identity requests then the connection times out.

Can you quantify or describe "bombards" ?

Are they CIP messages to read the various Attributes of the Identity Object (Class 1, Instance 1)?

Are you certain they are from the PLC, or might they be a "List Identity" broadcast packets from an RSLinx station ?

There may be more than one problem here, like a flaky network cable or switch that causes timeouts and then there's an unreliable re-connection.

One problem you might be experiencing is an error that existed in RSLogix 5000 several revisions ago related to electronic keying.

Ordinary A-B devices allow electronic keying to be enabled, and that's what generates inquiries from the controller to the device about its Identity Object. I was always surprised to learn that those queries are done *after* the connection is established, not before (at least, they were 10 years ago).

But Generic Ethernet Modules shouldn't have electronic keying at all. There was a bug where if you created the Generic Ethernet Module, then deleted it, then re-created it at the same address, the electronic keying would get enabled with no way to disable it from the GUI.

The solution was to delete the module, save as *.L5K, re-create the *.ACD file, then re-create the module without making any typos/reconfig. And of course it was fixed in later software.

I think I found this problem in v16 or v17, but I don't recall for sure. You probably have at least v21, so this probably isn't the issue.

In my experience, the symptoms of that Generic Ethernet Module keying problem was a cycle of connect / inquire / disconnect because of keying failure... reconnect and repeat. The I/O connection would come up running every 3 to 5 seconds and the shut down after half a second. So the symptoms you are seeing are different.
 
Can you quantify or describe "bombards" ?
Are they CIP messages to read the various Attributes of the Identity Object (Class 1, Instance 1)?
Are you certain they are from the PLC, or might they be a "List Identity" broadcast packets from an RSLinx station ?


Wireshark screenshots below. Unsure what to interpret from them other than "TCP retransmissions" are sent from the PLC before the connection ceases.

Connection Established -
Lj8hAwU.jpg



rcEw292.png



Connection Fails -
oKWVwnw.jpg



There may be more than one problem here, like a flaky network cable or switch that causes timeouts and then there's an unreliable re-connection.
I thought the same initially but the problem occurs on 5 controllers connected to 5 PLCs on different subnets. I am testing with a 6th controller on a test bench with an isolated network and able to reproduce the problem

Update: On Friday all controllers were changed to new IPs and a watchdog implemented that looks for a change in value every second. So far no failures have occurred, but have no idea why.
 
Last edited:
Hi Matt,

I wonder if you are dealing with a switch hub issue, ARP table or Connection Limit?

Sounds like your HB is keeping the connection going, have you gone direct without a switch?

Are you using certified cables?

Have you looked at noise? VFDs? Welding?

Are all the Ethernet cables run 90 degrees to high power if they cross?

Ethernet not running with high voltage?

How is the grounding situation?

How do you know it is related to RPI?

I searched the KB, and found no hits, but that does not say too much.

I have never, touched RPI, all of the systems I work on, are smaller and light on networking.

Have you checked for FW update on the Scale?

Did it ever work?

In the 2nd video when 5000 goes Not Responding...I started to get stressed and moved my mouse around because I had gone full screen. Feel your pain!

Don't give up!
 
I wonder if you are dealing with a switch hub issue, ARP table or Connection Limit?
Sounds like your HB is keeping the connection going, have you gone direct without a switch?
So the deal is I'm an OEM with a controller on a test bench at our shop and our customer two states away is has 5 controllers installed. I'm using the same routine they're using to communicate with the Weigh-Tronix. I was hoping and praying it was a bad cable but since I'm able to duplicate the problem off-site it can't be a bad plug, cable, or noise......right?
Are you using certified cables?
Nope. Spliced ends.🙃
Have you looked at noise? VFDs? Welding?
Using the weigh controller as a reference, I do not believe there is excessive noise. If there's noise usually the load cell cables will pick it up and we will see it in the weight readout.
Are all the Ethernet cables run 90 degrees to high power if they cross?
Nope
Ethernet not running with high voltage?
I'm sure it is. If I have to go back on site I will definitely run an ethernet cable outside of the conduit all the way to their PLC to test. Would a managed switch act as a 'spam filter' between the controller and the network? Reducing the load on the controller?
How is the grounding situation?
Nominal
How do you know it is related to RPI?
I don't believe it's related to RPI, but I think changing the RPI causes the connection to be reset in some manner. Inhibiting the device usually has the same outcome as changing the RPI - connection timeout.
Have you checked for FW update on the Scale?
WeighTronix sent us the latest rev. Understandably, WeighTronix's poisiton is this controller works perfectly well with thousands of PLCs all over the world so there is no need to suspect their device. On the other end, Rockwell TechConnect is not suited for this kind of troubleshooting. So I am left in middle, with a plant network I'm not allowed on, presenting my case to the wizards of PLCTalk in hopes of a breakthrough.
Did it ever work?
As of today the heartbeat and new IPs addresses have solved the communication timeout problem for now. The customer is requesting a reason they can't use the IPs they had designated for these devices. They tested the IPs on other devices with no issue, but the WeighTronix times out.


Another clue - Early last week the customer had a problem when they plugged in the ethernet cable, the controller would continue to power cycle until it was unplugged. Weigh Tronix looked at the program and said it wasn't the cause. The plant is not using POE. We were told the likelihood of a crash due to a buffer full is not possible. The problem went away when the customer changed the IP. What on earth could that be?

Thanks for your words of encouragement celichi. Network troubleshooting sucks. I thought I could skate through life just knowing how to ping things :p.
 
MattFromGA did you ever find a solution for this?

I am having the exact same problem with a ZM303 to a 5069-L320ER via ethernet. It works for a period of time, sometimes days sometimes minutes, and then randomly stops communication with the PLC.
 
Red,

No, the problem is still ongoing. After speaking with a Rockwell Engineer we suspect that the issue is the result of the Avery-Weigh Tronix device not being OBDV.org certified, which means the ZM615 has not gone through the "rigorous testing" to a true EIP device, hence why it does not have an AOI/AOP/EDS and can only be added as a generic ethernet device. He suspects that the ZM protocol for network refresh, reconnect, or reenable is not configured correctly in the device's firmware.

Our only options right now are adding another $2k ethernet card to their PLC chasis while we wait for Avery to contract out testing their device to a 3rd party company because they aren't able to support it. The Rockwell Engineer said if there is a 1-1 connection with a crossover ethernet cable from ZM to PLC that will solve the problem. < not a cheap workaround
 
Wanted to update this thread in case someone else has this problem -

After sending an Allen-Bradley PLC to AWTX for testing, their engineers identified a problem and released a new firmware update Ver24663. Not sure what the change was but now when you go to 'Controller Properties - Module Info' in RSLogix the page no longer times out and populates with the required information.
 
Thanks for posting the update !

I'm a fan of formal "compliance testing", but your case illustrates the need for even more basic functional testing, and an instance of a field-discovered bug being investigated, repaired, and corrected by a manufacturer.
 

Similar Topics

Greetings to those of you reading this, I am using a 1756-L62 with RSLogix 5000 to implement a new drive interface for a customer. The drives are...
Replies
13
Views
4,246
Is there a way to set the connection timeout for generic ethernet modules to eliminate periodic "IO Faulted - Connection Timeout" messages? This...
Replies
5
Views
5,254
I know, there's tons of resources out there on how to setup generic ethernet devices but I just cannot figure this out, I'm not sure what I'm...
Replies
7
Views
395
Hello, We are having trouble setting up a generic ethernet module for a Yaskawa GA80U4168 drive in RSLogix 5000 Version 20.01. All of the...
Replies
1
Views
533
Do you have a recommendation for a low count IO system that you would stand behind for a remote site? It 1.5 hours drive time to the site. I...
Replies
6
Views
918
Back
Top Bottom