PLC-5/80E Ethernet duplex settings?

[Stonent]

This will be one of those "I'm not a PLC guy, but I'm helping them troubleshoot" questions. I'm on the computers / network side.

We have a PLC-5/80E that has had some off and on issues with it suddenly not responding to the RSSQL. Almost always we can reset the network port on the Cisco switch and communications resumes.

One of our Cisco guys had said that the port went down due to the connection flapping.

We have had our wiring contractor previously run brand new network cable in and we moved to another port on the switch and that seemed fine for a few months but then the problems returned a couple of weeks ago.

I went ahead and ordered a new 10BaseT transceiver for the PLC just to rule that out.

Previously a former employee had locked that port to 10Mbps/Half Duplex several years ago because "the PLC couldn't keep up" But another person had said that it was set that way because the Transceiver didn't support full duplex. Well this new one does so we set the port on the switch to 10/Full and set the transceiver to Full as well.

It's been running fine tonight (but then again it tends to work fine for hours or days before it dies out again)

I've been keeping an eye on the General Ethernet Counters and this particular PLC seems to be getting lots and lots of Carrier Sense Errors. Thousands and suddenly jump to 4 billion and then reset.
Unfortunately I didn' t notice those errors were there before or not.

Here's a snapshot of what I just got:

Commands Sent 15846
Replies Sent 811082
Commands Received 811082
Replies Received 15768
Replies Sent With Error	0
Replies Received With Error 0
Replies Timed Out 81
In Octets 1265264
Out Octets 981409
In Packets 15449
Out Packets 10550
Alignment Errors 0
FCS Errors 0
Carrier Sense Errors 10541
Excessive Collisions 0
Excessive Deferrals 0
MAC Receive Errors 0
MAC Transmit Errors 17
Single Collisions 0
Multiple Collisions 0
Deferred Transmissions 22
Late Collisions 0
Packet Storms 18041

Do they need to have their ethernet duplex configured separately from the transceiver? I can pull up the PLC in RSLinx Classic on the RSSQL but I don't see any options for ethernet settings.

The cabling going to it is only 6 months old so I don't think it should have gone bad again (I plan on doing a continuity test on it either way whenever production isn't running)

In a nearby cabinet we have a 5/40E and it has zero carrier sense errors, but still gets a lot of packet storms.

The 5/80E is sort of the main PLC for that line from what I understand. It talks to the RSView computer, RSSQL, the 5/40E and another PLC cabinet with a CompactLogix in it that deals with some fan control stuff in the area.

Here's some information from the Module Information page as well (if that helps any)

Processor: PLC-5/80E  
Series/Revision: E/K.3  
RAM Status: Good  
Processor Mode: Remote Run  
Major Faults: 00000000 00000000  
Minor Faults 1: 00000000 00000010  
Minor Faults 2: 00000000 00000000  
Program Name: LINE_1  
Firmware Identification: 1785_np580e 2.64 16-Jan-07

 

[Ken Roach]

Return both the transceiver and the switch port to 10 Mb/ Half Duplex.

All PLC-5E controllers and 1785-ENET modules with an AUI port are 10 Megabit, Half-Duplex Ethernet devices only, and do not support auto-negotiation.

Fix that first, then you can try to do some more diagnostics to figure out what else might be failing on the controller or on the port.

 

[Stonent]

Thank you for your response. I will do that as soon as production will allow.
Out of curiosity, do you know by chance what kind of CPU or Microcontroller these PLC5 devices use? I've seen some el-cheapo no-name PLCs before that had Atmel ATMega chips in them, so basically a beefed up arduino.

Since the 5/80E has a built in web server, I would assume it has at least a 32bit processor inside to be able to handle serving up web pages while doing it's main job of taking ethernet requests and talking to the rack.

I've been told there is a project (probably next year) to convert those lines to ControlLogix but we have to keep things running until then.

 

[Ken Roach]

You don't need a 32-bit controller to run an HTTP Server... folks do it with 8-bit PIC microcontrollers, mostly to show off.

A PLC-5/80 has multiple Motorola 68000 series processors running its various subsystems. The Ethernet port has a daughtercard, as does each "Domino Plug" with DH+/RIO ports, plus the backplane interface, and the main CPU that makes it all dance.

The PLC-5E controllers and 1785-ENET "sidecar" modules got a boost ten years ago when their Ethernet hardware was upgraded. I don't know what it's running under the hood, but if a PLC-5E or 1785-ENET has an RJ-45 jack on the front instead of an AUI plug, it has a 10/100 BaseT port that supports autonegotiation and 100 Mb/s Full Duplex Ethernet and has much better performance than the original daughtercard. It sounds like yours, since it has an AUI transceiver, is an older model.

To troubleshoot your system, first get the big issue out of the way and make sure the controller, the transceiver, and the switch are all set for 10 Mb/Half Duplex.

I strongly recommend *against* enabling auto-negotiation on the switch. While it will always negotiate down to the lowest common settings (10/half), some switches and devices will periodically attempt to repair the port by shutting down the link and attempting a negotiation again.

Once you have the basic hardware and Ethernet link stable, you can start looking to see if broadcast, multicast, or mis-directed packets are burdening the PLC's Ethernet interface unnecessarily. The usual approach is to set up a mirroring port on the switch and connect up with Wireshark to see what's going on.

 

[Stonent]

Ok I came in on the weekend and changed back to 10/HDX on the AUI and had one of our networking guys lock the port back to 10/HDX as well. The carrier sense errors have completely gone. I zeroed out all the Ethernet stats in RSLinx on this and the 5/40E nearby so I can compare.

After about an hour I checked the stats again and here's what I've got now:

Commands Sent* 5703 *
Commands Received* 283260 *
In Octets* 4294390279 *
In Packets* 4294960136 *
Alignment Errors* 0 *
Carrier Sense Errors* 0 *
Excessive Deferrals* 0 *
MAC Transmit Errors* 0 *
Multiple Collisions* 4294967289 *
Late Collisions* 0 *
Replies Sent* 280992 *
Replies Received* 5656 *
Replies Sent With Error* 2268 *
Replies Received With Error* 26 *
Replies Timed Out* 20 *
Out Octets* 4294516756 *
Out Packets* 4294962595 *
FCS Errors* 0 *
Excessive Collisions* 0 *
MAC Receive Errors* 1 *
Single Collisions* 2 *
Deferred Transmissions* 4294967284 *
Packet Storms* 6374*

The Deferred Transmissions and Multiple Collisions have gone up quite a bit.
Looks like time to go back and check the ethernet cabling again.

I'll ask our network engineer if the switches we have can do port mirroring. They're Cisco 3750 series.

Yesterday when we put this change in place. I put a small switch in between the PLC and the network and sniffed for a few minutes. I filtered to port 2222 so I'd just see the Ethernet/IP traffic. I'll post a screen capture in a bit.

 

[Stonent]

Here's the wireshark screen shot with the traffic filtered to just to and from the 5/80E (the device ending in 43). Anything with a red dot next to the source is something from outside the PLC's VLAN. All other traffic is from other devices on that VLAN. This is a snapshot from yesterday when I was on 10/Full, so what it currently is, may be quite different. Something I can look at later.



Interestingly from this capture I found tons of red "spurious restransmission" entries from what appears to be going to and from a Panelview. Whatever it is , is running FactoryTalk Viewpoint but then says FactoryTalk View ME is not running. I can research that one separately.

[TCP Spurious Retransmission] 57673→EtherNet/IP-2 [SYN] Seq=0 Win=32768 Len=0 MSS=1460 SACK_PERM=1

 

[shawn_75]

I put a small switch in between the PLC and the network and sniffed for a few minutes.

Did you mean to say you put a hub in between the PLC and the network, or did you use a switch?

 

[Stonent]

Did you mean to say you put a hub in between the PLC and the network, or did you use a switch?

Depends on how technical you are on the meaning of hub vs switch.

It was a 5 port Linksys/Cisco 10/100 unmanaged device.

http://www.amazon.com/Cisco-SD205-5-port-100-Switch/dp/B0000ALFEU

Using the strictest definition, it was in fact a switch as hubs are almost (if not all) 10Mbit.

I'm not trying to be pedantic, I just had a former network engineer at the company correct people when they didn't use the terms correctly so I try to follow the most strict definition.

Sort of like how someone asks how you are and you say "I'm doing good" when the correct grammar is "I'm doing well". I catch and correct myself frequently.

 

[shawn_75]

Managed or unmanaged, a switch will prevent you from seeing all of the network traffic that is being sent to/from your PLC. Without using port mirroring, you need to use a hub if you want to see all of the traffic. You had 4 billion collisions in one hour and you're only capturing roughly 4000 packets in a minute that are marked for that PLC. You're missing a whole lot of traffic. The packets you marked as suspect are occurring once every 200 or so packets. Those don't seem to be your problem.

 

[Stonent]

Managed or unmanaged, a switch will prevent you from seeing all of the network traffic that is being sent to/from your PLC. Without using port mirroring, you need to use a hub if you want to see all of the traffic. You had 4 billion collisions in one hour and you're only capturing roughly 4000 packets in a minute that are marked for that PLC. You're missing a whole lot of traffic. The packets you marked as suspect are occurring once every 200 or so packets. Those don't seem to be your problem.

I understand that. I just can't do any more until Monday when everyone's at work.

 

[robertkjonesjr]

Is this what we are seeing in the wireshark screenshot? We only see one direction of flow - though we can infer there is two-way communication as it is TCP and not faulting. I assumed it was from a span port on your Cisco switch set for Rx or Tx only, where it could be more useful to see both Tx & Rx as data is missing in this case, and it's hard to know exactly what is missing.

From: http://wiki.wireshark.org/HubReference

If you have a dual-speed hub with an internal switch, it means that if you have a 10MBit host communicating with a 100MBit host and you will only be able to see one direction of that traffic; you will only see the traffic from the 10MBit host if the interface of the machine capturing traffic is configured for 10Mbit/s, and you will only see the traffic from the 100 Mbit host if the interface of the machine capturing traffic is configured for 100MBit/s.

I am no hub expert, but Cisco does not agree with you that there are no 100mb hubs, as they certainly reference them:

http://www.cisco.com/c/en/us/support/docs/lan-switching/ethernet/12006-chapter22.html

With half duplex, it is normal to see collisions. I recall 802.3 says it will do up to 16 retries on a detected collision, then drop the frame. I don't know this product at all so I can't really comment on the absolute value of the stats you provide. Of interest to me is the packet storm counter.

For 10mb Ethernet, the max rate is 15K pps (roughly - see http://kb.juniper.net/InfoCenter/index?page=content&id=KB14737). So in an hour, that would be

1 hr x 60min/hr x 60sec/min x 15 000 frames/sec = 54 000 000 frames total

Now there is no way this device can handle this many packets... but this is still very different than the very large numbers shown, which have octets roughly equal to packets. I would expect each packet to consist of multiple bytes, so

packets << octets

Here is a good write-up on spurious retransmissions:

https://blog.packet-foo.com/2013/06/spurious-retransmissions/

My next step would be to get a better packet capture, complete, likely from the Cisco with a span (mirror) port, unless you have a true network tap. Some people on the forum like this one:

http://www.fte.com/products/NetworkTap.aspx

Or this is may work in a pinch:

http://www.dual-comm.com/fast-ether...4BzvsZ5OU-juHfGTSFySg7jhJ6huSwhonoaAqCm8P8HAQ