Diagnose cabling problems in Parallel Redundancy Protocol PRP

[thingstodo]

I’m working on getting a Parallel Redundancy Protocol PRP system set up in a lab situation. The L73 PLC is in rack 4. 3 io racks are 1,2,3.

I have posted about this system in the past. See thread https://www.plctalk.net/qanda/showpost.php?p=937884&postcount=1

An update to the comments from the previous thread, Rockwell network engineers have reviewed a newer version of the sketch and approved it as a PRP network. It appears to follow PRP rules.

So parts were ordered, and have eventually arrived.

We’ve got some hardware, connected it together, and it’s working. I’m looking to detect failures and alarm them before they cause process issues (equipment stops running due to communication failure).

Racks 2 and 4 have a 1756-EN4TR set to mode 4 (PRP). Racks 1 and 3 have 1756-EN2TP.

All 4 racks each have 2 cat6 copper connections, Channel A to a Stratix 5400 switch and Channel B to a Stratix 5410 switch. The IT guys have not decided on the config for the Stratix switches as yet. The communications works just taking the switch out of the box, powering it up, and plugging in the RJ45 cat6 cables. Since the switches do not have IP addresses as yet, I can’t use the AOI to show which ports are up, get the temperature of the switches, see that only one power supply is working, etc. I’ll get there when IT gets the switch config nailed down. I believe that stuff is available from the AOI for the Stratix.

For now, the racks appear to communicate fine this way. We have redundant power supplies for each switch, redundant cabling to each EN2TP and EN4TR.

For all single points of failure the rack status for the input cards is solid - I can disconnect one cable at a time from an EN2TP or EN4TR, power down one switch, power down a power supply. No communication problems. This is as expected.

I’m trying to figure out how to detect when one of the above single points of failure has occurred so that it can be alarmed and someone can go fix it before a second failure takes down the process. I’d like to alarm when a network cable is cut or is experiencing retries, CRC errors, Frame errors, etc that indicate a cabling problem. Or when Port A or Port B on a particular EN2TP or EN4TR has no link, is not active, etc.

So far, going through various manuals I have found two recommended messages to each rack as below. But there is no doc for what the bits in the returned DINT mean:

Message type: CIP Generic
Service Type: Get Single Attribute
Class: 56
Instance: 1
Attribute: 11
Destination Element : a DINT for each message

Attribute 11 gets A port status, Attribute 12 gets B port status, that's why there are 2 messages to each.

I can't locate any documentation from Rockwell that tells me what the bits in the message above (destination DINT) represent. I had high hopes that I could alarm any single failure in the ethernet cabling using these messages.

I am monitoring the message instructions for .ER, but that error bit seems to take 30 seconds or more to detect an unplugged ethernet cable. If the ethernet cable is not totally disconnected for the entire time (RJ45 is unplugged, then plugged back in within 30 seconds), the message goes through and there is no .ER on the message. 30 second timeout seems to be
fast enough for a cut cable but it doesn’t appear to see an intermittent cabling problem.

Am I just being too paranoid?

Is anyone alarming issues with PRP and is willing to share?

 

[Forebiz]

I'm actually just playing with this in a lab setup too. Check out EIP_Diagnostics_AOI in the Rockwell sample library. Without the faceplates its a little hard to use but if you enter your path in EIP_NodeTable.Public_Operations.Set_NewNode and put a 1 in EIP_NodeTable.Public_Operations.CmdCreateNewNode it will add the module to EIP_NodeTable.Public_NodeTable[1] each additional one you create will add to that array of nodes.

This allows me to see a whole lot of data out of the module including link status.

 

[thingstodo]

I'm actually just playing with this in a lab setup too. Check out EIP_Diagnostics_AOI in the Rockwell sample library. Without the faceplates its a little hard to use but if you enter your path in EIP_NodeTable.Public_Operations.Set_NewNode and put a 1 in EIP_NodeTable.Public_Operations.CmdCreateNewNode it will add the module to EIP_NodeTable.Public_NodeTable[1] each additional one you create will add to that array of nodes.

This allows me to see a whole lot of data out of the module including link status.

Thanks for the response!

I'm having trouble finding the EIP_Diagnostics_AOI.
- under Sample code on the Rockwell site, searching EIP_Diagnostics_AOI gets no results. Searching only AOI get 191 results, but none of them the EIP_Diagnostics AOI
- I'm running V33.01 of Studio. The various Sample code loaded with the Studio package does not appear to list EIP_Diagnostics.

Can you point me to where I can locate this AOI?

 

[Forebiz]

I couldn't find it searching that either. Search 94841 it is named EtherNet/IP Diagnostics Faceplates for ME and SE V3.1. I originally found it searching EN2TR but same logic works for EN4TR.

 

[thingstodo]

I couldn't find it searching that either. Search 94841 it is named EtherNet/IP Diagnostics Faceplates for ME and SE V3.1. I originally found it searching EN2TR but same logic works for EN4TR.

Thanks again!

Yes, I think this is the one.

On the Rockwell Forums I received this link, which appears to work when I am not logged in so I don't think you need a service agreement to access it. The AOI is in the first download - you don't need the Guardmaster download at all:

https://www.rockwellautomation.com/...29fd-f3ff-d748-41975636ab5b;deepLinking=false

 

[thingstodo]

I got the AOI installed and operating in the lab. 4 racks each connected to 2 stratix switches as described above

It will run over Xmas break just to get some runtime on the code

The AOI user interface is a bit cumbersome running without Factorytalk View, but it works.

In January testing will begin on verifying that each single failure on the rack side is detected (ethernet cable disconnected, cable damage or noise, port speed dropping to 10 mbps) and alarmed

The Stratix AOI will be added to the lab program when the IT department have decided what config to load into the Stratix and what IP addresses they want to use

Using the Stratix AOI, verify that each single failure on the switch side is detected (power supply redundant power loss, sfp removed, network cable removed, noise on the network cable)

 

[thingstodo]

Update for March 1

Delays in 1756-EN2TP and 1756-EN4TR arriving have changed plans. PRP is still the eventual goal. But we don't have enough hardware to get it all put together. The system was documented as best we could, sketches, PLC programs, etc before we took it apart and re-arranged all of the pieces.

I know this description is hard to follow. I'll try to post a sketch, or a picture of the sketch, later.

The lab has been converted to DLR across switches and racks in 5 MCC rooms and 3 controllers.

The EN4TRs can be used for either PRP or DLR. We had a few 1756-EN2TRs from earlier testing that are being used in the DLR rings. The 1756-EN2TPs that we received are being used to connect the stratix switches (that are on the DLR) to racks with a single connection (only 1 port used). It's the best we can do with the hardware that has arrived.

This allows for converting all 5 MCC rooms to DLR. One DLR spans all MCC Rooms. 3 of the MCC Rooms have 1 or 2 racks which are connected to the local switch with an EN2TP using one port. There is one additional local DLR in each of the 2 larger MCC Rooms, where there is a controller in the rack. Multiple local racks are all connected as part of that local DLR. There are no switches on the 2 local DLRs. We would add a stratix if we could to give IT visibility into each DLR but we don't have enough stratix switches right now. If more stratix switches arrive before the end of shutdown (Mid April) then each DLR will have at least one Stratix.