Troubleshooting ControlNet connections between 1756-CNBR modules

I have a ControlLogix system using ControlNet to connect to the remote I/O. I am using 1756-CNBR modules with redundant media in both the CPU and remote racks. The system is a redundant system using the 1756-RM modules. The CPUs are 1756-L63 with firmware 16.57. I have 13 remote I/O racks. Due to the bandwidth of the CNBR modules I am using 3 of them in each of the CPU racks for a total of 19 CNBRs.

My concern is troubleshooting broken connections. Let's say I pull the "B" cable from the bottom of a CNBR module. The "B" LED on every CNBR module flashes red, not just the one with the pulled cable. The "A" LEDs all stay green and comms remain intact, as expected. It does not matter if I pull the cable from the active keeper or inactive keeper. Same thing happens if I break the trunk cable anywhere along its path. Is this normal? It sure makes finding the broken connection a difficult task.

I am in the process of creating a communications overview screen that will show the status of each CNBR on the network. I can ascertain the status of the LEDs using MSG instructions but showing 19 red LEDs on the screen won't help the technicians pinpoint the problem.

Is there another way to isolate the faulty connection and display it on the HMI?

You can go online with NetWORX. There are troubleshooting tools there. Do you have that?

I really have quesions instead of answers.
What is the active keeper?
Can you use a GSV to read the faulted node? I was thinking of an indexing GSV until you find the one with a bad comm. What about in the I/O tree can you see that info?

Media redundancy is done at a very low level by the ControlNet transceivers and I've also been surprised at how difficult it is to programmatically find a disconnected tap on just one channel.

In real life, I use the 1788-CNCHKR signal meter to find disconnected nodes on a faulted channel. You connect to a diagnostic tap on the faulted channel (I always provide one with a dummy load in the cabinet with the controller) and scroll through the node numbers until you find the one with no signal or a distorted signal.

When the network is configured for redundant cabling, all nodes will simultaneously transmit and receive on both CHA and CHB. Each node will independently decide which channel it will listen to, based on low-level noise counters that are internal to each module. The most recent channel to have noise detected will be deemed the 'backup' channel. At any time whether a network has valid or invalid redundant cabling there can be a mix of which channel any node is listening to.

If a node is disconnected from one Channel only, or the trunk is broken, then all the nodes on the ControlNet are going to switch to the remaining Channel.

When I write monitoring logic for my networks, I use a CIP Generic Message that returns the state of the ControlNet Object (Class 0xF0, Instance 0x01, Attribute 0x83). The Channel State byte is the 6th byte returned from that object, and it gives you the LED state, the Redundancy Warning state, and the Active Channel.

Bits 00,01,02 Channel A LED State
0 = Off
1 = Solid green
2 = Flashing green-off
3 = Flashing red-off
4 = Flashing red-green
5 = Railroad red-off
6 = Railroad red-green
7 = Solid red

Bits 03,04,05 Channel B LED State
Indications same as channel A LED.

Bit 06 Redundancy Warning – When warning, the non–active channel in a redundant configuration is unusable by the controller.
0 = normal (no warning)
1 = warning

Bit 07 Active Channel – Indicates which of the two channels the receiver is currently listening to.
0 = Channel B
1 = Channel A

In practice, when I disconnect Channel A from a running system, every node notices the noisy departure of that connection and jumps to Channel B, so the active channel is always the same on every device in the system.

What I haven't looked into in detail is whether or not the error counters for the nodes with both ChA and ChB connected still increment for the channel with the missing cable connection. I suspect they do, which leads us back to getting out the 1788-CNCHKR.

I do have RSNetWorx for ControlNet and I have a 1788-CNCHKR on order. However, I was trying to create an HMI screen that would clearly indicate to the operator/technician when only one channel has been disconnected from a single node and where the bad connection was located.

If a single node is disconnected from one channel only, or the trunk is broken, the ControlNet object 0x83 returns a value of 3 (flashing red-off) for every node on the network. Likewise, bit 06 returns a value of 1 (Redundancy warning) for every node. I can easily detect when both channels are disconnected.

Knowledgebase article 16584 says ControlNet object 0x82 returns Channel A and B frame errors. I'll see if I can use these to detect a disconnected channel from a single node.

Hi Ken.

Thanks for the additional information. I just went through the same process of adding CIP messaging for additional monitoring of the redundant controlnet network.

Looking at the trend data I can see from time to time, CH A or B will indicate flashing green momentarily (3 seconds or less). I can also see when it switches from CH A to CH B and vice versa. Both seem pretty random when it happens. The Warning bit was not toggled during the flashing or switch over.

Appendix A from CNET-IN005C-EN-P indicates flashing green to be temporary errors.

I will look into the tool mentioned above. Initially on CH B I was seeing where it would flash red. Checking all of the cable ends, eventually I found one that was slightly loose. So I cut it and terminated with a new end.

Redundancy error

I am having the same type of errors
After surveying I noticed that the b side was disabled
I enabled a/b through rsnetworx and surveyed again
The b led started flashing red
If I disconnected a it would go to b but as soon as I connected s again a would
Connect and b would start flashing again
A i online right now so I cannot interrupt it
I would like to test with the 1788-mchkr
What do I need to do
I think I have to disconnect all of the devices
Do I need to disconnect the repeater
Also once I have full access if I disconnect the devices on the a side would the network go to the b side and allow me to see what is not talking
Is there’s any other things I can try to isolate the issue