CIP Message Reference?

[RSL]

I am running version 13.53 in my controller and version 3.3 on my redundant ENBT's. These revisions were selected from a matrix provided in the release notes for version 13.53. You cannot get away with going outside these matrix parameters in a redundant system. There are too many variables. As stated by Contr Conn you should be using version 3.7 in your ENBT's.

We use the IP swap feature. It works great. It takes a little while for the ENBT to resolve it's adress and MAC ID's with our switch, but it always comes up after a switchover. Their is a brief interuption of HMI data. 15-20 seconds I think. This is a mature system that has been running version 13.53 since shortly after it was available. My disqualified rack will always requalify and synchronize. (unless their is a real problem with it)

It sounds like the duplicate IP address detection is latching in on one or the other of your ENBT's. This is a newer feature for standard racks not redundant systems. The SRM manages the IP's and the node numbers on the ControlNet's in a redundant system. I suspect this problem will go away if you use the EXACT firmware revision from the matrix on your ENBT's.

I have had no problems and did not see a need to update to 13.7x. We were soooooo happy with it compared to version 8 and that we left well enough alone. I cannot swear that 13.7x and 3.7 firmware in a redundant system work flawlessly. I am still at 13.53 and 3.3.

The only thing I saw 13.7x gives you is an enhanced upgrade procedure for future releases. It sounds desirable but I am not taking my stable system down just for that. There is some mention of the problem you describe in the release notes for version 13.7x as a corrected issue. I have never seen this problem in my system though. My whole system is on a UPS and NEVER loses power to both racks at the same time. This might be how I am avoiding the problem.

RSL

 

[Contr_Conn]

I cannot swear that 13.7x and 3.7 firmware in a redundant system work flawlessly. I am still at 13.53 and 3.3.

RSL, I can confirm that 13.7x will work exact same way 13.53 does.

 

[RSL]

Contr Conn, Thanks for the info.

From the release notes I leaned towards that being the case. I just have not ran 13.7x.

Side note, do you know the scheduled release of the next redundancy firmware?

RSL

 

[akreel]

This is not a bug, this is a feature called IP swap. It is well documented in redundancy manual. There are no issues with ENBT redundancy firmware. If ENBTs correctly configured: both ENBTs have exact same configuration and IP, then secondary ENBT will always get IP+1 from SRM.
After switchover primary chassis ENBTs will swap IPs and primary will always have IP and secondary IP+1.
I am using redundancy v13 on multiple test systems and not aware of any issues with ENBTs.

That part works fine when you first power up the racks. After a fail-over the second rack will not Sync, even if you cycle power. I've let the PLC run for a week unsynchronized, all the while it said it had a fully qualified partner. That's the bug that I was talking about.

If you're interested or curious, I can fail the rack and upload the log file from the SRM. I think that there's some interesting gibberish in there when the event occurs.

Did you talk to techsupport about this issue?

"Yes, the PLC is plugged in."

As I said, we have Rockwell people on site. They are generally more responsive than tech support. Without them on site, a number of our issues with RSView SE would still be hanging out there. You can thank us for some of the latest patches (if they've gone public). I showed them the problem, they agreed with my assessment. I've also asked them to see if this is already a known issue.

After I play with it a little bit more, I'll be sure to have them enter it into the Rockwell issues database, so that it can get put into the revision schedule.

One thing that is unique about this PLC is that we have one remote rack of IO with two CNET modules in it, to avoid the "lonely module" problem. I don't see how that is causing the problem, though.

 

[akreel]

By the way, here's why I THINK we're seeing what we're seeing:

Typically, there is a period of auto-negotiation to set baud rates and stuff between an ethernet device and an ethernet switch.

We're using CISCO switches and we've configured the ports to power up in the configuration that we need. There's really no wait time when you connect the ENBT.

The SRM module takes an awfully long time (up to 30 seconds) to perform the IP address change. Meanwhile, the ENBT is doing its own thing and connecting up to the network. I know that the ENBT will respond badly if it sees a duplicate IP address. We've seen it in other PLCs. You have to remove power from the module to clear whatever flag is being set, and then it will return to normal.

In the redundant system, I think the ENBT is voiding itself. It does the same thing if you unplug the ethernet cable.

If you want to test my configuration, try putting a crossover cable between the two ENBTs and see what happens. I haven't tried it yet, but I bet you'll get a similar result!

AK

 

[akreel]

It takes a little while for the ENBT to resolve it's adress and MAC ID's with our switch

That, I think, is the key!

I'm not bashing the hardware. Our installation is one of many thousands of combinations that one could dream up. I think that we might not see this if our switches were set to auto-baud or if we were using RadioShack gear.

This, to me, is just one more challenge to overcome.

 

[Oakley]

We chose not to implement the failover ENBT's in our redundant setups.

The main reason is our switches are set up as static (as per IT). We have no choice there.

The other is we have logic to utilize the two CNB's in the chassis (fail over there) as well as multiple ENBT's with multiple failover paths. A lot of additional programming but we live with it.

One upgrade to this scenario is that the comm chassis is a single point of failure. So we are going to add an additional comm chassis, one CNB per, and one ENBT per.

Also on each chassis is a direct Ethernet to our redundant Scada servers. One will be on each chassis.

A bit overkill, but it works.

 

[Contr_Conn]

do you know the scheduled release of the next redundancy firmware

ver 15 few feeks away, I have no date.

The SRM module takes an awfully long time (up to 30 seconds) to perform the IP address change

This is incorrect, SRM changes IP immidiately, but it takes about 30 seconds before switch updates ARP table. This has nothing to do with SRM or ENBT.

They are generally more responsive than tech support.

Techsuppor would resolve all your issues long time ago if you actually call them. How I know? PM me if you need the answer.

 

[RSL]

There is no doubt about it, ANY delay in the ENBT establishing a link after a switchover is the managed switch being finicky. I have looked at this right after we implemented version 13. Even though the IP swaps out instantly the switch suddenly sees a different MAC ID on the hardware. This causes a short delay in the switch "resolving" what the heck happened. (this is what I was told anyway)

Our IT people said we could likely configure the ports involved for faster switchover, in the end we decided to live with it. This way we use the companies standard switch configuration.

Our original configuration involved using a 3rd rack as a communication gateway. It worked fine but all that data for our HMI's ate up an entire ControlNet and then some. We really like the redundant ENBT's. We route all critical comms through them at a fast rate. We still have the 3rd rack and use it as a data concentrator for other applications to "hit" for large amounts of data at a much slower rate.

RSL

 

[akreel]

Here's an update for anyone who's curious:

I located this document by using the key word "IP Swap." I thought for sure it would be the magic bullet. I did find that we probably didn't follow the correct proceedure for assigning IP addresses. You need to put the ENBT modules in a non-redundant rack first, apparently. I tried that, and the ENBT modules seemed to change addresses faster, but the partner rack still would not SYNC.

I might be able to make it work, if I picked the right combination of firmware. But, we've specified firmware versions for all modules on this project. Making exceptions is not a good idea.

So, I tinkered with CIP messaging and came up with a solution that almost works perfectly. I assigned two non-sequential addresses and let the PLC handle the address changes. There were only two gotchas:

1. During fail over, the IP addresses might match, causing IP Swap to take place. By reserving a block of four addresses and using the first and third, I'll be able to keep that from causing issues elsewhere on the network.

2. If both racks loose power, the "wrong" one might boot first and become primary. The racks still SYNC, but there will be no connection to the HMI. Of course, this would flag someone that there is a problem.

Now that I'm done with my little training exercise, I might call tech support to see what they have to say. I'm pretty sure they'll come back with "mismatched firmware" as the diagnosis.

 

[Contr_Conn]

pretty sure they'll come back with "mismatched firmware" as the diagnosis.

This will be a first question. For sure

No matter what your customer said, you have to understand that you can't use a random FW releases, all releases must be taken from redundancy bundle.

Why? Because these revisions have additional "hooks" and features normal FW does not have. Also if FW released for one redundancy package it does not mean it will work with another redundancy setup, they simply MUST MATCH. You have to treat redundancy FW as one big file that needs to be downloaded to the multiple modules.

Here is the latest ver 13 package, this is not a "recomendation", this is a requirement.
http://support.rockwellautomation.com/controlflash/FirmwareFiles/1756-RN608F-EN-E.pdf

So most likely you just trying to resolve a non-existent issue that possibly caused by FW mismatch.

Now if you really want to disable IP swap, simply assign 2 different IP addresses IP and IP+1 to two ENBTs and IP swap will be automatically disabled. IP swap enabled ONLY if both IP addresses are exactly the same.

 

[akreel]

all releases must be taken from redundancy bundle.

Great! That's exactly the firmware we have. Except for the L63 which we have at Version 13.70. That should make the call to tech support especially easy.

I was concerned that CNB version 5.51 might not be validated with the IP Swap feature yet, or something.

 

[Contr_Conn]

CNB 5.51 has nothing to do with IP swap.
But L6x must be 13.71, no exceptions.

 

[Contr_Conn]

You may try very simple test:
Remove all modules, except SRM and ENBT. No processor, no CNB

Try to SYNC system like this.
Post SRM logs.

If you have switch forced to 100/FULL, did you force ENBT as well? or you left it with autonegotiation?

 

[RSL]

That is the firmware matrix I was referring too earlier in this thread. It is in the release notes of the bundle.

I had never read that troubleshooting document before. I also did not use a seperate rack to configure my ENBT's either.

That procedure Contr Conn mentioned looks like a good way to test your IP swap.

I bet if you change the controller minor revision to 13.71 you will be good to go.

RSL