Hi:
We have 2 ControlLogix 5000 systems working on different parts of a large conveyor system. They are interconnected via reduntant CNBR cards. They share some info, but work by themselves for the most part. Both systems are redundant, using SRM Modules each of them. The processor is in a separate rack from the Ethernet module in both systems. Processor and rack with Ethernet module are connected via CNBR cards.
A few weeks ago, one of those systems had a problem where one module -'apparently' the Ethernet module- that talks with the sort computer was not working, so the system could not sort parcels; a Cisco switch is used for comms between the PLC and Sort PC (I say 'apparently' because ultimately the problem was something else). This issue took the system out of operation. Fortunately we can divert everything from one system to the other for this kind of emergencies, and that's what we did.
I went ahead and troubleshot the system (let's call it Sys1) doing what was a sensible technique in my opinion: First try a different Ethernet cable, then try using a different port in the switch, and finally, try a different Ethernet module. Nothing fixed this issue. I did my troubleshooting while both systems were down for the day, in the wee hours.
What I did that night was to test a different Ethernet module, i.e., I would take the one I thought was bad, put a new one, and see if Ethernet comms happened. In the end I did not even get that far, or leave the new module installed because I had to configure the IP Address on that Ethernet module and I was not prepared to do it that night, so I simply replaced the original Ethernet module and left...
The next day the other system (the good one, Sys2), would not work. We hit a brickwall because the Ethernet module was not working, so we could not talk to the system, and I did not have a PCC card to use one of the available ControlNet ports.
AB was called in, and what the tech found was a CNBR card that had to be reseated in Sys2 (nothing to do with Sys1, where the suspect Ethernet module is installed). That card provided comms for the backplane of the racks.
According to the tech, the Ethernet module in Sys1 was fine, but comms could not get any farther than the rack backplane because after that, the CNBR was frozen, and did not allow comms in the ControlNet network that links the processors among themselves; after reseating it in the rack, things started working, including the Ethernet module. I thought it was a sensible explanation, and quite frankly was surprised the problem turned out to be a CNBR card in the other system, and not the Ethernet module in the system with the symptoms...
Now my question (finally! Sorry for the long preamble): The boss (not a controls guy) is insisting that the Ethernet module I tried out the previous night broke the ControlNet module that had to be reseated. I tell him that is not possible. They are in different systems and they talk different languages (CIP vs TCP/IP).
Even more, he says that somehow firmware in the ControlNet module on Sys2 was corrupted by the Ethernet module I tested out on Sys1 the previous night. I have explained that 'flashing' firmware into a module is an elaborate process, and that it is impossible that the Ethernet module could flash anything by itself. As I mentioned, I could not even configure the IP address that night, so that module did not talk to anything whatsoever.
Even worse, another tech is telling him that the Ethernet module had to be 'flashed' before being installed (?). I said that 'flashing' the module with updated firmware is not a given, it may be needed if the current firmware is not compatible with the rack or processor, but in this case it was fine. The only thing that needs to be done to get that card working is configure the right IP settings (IP Address, Network Mask, etc.), and that is it.
I would appreciate opinions from you guys. This is a weird problem, and would appreciate any thoughts, or similar situations if you ever had them.
Thanks!
Cipitio.
We have 2 ControlLogix 5000 systems working on different parts of a large conveyor system. They are interconnected via reduntant CNBR cards. They share some info, but work by themselves for the most part. Both systems are redundant, using SRM Modules each of them. The processor is in a separate rack from the Ethernet module in both systems. Processor and rack with Ethernet module are connected via CNBR cards.
A few weeks ago, one of those systems had a problem where one module -'apparently' the Ethernet module- that talks with the sort computer was not working, so the system could not sort parcels; a Cisco switch is used for comms between the PLC and Sort PC (I say 'apparently' because ultimately the problem was something else). This issue took the system out of operation. Fortunately we can divert everything from one system to the other for this kind of emergencies, and that's what we did.
I went ahead and troubleshot the system (let's call it Sys1) doing what was a sensible technique in my opinion: First try a different Ethernet cable, then try using a different port in the switch, and finally, try a different Ethernet module. Nothing fixed this issue. I did my troubleshooting while both systems were down for the day, in the wee hours.
What I did that night was to test a different Ethernet module, i.e., I would take the one I thought was bad, put a new one, and see if Ethernet comms happened. In the end I did not even get that far, or leave the new module installed because I had to configure the IP Address on that Ethernet module and I was not prepared to do it that night, so I simply replaced the original Ethernet module and left...
The next day the other system (the good one, Sys2), would not work. We hit a brickwall because the Ethernet module was not working, so we could not talk to the system, and I did not have a PCC card to use one of the available ControlNet ports.
AB was called in, and what the tech found was a CNBR card that had to be reseated in Sys2 (nothing to do with Sys1, where the suspect Ethernet module is installed). That card provided comms for the backplane of the racks.
According to the tech, the Ethernet module in Sys1 was fine, but comms could not get any farther than the rack backplane because after that, the CNBR was frozen, and did not allow comms in the ControlNet network that links the processors among themselves; after reseating it in the rack, things started working, including the Ethernet module. I thought it was a sensible explanation, and quite frankly was surprised the problem turned out to be a CNBR card in the other system, and not the Ethernet module in the system with the symptoms...
Now my question (finally! Sorry for the long preamble): The boss (not a controls guy) is insisting that the Ethernet module I tried out the previous night broke the ControlNet module that had to be reseated. I tell him that is not possible. They are in different systems and they talk different languages (CIP vs TCP/IP).
Even more, he says that somehow firmware in the ControlNet module on Sys2 was corrupted by the Ethernet module I tested out on Sys1 the previous night. I have explained that 'flashing' firmware into a module is an elaborate process, and that it is impossible that the Ethernet module could flash anything by itself. As I mentioned, I could not even configure the IP address that night, so that module did not talk to anything whatsoever.
Even worse, another tech is telling him that the Ethernet module had to be 'flashed' before being installed (?). I said that 'flashing' the module with updated firmware is not a given, it may be needed if the current firmware is not compatible with the rack or processor, but in this case it was fine. The only thing that needs to be done to get that card working is configure the right IP settings (IP Address, Network Mask, etc.), and that is it.
I would appreciate opinions from you guys. This is a weird problem, and would appreciate any thoughts, or similar situations if you ever had them.
Thanks!
Cipitio.