ControlLogix ENBT Problem

davidg68124

Member
Join Date
Feb 2005
Location
Omaha, NE
Posts
313
I work for a pharma on contract right now and we had a major event over the weekend. We had several ControlLogix ENBT cards of various revision levels (1.4 to 3.3) and a few CompactLogix at R13 fail on their ethernet communications. We seemed to have a cascading failure effect over 29 hours where each one failed at different times. This forced the on duty tech to cycle power by removing the cards and actually power cyclying the CompactLogix. The link lights were solid red and there were no comms on Ethernet. The ENBT had cycling on their Displays EXCP 300. I have called rockwell and discussed this with them and they say unsoliced messages caused the message buffers to overrun. They are suggesting a firmware upgrade but one of the other engineers here on site is resisting this as he found a code listing that list FAIL 300 as an invalid ARP Command.

I will add that on thursday, we had some DHCP issues. WIT believes a rogue DHCP server was added on our local network. They were using an automated process that was using ARP commands trying to locate the problem. WE are leaning this way as well.

The tech at the time did not have the capability to go online with the units through serial to get an idea what the properties of the cards and ports were telling him.

This event cost us 9 hours of data, so we are having issues with releasing the batch.

What I am asking is, have any of you run into something like this? If so, what were your resolutions for it.

I am wondering if I can use system variables to reset the card and bring it out of a fault like this or will I still need to cycle power.

Thanks in advance,
David R. Gulick
 
Some of the things you mentioned worry me.

Firstly is using DHCP instead of hard-coded IP addresses.
If you lose your DHCP server or it is compromised it causes (as you discovered) alot of issues.

Secondly is the fact someone could 'add a rogue' element into the control system.
Is there no firewall or secure router?

Finally it sounds like the control systemm shares its ethernet network with the plant LAN.
Please tell me this is not so!

I have found Ethernet IP to be very rugged and with the correct switches and security very reliable.
 
We are not using DHCP with the control systems.

When I left here a year ago, this particular control system network was behind a firewall with no general Lan connection. It has been since, opened to the General plant WAN due to SOP's being unavailable unless a port was opened, printing access was located on the WAN with WIT saying no to adding additional printers in the expansion area, engineering wanting access from there office, etc, etc. The decision was made to remove the firewall and have WAN access to the expansion area.

I have stated that this needed to change or stuff like this would happen.

Maybe I should be a little more descriptive. The Control network PLC's are all static. iFix and iBatch are static ip's as well. We are merely wondering if this 'scan' is responsible for bringing down those ports on the PLC's. If anyone has had this type of experience.

I will add, this is a tightly controlled network. Theoretically, this should not happen. People are well aware of the consequences of adding not approved items to the network. This is just their belief as there were DHCP issues on thursday and this led them down the path of the automated search to locate. I am wondering, since the ENBT's are proprietary, could this search have caused this issue with an unknown type request to it from the automated process?

David
 
Good to hear you follow the Integrators line of thought and not the average IT person.

What kind of ethernet switches are installed?

AB recommends in this kind of application that the switch has IGMP Snooping.

I've seen Garrett or Phoenix managed switches used successfully.
 
Well, since this is a pharm, redundancy and high performance are a must.

We use Cisco Catalyst 6506 switches. Redundancy enabled, 3gb backbone with 1GB Fiber Link installed.

There are redundant pathways on the fiber as well, so if one link fails, there is another path available for the link. It is a very fault tolerant network.

IT here considers those switches to be'Best Buy' specials and does not allow them on the network. Each unit gets a redundant network drop, directly for the associated IT Switch Closet.

David
 
Was there any other devices dropping offline when this occured?


Have you confirmed that the ENBT firmware does not have the issue that they disconnect after a specific time (I believe it is 720 days)?
 
Goes off after 720 days? I have not heard of this.


The only devices that were affected were the ControlLogix ENBT's and the CompactLogix PLC's. FlexLogix and SLC's were not affected. The HMI PC's were not affected.

David
 
a cisco cat 6500 is definitely far superior to Garrett or Phoenix

it sounds like they don't know how to properly use a cisco switch if they were doing scans to find a dhcp server

its easy just get a computer to get an address from the bad dhcp server then look at your own local arp table (not sacning like crazy) then review the mac-adddress-table on one of the switches (I've done this several times (theres even a few tricks to searching the tables faster) I don't care how large the table is be it 1000 mac addresses or 1,000,000,000

also if they're going to use cisco switches (especially of that level) they should have the PLC network in an isolated VLAN controled with acl's and only assign ports to that VLAN that go to PLC's

the vlans could've helped when they did the scans as arp is a layer 2 function the scans wouldn't have crossed vlans

yes I know arp links layer 2 and 3 but its a layer 2 function and shouldn't be routed

I am an IT person and run a cisco network at home so I do know what I'm talking about here
and I've had to track down mistakes made by other IT people
 

Similar Topics

Hi, I have been trying to connect ControlLogix with a Parker Drive (SSD 590+) using a LINKNet Card; the PLC is a L61 in Redundancy over...
Replies
2
Views
1,169
Hi guys, I've come across a very odd problem that I can't figure out with testing communications failures and my 1756-L72 ControlLogix PLC with a...
Replies
4
Views
5,561
Hi Technocrats, I've the below basic Controllogix Redundancy Configuration, could you please confirm if it will work. Have dual redundant...
Replies
3
Views
3,288
Hi all, I'm experimenting with using 2 1756-ENBT's in a single control logix backplane. My purpose is to eventually isolate some of my remote I/O...
Replies
7
Views
2,814
Hello Everyone, I am working with an Allen Bradley Controllogix PLC and I have the following problem. This PLC is connected to a switch that also...
Replies
3
Views
4,581
Back
Top Bottom