You are not registered yet. Please click here to register!


 
 
plc storereviewsdownloads
This board is for PLC Related Q&A ONLY. Please DON'T use it for advertising, etc.
 
Try our online PLC Simulator- FREE.  Click here now to try it.

---------->>>>>Get FREE PLC Programming Tips

New Here? Please read this important info!!!


Go Back   PLCS.net - Interactive Q & A > PLCS.net - Interactive Q & A > LIVE PLC Questions And Answers

PLC training tools sale

Reply
 
Thread Tools Display Modes
Old October 30th, 2007, 11:20 AM   #1
davidg68124
Member
United States

davidg68124 is offline
 
davidg68124's Avatar
 
Join Date: Feb 2005
Location: Omaha, NE
Posts: 313
ControlLogix ENBT Problem

I work for a pharma on contract right now and we had a major event over the weekend. We had several ControlLogix ENBT cards of various revision levels (1.4 to 3.3) and a few CompactLogix at R13 fail on their ethernet communications. We seemed to have a cascading failure effect over 29 hours where each one failed at different times. This forced the on duty tech to cycle power by removing the cards and actually power cyclying the CompactLogix. The link lights were solid red and there were no comms on Ethernet. The ENBT had cycling on their Displays EXCP 300. I have called rockwell and discussed this with them and they say unsoliced messages caused the message buffers to overrun. They are suggesting a firmware upgrade but one of the other engineers here on site is resisting this as he found a code listing that list FAIL 300 as an invalid ARP Command.

I will add that on thursday, we had some DHCP issues. WIT believes a rogue DHCP server was added on our local network. They were using an automated process that was using ARP commands trying to locate the problem. WE are leaning this way as well.

The tech at the time did not have the capability to go online with the units through serial to get an idea what the properties of the cards and ports were telling him.

This event cost us 9 hours of data, so we are having issues with releasing the batch.

What I am asking is, have any of you run into something like this? If so, what were your resolutions for it.

I am wondering if I can use system variables to reset the card and bring it out of a fault like this or will I still need to cycle power.

Thanks in advance,
David R. Gulick
__________________
The PRICE of FREEDOM is not FREE! My son is now a United States Marine and I could not be more proud of what he and others will do to protect our freedoms here at home.
  Reply With Quote
Old October 30th, 2007, 11:38 AM   #2
SLC_Integrator
Member
United States

SLC_Integrator is offline
 
Join Date: May 2007
Location: Salt Lake City, Utah
Posts: 415
Some of the things you mentioned worry me.

Firstly is using DHCP instead of hard-coded IP addresses.
If you lose your DHCP server or it is compromised it causes (as you discovered) alot of issues.

Secondly is the fact someone could 'add a rogue' element into the control system.
Is there no firewall or secure router?

Finally it sounds like the control systemm shares its ethernet network with the plant LAN.
Please tell me this is not so!

I have found Ethernet IP to be very rugged and with the correct switches and security very reliable.
  Reply With Quote
Old October 30th, 2007, 12:59 PM   #3
davidg68124
Member
United States

davidg68124 is offline
 
davidg68124's Avatar
 
Join Date: Feb 2005
Location: Omaha, NE
Posts: 313
We are not using DHCP with the control systems.

When I left here a year ago, this particular control system network was behind a firewall with no general Lan connection. It has been since, opened to the General plant WAN due to SOP's being unavailable unless a port was opened, printing access was located on the WAN with WIT saying no to adding additional printers in the expansion area, engineering wanting access from there office, etc, etc. The decision was made to remove the firewall and have WAN access to the expansion area.

I have stated that this needed to change or stuff like this would happen.

Maybe I should be a little more descriptive. The Control network PLC's are all static. iFix and iBatch are static ip's as well. We are merely wondering if this 'scan' is responsible for bringing down those ports on the PLC's. If anyone has had this type of experience.

I will add, this is a tightly controlled network. Theoretically, this should not happen. People are well aware of the consequences of adding not approved items to the network. This is just their belief as there were DHCP issues on thursday and this led them down the path of the automated search to locate. I am wondering, since the ENBT's are proprietary, could this search have caused this issue with an unknown type request to it from the automated process?

David
__________________
The PRICE of FREEDOM is not FREE! My son is now a United States Marine and I could not be more proud of what he and others will do to protect our freedoms here at home.
  Reply With Quote
Old October 30th, 2007, 01:26 PM   #4
SLC_Integrator
Member
United States

SLC_Integrator is offline
 
Join Date: May 2007
Location: Salt Lake City, Utah
Posts: 415
Good to hear you follow the Integrators line of thought and not the average IT person.

What kind of ethernet switches are installed?

AB recommends in this kind of application that the switch has IGMP Snooping.

I've seen Garrett or Phoenix managed switches used successfully.
  Reply With Quote
Old October 30th, 2007, 01:34 PM   #5
davidg68124
Member
United States

davidg68124 is offline
 
davidg68124's Avatar
 
Join Date: Feb 2005
Location: Omaha, NE
Posts: 313
Well, since this is a pharm, redundancy and high performance are a must.

We use Cisco Catalyst 6506 switches. Redundancy enabled, 3gb backbone with 1GB Fiber Link installed.

There are redundant pathways on the fiber as well, so if one link fails, there is another path available for the link. It is a very fault tolerant network.

IT here considers those switches to be'Best Buy' specials and does not allow them on the network. Each unit gets a redundant network drop, directly for the associated IT Switch Closet.

David
__________________
The PRICE of FREEDOM is not FREE! My son is now a United States Marine and I could not be more proud of what he and others will do to protect our freedoms here at home.
  Reply With Quote
Old October 30th, 2007, 02:06 PM   #6
Oakley
Member
United States

Oakley is offline
 
Oakley's Avatar
 
Join Date: Oct 2004
Posts: 1,082
Was there any other devices dropping offline when this occured?


Have you confirmed that the ENBT firmware does not have the issue that they disconnect after a specific time (I believe it is 720 days)?
__________________
Artificial intelligence never overcomes natural stupidity.
  Reply With Quote
Old October 30th, 2007, 02:08 PM   #7
davidg68124
Member
United States

davidg68124 is offline
 
davidg68124's Avatar
 
Join Date: Feb 2005
Location: Omaha, NE
Posts: 313
Goes off after 720 days? I have not heard of this.


The only devices that were affected were the ControlLogix ENBT's and the CompactLogix PLC's. FlexLogix and SLC's were not affected. The HMI PC's were not affected.

David
__________________
The PRICE of FREEDOM is not FREE! My son is now a United States Marine and I could not be more proud of what he and others will do to protect our freedoms here at home.
  Reply With Quote
Old October 30th, 2007, 02:36 PM   #8
1_1_1
Member
United States

1_1_1 is offline
 
Join Date: Oct 2007
Location: texas
Posts: 43
a cisco cat 6500 is definitely far superior to Garrett or Phoenix

it sounds like they don't know how to properly use a cisco switch if they were doing scans to find a dhcp server

its easy just get a computer to get an address from the bad dhcp server then look at your own local arp table (not sacning like crazy) then review the mac-adddress-table on one of the switches (I've done this several times (theres even a few tricks to searching the tables faster) I don't care how large the table is be it 1000 mac addresses or 1,000,000,000

also if they're going to use cisco switches (especially of that level) they should have the PLC network in an isolated VLAN controled with acl's and only assign ports to that VLAN that go to PLC's

the vlans could've helped when they did the scans as arp is a layer 2 function the scans wouldn't have crossed vlans

yes I know arp links layer 2 and 3 but its a layer 2 function and shouldn't be routed

I am an IT person and run a cisco network at home so I do know what I'm talking about here
and I've had to track down mistakes made by other IT people
  Reply With Quote
Old October 30th, 2007, 02:38 PM   #9
Oakley
Member
United States

Oakley is offline
 
Oakley's Avatar
 
Join Date: Oct 2004
Posts: 1,082
See AB Answer 36357.
__________________
Artificial intelligence never overcomes natural stupidity.
  Reply With Quote
Old October 30th, 2007, 03:19 PM   #10
davidg68124
Member
United States

davidg68124 is offline
 
davidg68124's Avatar
 
Join Date: Feb 2005
Location: Omaha, NE
Posts: 313
thanks 1_1_1,

I amk almost positive they did not set up a vlan. I will have to check into that.
__________________
The PRICE of FREEDOM is not FREE! My son is now a United States Marine and I could not be more proud of what he and others will do to protect our freedoms here at home.
  Reply With Quote
Old August 8th, 2011, 11:32 AM   #11
Jeff23spl
Member
Canada

Jeff23spl is offline
 
Join Date: Jan 2010
Location: Canada
Posts: 806
enbt FAIL t006 ?

I have posted at the wrong place...
__________________
Siemens Addict

Last edited by Jeff23spl; August 8th, 2011 at 11:37 AM.
  Reply With Quote
Reply
Jump to Live PLC Question and Answer Forum

Bookmarks


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Topics
Thread Thread Starter Forum Replies Last Post
Fault Alarm for Controllogix ENBT Module mcalidl LIVE PLC Questions And Answers 0 May 18th, 2007 02:00 PM
ControlLogix analog out problem hpeck LIVE PLC Questions And Answers 4 June 16th, 2006 06:41 AM
Slide Runaway Problem in Siemens Master Drive riyajahamad LIVE PLC Questions And Answers 13 February 15th, 2006 07:06 AM
Ab Plc5 Rio Problem. fernandes LIVE PLC Questions And Answers 5 March 7th, 2004 01:25 PM
N8 file problem LIVE PLC Questions And Answers 0 February 6th, 2003 07:36 AM


All times are GMT -5. The time now is 02:29 AM.


.