Scada Network Troubleshooting

We are having a network problem. Communications are going down across our network and I'm trying to find the common thread.

Multiple processors with Net-Eni modules have been lost, communications wise. I can ping the net-eni, I can see it on RSLinx if I use the ETHIP driver, although I can't look at the properties and I cannot see the processor behind it using the Ethernet driver as I normally would.

If I disconnect the Eni module at the central switch in our comm hub and plug that cable into my computer, I can fully link up with the PLC after I reboot the ENI module. But when I plug it back into the switch I still can't access it over the network, whether I reboot the ENI or not.

This is occurring with multiple devices. I've reprogrammed one module, thinking it was bad, but it seems to be working fine and the settings had not changed.

I know just enough about networking to be dangerous. All the devices have static IP's that are assigned to them and those IP's are still valid, but it seems like the network isn't recognizing them on their port. I've read the documentation on the ENI modules and it was very specific about having to reboot the module if you plug it into a new port so the module itself can "find" where it is at. These SEEM to be acting like they can't find anything. It's only when I plug them into my computer that they can connect.

I suspected the switch, although there are no fault indications on it and I get good indication on all of the ports activity lights. The IT department shot me out of the water on that one, so I have to figure out the next step.

I realize I'm not giving a lot of information, but if someone has an ideas, I'd appreciate it.

From the information provided, it does sound somewhat like the switch is at fault... or something connected via that switch is causing problems. How many PLCs are we talking about here? I would disconnect them all from the switch, connect only your laptop and then connect each one back individually, checking that you can communicate with each one as you add it back in.

If the first one won't even talk, then definitely suspect the switch (or its configuration).

Disregard the IT department. Just because the link light is on for the port, doesn't mean that they haven't totally ****ed up the configuration (I'm assuming its a managed switch). I've had an IT guy argue with me for 2 weeks that a router couldn't possibly have died and it must be the phone line. finally got him to send a new one out... replaced and working again in 5 minutes.

If it's only a handful of processors, grab an off-the-shelf switch for testing from your local computer store. Not ideal for long term, but good enough to prove whether the switch is faulty or not.

I've also seen some normal desktop software cause havoc with control gear. Symantec PCAnywhere used to have this "scan" mode that would basically consume all the network resources of some modbus TCP slave units i had, causing them to stop responding to the main PLC. If you can temporarily isolate the rest of the network (if there are other non-control type devices) that'd be useful too.

Thanks, I'm still working the issue but we proved conclusively that it is related to the managed switch. Pulled as much of the Scada off of it as we could and ran it all into the server through a couple of 8 port switches we had on hand and those inputs cleared up instantly. As soon as we run one line over to the managed IT switch, it fails again. And yes....they are still arguing that it isn't the switch.

The truly frustrating part is that this isn't my normal work area, so I'm not very familiar with the layout, and there is no network map, no diagrams, no layouts. It's all just been hodgepodged over the years and now suddenly it doesn't work and I'm expected to wave some magic fingers at it and work miracles.

Time to earn my pay.

SiriusMark said:

As soon as we run one line over to the managed IT switch, it fails again.
.

Click to expand...

When you say run a line over to the IT switch, do you mean connecting a single network cable between your 8 port switches and the IT switch? i.e. it's totally out of the network to start, then you connect it back into the network and ALL the PLC's lose comms?

If that's the case, then it could possibly be a network address conflict between your server and some other machine on the network. Try connecting your laptop to the IT switch (with the link from the IT switch to your temporary network removed) and pinging the IP address of the SCADA server. If you get a reply, then there's another device on the network with a duplicate address.

Although in saying that, normally Windows alerts you with a little popup if there is an IP address conflict.

IT departments are notorious for lack of drawings. Where i work, the last guy did do some reasonable "functional" schematics, i.e. this port of the switch is connected to this port, etc. But what he didn't put in was "and the cable is spliced in this JB, then jumps over to this rack, then back again through this patch panel...".

I can ping the 1761-NET-ENI, I can see it on RSLinx if I use the ETHIP driver, although I can't look at the properties and I cannot see the processor behind it using the Ethernet driver as I normally would.

Click to expand...

The 1761-NET-ENI is notorious for its limited TCP connection capability. It supports just six TCP connections; two outgoing, two incoming, and two either direction.

PING doesn't create a TCP connection. Neither does the ETHIP browser; it sends out a broadcast message with the "List Identity" CIP function code, and gets a reply without creating a TCP connection.

So it's possible, or even probable, that something connected to or associated with that managed switch is using up your 1761-NET-ENI device TCP connections.

The most common culprit is unattended RSLinx browses. Once they've identified the device they'll make a TCP connection to read its identity information.

Other network scanning tools like security scanning software might also be doing this. Or maybe there are new HMI devices or workstations on the network that are trying to connect to the controllers as well.

One method I have used is to try establishing a Telnet session with the Net-ENI. It doesn't do a whole lot via Telnet, but it can tell you the state and number of its TCP connections.

The instructions for logging into the 1761-NET-ENI over Telnet are in RA Knowledgebase Article ID # 27418 (TechConnect required).

Thank you everyone for the various replies.

The initial problem has been solved. Another group of technicians had, unknown to me, been brought in to hook up a second Freewave radio to the network. While doing so, they hooked an ethernet cable from the switch that is designed to handle the radios and ran it to the previously mentioned IT industrial switch. The problem is, there was already a cable ran between those two devices, so they created a loop. Once we removed the second cable we were able to get the network up and running again. I have now removed all Scada inputs from the IT switch and installed them onto an isolated set of Scada switches that have one uplink back to the IT switch.


Next problem is, whatever they did with the Freewave radios, now the field's entire radio network is acting haywire. I can communicate with the master radio here on the platform fine, but it isn't communicating with ANY of the slave radios out in the field. I'm running some diagnostics with the master first, since a bad master is the most logical answer. However, I'm pretty sure it's going to end up being another result of the glorious work of the two outstanding unknown technicians who came out and did their work without explaining to anyone what they were doing.