OT: IP network traffic that doesn't belong

[Ken Roach]

I hope that there might be some Ethernet and TCP/IP experts here on the Forum who can give me some advice.

I was recently troubleshooting a telemetry network that uses GE MDS iNet 900 spread-spectrum radios, and one of my troubleshooting steps was to plug in Wireshark to the Ethernet port of a "spare" radio to listen to any broadcast/multicast traffic that might be on the network.

It didn't surprise me to see some broadcast and multicast packets; the usual N-Tron and Cisco management packets, ARP broadcasts, IGMP Group messages, etc. This accounted for only about 25 packets per second, so I don't think this is a big deal.

What surprised me is when I sorted the traffic by protocol and found a handful of TCP frames that were not addressed to the radio I was analyzing or to the PC I was connected with.

There are 51 total hosts on the network, attached to six different switches around the facility. A couple of the switches are managed (Cisco and NTron) but none of them are configured, as far as I know.

The Access Point radio is 192.168.0.53
My spare radio is 192.168.0.113

But I saw TCP packets that were addressed to 192.168.0.1,2,3,13,14,15,16,17,and 18, which came from three different sources (192.168.0.54, .60, and .150).

All of those nodes except one (.150) are hardwired to the network; they are not connected to a radiomodem.

There were no complete conversations or pairs of packets that appeared to be replies to one another. There were some flagged as TCP Retransmissions and a lot flagged as "previous segment lost" (no surprise).

The way I understand Layer 2 Ethernet switches, this should not be possible. The traffic has to get to the Access Point radio in the first place, and it shouldn't because all these devices are hardwired to the network, not part of the radio system.

And even if the Access Point were connected to an Ethernet Hub, then it should not send out every packet it gets, but rather only packets addressed to the remote radios and devices connected to the remote radio Ethernet nodes. If the Access Point sent out all traffic that arrived at its Ethernet port, then I'd see a lot more than these random frames.

Any ideas about what might be happening ? And MDS iNET 900 users out there who have seen something similar ?

 

[Gil47]

What I know about Ethernet TCP/IP would fit on a postage stamp and a small one at that.

I look at those addresses, 1 to 18, to me that appears as if they are a group, binary bits does not fit, but maybe an area in the plant near each other, same switch dropping out could, maybe they are all the same type of equipment, say VFD or HMI grouping, are you able to find a connection along those lines.

Could the radio modem be on default settings, and another also on default settings, just on range limit be trying to call each other.

If nothing else I have given you a bump.

 

[SteveMaves]

I won't claim to be an expert, but one thing I noticed, you mention Layer 2 behavior, but managed switches are Layer 3 devices. I understand that the Cisco and Ntrons switches are "not configured", but isn't that just another way of saying that they are in their default configuration? Maybe there's still some Layer 3 stuff going on? I don't suppose you had time to indulge your curiosity by making a capture going into the AP??

 

[The Plc Kid]

Could it be IPv6 traffic from a device?

 

[The Plc Kid]

http://www.networkworld.com/news/2009/071309-rogue-ipv6.html

 

[bulletin blues]

Did you try running a wireless access point sniffer, such as

http://www.netstumbler.com/downloads/

Not sure if that would've helped, but I would've tried it anyway.

Did you save the wireshark capture by any chance. I'd be interested in seeing it. If I had a similar problem, I'd probably ask Ken Roach...

 

[nic00]

Do you have a router connected between the radio/wired? If not i would suspect that its broadcast......Hopefully i've read this post correctly

 

[Ken Roach]

Thanks, guys, for the comments.

It's definitely ordinary IPv4 traffic; in fact, I can decode the Allen-Bradley data payloads inside some of the packets. This is traffic I expect to be happening on the network; I just should not see it on a port it's not addressed to !

Because these are MDS iNet 900 MHz radios, they fortunately are not on the common WiFi frequencies so there's no chance of this being traffic from an unauthorized client. These are devices that are supposed to be on the network, and the OIDs of the MAC addresses match what the devices are supposed to be (SLC-5/05 controllers and Dell computers).

I think maybe there's a malfunction in the switch to which the Access Point is connected, causing it to overflow the CAM table or otherwise behave like a hub. This might be at the root of some of the non-wireless performance issues they are seeing as well.

 

[Ken Roach]

Just for curiosity's sake, here's a segment of a capture file.

In Wireshark, use the display filter "tcp" to show the four TCP packets that snuck into this otherwise ordinary background broadcast/multicast environment.

I think that I'm going to recommend a router in between the main system and the wireless so that even the multicast and switch management data doesn't go out over radio. The radios themselves perform Spanning Tree, which is about all I need.

 

[JeffKiper]

OT sniffing

Ken sorry to side track this one. How are you capturing the packets in a switch that has port forwarding? If you where to buy switch / hub just to monitor systems what would you buy ?

 

[Ken Roach]

Interesting you should mention that, Jeff. My approach has been evolving.

I very seldom have the good fortune to arrive on a site where all the switches are managed and they have instructions or at least access for configuring them for port mirroring. I can think of two sites that's happened in, and one of them rhymes with Going.

Since I had my laptop directly plugged into the Ethernet port of a radiomodem when I did these captures, there was no need for a tap or Span/Mirror port.

While some of the switches in the network I was working on are capable of port mirroring, more of them are unmanaged. This network was set up by a handful of integrators and the configuration and management of the automation network was apparently not a priority.

Earlier this week I had been tapping in to an Ethernet switch to capture data from a particular SLC-5/05, but the switch was an ordinary NTron 900B, a durable but non-managed switch.

Therefore I had a 10/100 Ethernet Tap, plus a Netgear DS104 hub to provide aggregation of the two split-out data streams from the Tap. The whole thing required three extra cables, two power supplies, and a shelf in the control cabinet.

I recently bought a DualComm DCSW-1005 switch, which I hope will cut down on the amount of gear I have to bring to Ethernet diagnostic engagements.

It's a five-port miniature unmanaged switch with one important feature: preconfigured port mirroring from Port 1 to Port 5. And it's powered by 5VDC, so you can actually run a power cable to it from a USB port. About $45 with shipping.

It will still require two extra cables, but is tiny and uses a commonly available power source. It's not perfect, but it should give me a good tool for 80% of the Ethernet systems I need to connect to and intercept traffic from.

 

[monkeyhead]

I'm having a hard time understanding your network topology, but maybe your picking up some the frames forwarded by one of the switches trying to discover where a host is?

http://en.wikipedia.org/wiki/Network_bridge

Typically a switch has an address table to store what addresses are located at what ports. If a frame comes in that has an address that is unknown to the switch it will forward the frame out on all ports. When the unknown device responds, the switch can then associate it's address to a specific port.

 

[JeffKiper]

Ken it looks like I get to go shopping again. By the wat a USB serial tap is so cool 1 serial cable and a few gender benders and you can snoop all the 232 data.

I only ask because of some network issues that I kind of had I posted over at that other site. In the AB under MSG connections.

 

[Ken Roach]

That is right in line with what I was thinking, monkeyhead.

It makes sense that the Access Point radiomodem acts as a network bridge and sent these packets out onto the radio side when they arrived on the Ethernet side. Likewise, the Client radio modem received them on the radio side and sent them out the Ethernet side. They didn't know any better because they didn't have the destination MACID in their address tables, so they treated it the same way they'd treat a broadcast.

But it should not have arrived on the Ethernet side. These packets are clearly part of a TCP stream, so the switches involved have to know where the endpoints are.

I think that the switch to which the Access Point is connected is malfunctioning, and sometimes sending packets to every port. Or worse, to the wrong port.

I posted on the Forum because this lies outside the area of my expertise. I was expecting maybe to hear "oh, yeah, sometimes you're just going to see that" or "sounds like a network loop being quashed by spanning tree, but some of the packets sneak through".


This was an incidental finding for me, that showed up after I left site scratching my head about why all the Message instruction programming repairs I'd done had not resulted in improved wireless performance. I never did see the switch about which I am suspicious.

I definitely need to press hard for the customer to get a good physical diagram of their network for the maintenance group to use.

 

[Ken Roach]

As long as we're ranging far and wide on the discussion, I wanted to post something neat I learned using Wireshark during this project. It's probably old hat for network analysts and Cisco-certified folks, but was new to me.

I recently used a Fluke EtherScope Series II, and one of the things it did was list the managed switches in the network, including telling me which port I was connected to.

Here's a little screencast that explains how Wireshark can do a little of the same, by watching for packets of the Cisco Discovery Protocol (as long as the switches support that). It's from the Open Source Tools University project of lovemytool.com and Wireshark University.

http://www.lovemytool.com/blog/2009/08/chris_greer_7.html

It was neat to figure out that this part of the plant was plugged into Port 6 of a 24-port Cisco 2950 switch that was probably still configured for defaults, since its name was "Switch".