Investigating Mysterious Network Traffic

I discovered in our network a host that was generating a huge amount of traffic compared to every other host in the network. All I had to go on initially was the IP address – so how do we track down the culprit and see what is going on?

First, we have the IP address and access to the network. Thus, my next step was to log into the firewall and sniff the network to see where the traffic is going. That did not prove to be helpful.

However, if we have an IP address, we have a MAC address in the ARP table. There are lots of ways to get this, including the arp command. I used the -e option to tcpdump to get it during the sniff of the network.

If you have a MAC address, you can look up the manufacturer of the network card, which in many cases may be the computer manufacturer. In this case, it was AsusTek. A search of the premises for Asus equipment turned up nothing.

Since this was almost certainly a Windows machine, using nmblookup -A $ip may turn up something useful: in particular, it may return a host identifier or name that can identify its owner. In this case, it turned up a name that had no meaning to me.

The traffic will be viewable at the switch where the equipment is plugged in. Thus, we can go to the switch where all of the workstations are plugged in, and plug a laptop into a port to sniff traffic. Once I had done this, I logged into the web interface, then mirrored 12 of the 24 ports to the port the laptop was in even as the laptop was sniffing the network for the suspect host.

Doing this will send all traffic on those ports also to the port the laptop is listening on – and in this case, turned up the suspect host. Performing a binary search will narrow it down – that is, mirror half (6) of the 12 ports and see if the traffic continues to flow, then mirror half of that (3).

Once I narrowed down the ports to a single port, I tested it. Does the traffic stop when we stop mirroring that port? Once the mirroring of the port was stopped, the network traffic seen by the laptop stopped. Mirroring again resumed the suspect traffic.

Now we know what port it is using. Following the cable from the switch to the patch panel shows which physical outlet is connected, and with the map showing where all of the outlets are we can track the outlet down.

Going to the outlet, I found that there were several devices plugged into a cheap hub. Since there was no one in that office, I pulled each computer’s link to the hub and plugged it into a laptop. This laptop, again, was used to sniff the traffic coming off of the host. A couple of tests and the host was identified.

Next step is to get a virus checker on it and run that to see if anything is running that shouldn’t be.

Investigating Mysterious Outbound TCP Connections

Recently, I had a situation where a firewall had outgoing TCP connections I knew nothing about. If you are to maintain a secure system and a secure network, this sort of thing demands investigation. (I won’t report full details in order to maintain anonymity for various entities.)

Where to start, then? First, use tcpdump and capture the traffic. It may be useful to capture it into a file for looking at with Wireshark. I watched the traffic flow across all interfaces by using tcpdump:

tcpdump -s0 -n -i any host 999.999.999.999

I noticed that there was no traffic coming from anywhere except the outgoing port on the firewall.

Then I became more interested in the IP address being connected to and the port (443 or HTTPS in this case). Connecting to the IP on port 443 didn’t turn up anything interesting (except they used Red Hat Enterprise Linux). Looking up the IP address in a whois listing showed that the IP address was very similar to that of the firewall maker – very interesting indeed. Looking up the IP in reverse DNS showed it to be an Amazon AWS host in Ireland.

Then I wrote a script that used lsof to watch for a connection and find the program making the connection:



# Prep: erase if present
rm -f $WORK

while true ; do
    if $(lsof -ni $PORT > $WORK) ; then
        ( echo "Found ports open:" ; echo ; cat $WORK ; echo ; echo "Process data:" ; echo
          lsof $(cat $WORK | sed -n '1d; s/^[^ ]*  *\([^ ]*\).*$/-p \1 /p;') ) | \
        mail -s "Found something on port $PORT"
        echo "Sent message at $(date)..."

# Clean up
rm -f $WORK

Because lsof returns 0 only if it has something to report, this works beautifully. I could have slowed it down with a sleep command, but this worked for my purposes.

It showed a program being run that was part of the firewall. Since it was running periodically, I went and looked for it in the crontab files:

grep program /etc/cron/

I found this program in a file in the /etc/anacron.hourly directory. If I had wanted to, I could have stopped the program from running at all by changing this file. I ran the commands independently of the crontab file to see what the output would be.

I was also able to get help from the program by using the option --help. The program was actually a python script located in /usr/bin, and I searched out the actual code that was called: it was compiled python source (a *.pyc file) found in /usr/lib/python2.4/site-packages/ – the compiled source can be decompiled and investigated.

If I wanted to take complete control, the program could have been renamed and a script put in its place which called the original script and did a little extra – such as report by mail every time the command runs, what the command line was, what the output was, and more.

There’s a lot that can be found out if you just know where to look.