Investigating Mysterious Network Traffic

I discovered in our network a host that was generating a huge amount of traffic compared to every other host in the network. All I had to go on initially was the IP address – so how do we track down the culprit and see what is going on?

First, we have the IP address and access to the network. Thus, my next step was to log into the firewall and sniff the network to see where the traffic is going. That did not prove to be helpful.

However, if we have an IP address, we have a MAC address in the ARP table. There are lots of ways to get this, including the arp command. I used the -e option to tcpdump to get it during the sniff of the network.

If you have a MAC address, you can look up the manufacturer of the network card, which in many cases may be the computer manufacturer. In this case, it was AsusTek. A search of the premises for Asus equipment turned up nothing.

Since this was almost certainly a Windows machine, using nmblookup -A $ip may turn up something useful: in particular, it may return a host identifier or name that can identify its owner. In this case, it turned up a name that had no meaning to me.

The traffic will be viewable at the switch where the equipment is plugged in. Thus, we can go to the switch where all of the workstations are plugged in, and plug a laptop into a port to sniff traffic. Once I had done this, I logged into the web interface, then mirrored 12 of the 24 ports to the port the laptop was in even as the laptop was sniffing the network for the suspect host.

Doing this will send all traffic on those ports also to the port the laptop is listening on – and in this case, turned up the suspect host. Performing a binary search will narrow it down – that is, mirror half (6) of the 12 ports and see if the traffic continues to flow, then mirror half of that (3).

Once I narrowed down the ports to a single port, I tested it. Does the traffic stop when we stop mirroring that port? Once the mirroring of the port was stopped, the network traffic seen by the laptop stopped. Mirroring again resumed the suspect traffic.

Now we know what port it is using. Following the cable from the switch to the patch panel shows which physical outlet is connected, and with the map showing where all of the outlets are we can track the outlet down.

Going to the outlet, I found that there were several devices plugged into a cheap hub. Since there was no one in that office, I pulled each computer’s link to the hub and plugged it into a laptop. This laptop, again, was used to sniff the traffic coming off of the host. A couple of tests and the host was identified.

Next step is to get a virus checker on it and run that to see if anything is running that shouldn’t be.

Investigating Mysterious Outbound TCP Connections

Recently, I had a situation where a firewall had outgoing TCP connections I knew nothing about. If you are to maintain a secure system and a secure network, this sort of thing demands investigation. (I won’t report full details in order to maintain anonymity for various entities.)

Where to start, then? First, use tcpdump and capture the traffic. It may be useful to capture it into a file for looking at with Wireshark. I watched the traffic flow across all interfaces by using tcpdump:

tcpdump -s0 -n -i any host 999.999.999.999

I noticed that there was no traffic coming from anywhere except the outgoing port on the firewall.

Then I became more interested in the IP address being connected to and the port (443 or HTTPS in this case). Connecting to the IP on port 443 didn’t turn up anything interesting (except they used Red Hat Enterprise Linux). Looking up the IP address in a whois listing showed that the IP address was very similar to that of the firewall maker – very interesting indeed. Looking up the IP in reverse DNS showed it to be an Amazon AWS host in Ireland.

Then I wrote a script that used lsof to watch for a connection and find the program making the connection:

#!/bin/bash

WORK=/tmp/work.$$
PORT=":443"

# Prep: erase if present
rm -f $WORK

while true ; do
    if $(lsof -ni $PORT > $WORK) ; then
        ( echo "Found ports open:" ; echo ; cat $WORK ; echo ; echo "Process data:" ; echo
          lsof $(cat $WORK | sed -n '1d; s/^[^ ]*  *\([^ ]*\).*$/-p \1 /p;') ) | \
        mail -s "Found something on port $PORT" me@myhost.example.com
        echo "Sent message at $(date)..."
    fi
done

# Clean up
rm -f $WORK

Because lsof returns 0 only if it has something to report, this works beautifully. I could have slowed it down with a sleep command, but this worked for my purposes.

It showed a program being run that was part of the firewall. Since it was running periodically, I went and looked for it in the crontab files:

grep program /etc/cron/

I found this program in a file in the /etc/anacron.hourly directory. If I had wanted to, I could have stopped the program from running at all by changing this file. I ran the commands independently of the crontab file to see what the output would be.

I was also able to get help from the program by using the option --help. The program was actually a python script located in /usr/bin, and I searched out the actual code that was called: it was compiled python source (a *.pyc file) found in /usr/lib/python2.4/site-packages/ – the compiled source can be decompiled and investigated.

If I wanted to take complete control, the program could have been renamed and a script put in its place which called the original script and did a little extra – such as report by mail every time the command runs, what the command line was, what the output was, and more.

There’s a lot that can be found out if you just know where to look.

Window Manager and xcompmgr

In my current Window Maker configuration, I use xcompmgr to create some nice shadows for the windows. The problem is that xcompmgr sometimes doesn’t realize that a window has closed, and thus the shading remains while the window is actually no longer there. It is possible to “fix” this problem by reopening the window and closing it, but this is ot workable: the only workable work-around is to restart xcompmgr; since xcompmgr is small and fast, this is not a problem.

There are a number of ways to automate this…

Currently, xcompmgr is started in ~/GNUstep/Library/WindowMaker/autostart:

xcompmgr -c -C -r 8 -l -12 -t -12 &

To make it easy to restart xcompmgr, let’s put a program around it that automatically starts the program again if it dies. Since xcompmgr does not go into the background, we can just start it up and when it exits, restart (in a loop).

With Ruby, we can do this:

#!/usr/bin/ruby
#
# xcompmgr2: run xcompmgr and keep it running:
#            permits autorestart of xcompmgr

# Get rid of any programs already running...
`pkill -x xcompmgr`

while (1==1) do

    # Main program: run it!
    `/usr/bin/xcompmgr -c -C -r 8 -l -12 -t -12`

    # Catch and log unusual exitstatus...
    if ($?.exitstatus != 15 && $?.exitstatus != 0) then
        # Not good
    end
end

If xcompmgr exits for any reason, it loops again in the infinite loop. This sort of program could be done in almost any language: certainly Perl, ksh, and PHP are up to the task (as are many others).

Thus, when xcompmgr gets confused, do this command to fix it:

pkill -x xcompmgr

The -x option makes it so that xcompmgr is killed, and not xcompmgr2: -x matches a command name exactly.