Recently, I had a situation where a firewall had outgoing TCP connections I knew nothing about. If you are to maintain a secure system and a secure network, this sort of thing demands investigation. (I won’t report full details in order to maintain anonymity for various entities.)
Where to start, then? First, use tcpdump and capture the traffic. It may be useful to capture it into a file for looking at with Wireshark. I watched the traffic flow across all interfaces by using tcpdump:
tcpdump -s0 -n -i any host 999.999.999.999
I noticed that there was no traffic coming from anywhere except the outgoing port on the firewall.
Then I became more interested in the IP address being connected to and the port (443 or HTTPS in this case). Connecting to the IP on port 443 didn’t turn up anything interesting (except they used Red Hat Enterprise Linux). Looking up the IP address in a whois listing showed that the IP address was very similar to that of the firewall maker – very interesting indeed. Looking up the IP in reverse DNS showed it to be an Amazon AWS host in Ireland.
Then I wrote a script that used lsof to watch for a connection and find the program making the connection:
#!/bin/bash WORK=/tmp/work.$$ PORT=":443" # Prep: erase if present rm -f $WORK while true ; do if $(lsof -ni $PORT > $WORK) ; then ( echo "Found ports open:" ; echo ; cat $WORK ; echo ; echo "Process data:" ; echo lsof $(cat $WORK | sed -n '1d; s/^[^ ]* *\([^ ]*\).*$/-p \1 /p;') ) | \ mail -s "Found something on port $PORT" firstname.lastname@example.org echo "Sent message at $(date)..." fi done # Clean up rm -f $WORK
Because lsof returns 0 only if it has something to report, this works beautifully. I could have slowed it down with a sleep command, but this worked for my purposes.
It showed a program being run that was part of the firewall. Since it was running periodically, I went and looked for it in the crontab files:
grep program /etc/cron/
I found this program in a file in the /etc/anacron.hourly directory. If I had wanted to, I could have stopped the program from running at all by changing this file. I ran the commands independently of the crontab file to see what the output would be.
I was also able to get help from the program by using the option
--help. The program was actually a python script located in /usr/bin, and I searched out the actual code that was called: it was compiled python source (a *.pyc file) found in
/usr/lib/python2.4/site-packages/ – the compiled source can be decompiled and investigated.
If I wanted to take complete control, the program could have been renamed and a script put in its place which called the original script and did a little extra – such as report by mail every time the command runs, what the command line was, what the output was, and more.
There’s a lot that can be found out if you just know where to look.
One thought on “Investigating Mysterious Outbound TCP Connections”
These types of investigations are one of the reasons I like my job so much.
And, for me, they read like a mystery novel 🙂