Wednesday, February 20, 2013

psad - log unsolicited internet traffic (too much of it)

Psad (Port Scan Attack Detector) is a useful daemon that offers a number of features and is very configurable. I'm always curious to know who has been probing my servers and psad along with some custom logging and other scripts seems to do this well.



Psad Log files
The psad daemon will poll iptables log files. It will then create a directory for each scanning IP in  /var/log/psad/xxx.xxx.xxx.xxx/ and log/lookup some information. The IP  is your IP, you can have psad running on multiple IP addresses (great if they are in different ranges!) which will be logged too.
  • - a summary of all information.
  • - packets (INPUT_eth0_tcp:  51 [606] \n INPUT_eth0_udp:  69 [606]).
  • - first packet (1352391283).
  • XXX.XXX.XXX.XXX_whois - A whois lookup of the attacker.
  • danger_level - Different danger levels are configurable based on the number of packets.
  • email_ctr - Number of email alerts.
  • p0f_guess - Passive fingerprinting guess of OS (@Windows:XP:RFC1323:Windows XP/2000 (RFC1323 no tstamp)).
File system limitations reached
Thinking that I could build up some interesting data over time I configured psad not to remove any logs from old scans and left it running. After a couple of months I hit a limitation in the ext3 filesystem I was using, 31998 subdirectories. After checking ext4 did not offer that much of an advantage.

I settled on creating a ReiserFS partition to hold my psad logs. There is a noticeable performance improvement and it seems to be scaling well . I'm sure there are further tweaks to be done here.

Interesting Sources

 Completely unsolicited network traffic originates from some rather interesting places, for a wide variety of reasons.

A quick search of some terms that I could think of: 
$ find /var/log/psad/ -name "*_whois" -exec egrep -H -i "finance|bank|military|royal|defence|provider|industrial|health|government|\.gov|\.mil|\.aero" -m1 '{}' \;

Some of the results this search yielded:
  • whois: OrgName: Royal Military College of XXXXXX
  • whois: descr: Commercial Bank of XXXXXX
  • whois: descr: XXXXX Nation-wide Network of XXXXbank XX Corp.
  • whois: network: Org-Name; XXXXXHEALTHCARE
  • whois: descr: XXXXXX Internet Service Provider
The "College" may have been looking for sip ports, I logged x2 packets to udp 5060.


  • You can call an external script on a hit and pass the source IP to it. The psad.conf:
EXTERNAL_SCRIPT             /opt/scripts/psad_hai-back.sh SRCIP -v;
### (only once per IP, or every time a scan is detected for an ip).
  • There is also the ability to automatically block (or do anything as you can specify the rules ran) an ip with iptables after a threshold. Real time black listing or redirecting:
### Based on Number of packets.
DANGER_LEVEL1               1;  
DANGER_LEVEL2               10;
DANGER_LEVEL3               100;
DANGER_LEVEL4               500;
DANGER_LEVEL5               1000; 
  • I have also been toying with inotify-tools and Sec (simple event correlation)  for custom logging and information gathering.
More on this coming.

Tuesday, December 4, 2012

manage iptables logs

Iptables default drop and log rules

These rules come after state tracking and all incoming/outgoing rules for specific services.

# Default Incoming traffic log+block
$IPTABLES -A REJECTLOG -j LOG --log-level debug --log-tcp-sequence --log-tcp-options --log-ip-options -m limit --limit 3/s --limit-burst 8 --log-prefix "unsolicited "
# Reject all other incoming traffic:

# default all other outgoing traffic log+block:
$IPTABLES -A OUTPUT -j LOG --log-prefix "bad outgoing " --log-tcp-sequence --log-ip-options --log-tcp-options --log-uid -m limit --limit 2/s --limit-burst 4

Iptables and syslogd
On Debian rsyslog will write the logs from iptables to the following locations: Incoming drops will be logged to debug and outgoing drops to messages. The section in rsyslog.conf that deals with this:

# Some "catch-all" log files.
news.none;mail.none -/var/log/debug

mail,news.none -/var/log/messages

On CentOS I had to add a line in the rsyslog.conf for debug level messages to even be written anywhere:

*.=debug -/var/log/debug

Move noisy iptables drops elsewhere

I like to log all unsolicited traffic to one place, rsyslog (default syslog on CentOS and Debian) will let us filter alerts like this. Having iptables fill up my system logs is not too useful.

$ vim /etc/rsyslog.d/iptables.incoming.conf
:msg, startswith, "unsolicited" -/var/log/iptables.incoming.log
& ~
$ /etc/init.d/rsyslog restart

The first line means send all messages that start with "unsolicited" to iptables.incoming.log. The next line (the: & ~) will discard anything not matched on the previous line. If this does not work for you try using "contains" instead of "startswith".

Log rotate

Add the new iptables log to be managed by log rotate.

$ vim /etc/logrotate.conf
/var/log/iptables.incoming.log {
  create 0660 root root
  rotate 6
  size 2048k
  rotate 1

Useful programs to complement this

Tuesday, November 13, 2012

Obfuscate ssh traffic with obfsproxy

While travelling around the globe I wanted to ensure that I could always SSH into my server.. one way or another.. even if I had to use an ISP that was using using deep packet inspection to fingerprint and block SSH (changing the port is not enough). This is why I deployed obfsproxy, should I encounter such nasty and invasive filtering of SSH traffic.

Compile some software

  • Download Libevent (I'm using 2.0.20-stable), verify the file with pgp then untar it.
  •  ./configure --prefix=/home/user/tools/libevent2/
  • make && make install
  • Download obfsproxy, verify and extract it.
  • export libevent_CFLAGS=-I/home/user/tools/libevent2/include
  • export libevent_LIBS="-L/home/user/tools/libevent2/lib -levent"
  • export LD_LIBRARY_PATH=/home/user/tools/libevent2/lib
  • ./autogen.sh && ./configure && make

Setup obfsproxy on the SSH server

  • Set your sshd to listen on port 22. Any server/client combination can be used like this.
  • Allow Incoming tcp 2222 in your firewall.
  • screen obfsproxy --log-min-severity=info obfs2 --shared-secret=password --dest= server 210.XX.XX.XX:2222

The obfsproxy client

  • The client needs to permit outgoing 2222 tcp if you run a local firewall.
  • screen
  • obfsproxy --log-min-severity=info obfs2 --dest=210.XX.XX.XX:2222 --shared-secret=password client
  • make another tab in screen, proceed:
  • ssh-add /media/mount/key
  • ssh user@ -p 8022

Wednesday, October 10, 2012

Tight iptables rules for outbound DNS

My Iptables script limits which DNS servers can be used and also restricts which user accounts on my Linux server can do lookups.

I don't see firewall scripts utilising iptables owner matching posted that often so I hope this is a sane way of doing things.. . If I'm taking the time to hand craft each rule then I want my firewall to be as limiting as is possible. Without this malicious users on my system could download files over DNS queries. 

Here are the relative excerpts from said bash script in an example (those are google's dns servers):

DNSUSERS="root postfix httpd userperson"

### DNS rules
# outgoing requests for these users
for dnsip in $DNSSERVER
   for dnsuser in $DNSUSERS
$IPTABLES -A OUTPUT -p udp -m udp -d $dnsip --dport 53 \ 
--sport 1024:65535 -s $SERVER_IP -m conntrack \
--ctstate NEW,ESTABLISHED --match owner --uid-owner \ 
$dnsuser -j ACCEPT -m comment --comment "DNS out for $dnsuser "
# incoming established
for dnsip in $DNSSERVER
$IPTABLES -A INPUT -p udp -m udp -s $dnsip --sport 53 \ 
--dport 1024:65535 -d $SERVER_IP \ 
-m conntrack --ctstate ESTABLISHED -j ACCEPT \
-m comment --comment "DNS in "
# Deny all other DNS and log the UID
$IPTABLES -A OUTPUT -p udp --sport 53 -j LOG --log-prefix "NO DNS for user " --log-uid
$IPTABLES -A OUTPUT -p tcp --sport 53 -j LOG --log-prefix "NO DNS for user " --log-uid
$IPTABLES -A OUTPUT -p udp --dport 53 -j DROP
$IPTABLES -A OUTPUT -p tcp --dport 53 -j DROP

Note that in order to use this matching feature of iptables you must have "owner" match support enabled under the Core Netfilter Configuration options of your kernel. Also during my testing I found I needed separate rules for tcp and udp for reliability, this may have been the old version of iptables that was with debian?

I created a user account with a UID of 12346 for testing the outgoing blocking rules, here is what shows up in my syslog when that user tries to make a request:

kernel: NO DNS for user IN= OUT=eth0 SRC=X.X.X.X DST= LEN=75 TOS=0x00 PREC=0x00 TTL=64 ID=28882 DF PROTO=UDP SPT=44654 DPT=53 LEN=55 UID=12346 GID=12346

Thanks to the --log-uid option you can figure out which users are making the requests. This is another option I don't see used all that often (I add --log-uidall outgoing log+drop rules).

Related dns settings:

Set "options rotate" in resolv.conf. There are also some hardening settings available in host.conf (nospoof).

Sunday, September 30, 2012

Notes on operating a Tor relay

Chroot Tor

I found the instructions of building a chrooted Tor server (0.2.2.*) on Debian 6.0.6 (Squeeze) to work fine

Debian stable only has libevent 1 in the repositories, I downloaded and installed 2 from source on my system (./configure && make && make install). I then added the following line to make sure this library could be found:

$ echo "/usr/local/include" >> /etc/ld.so.conf.d/libc.conf

My compile options are a little different from those in the tutorial:

 $ ./configure --prefix=/tor --with-tor-user=debian-tor --with-tor-group=debian-tor \
--enable-gcc-hardening --enable-linker-hardening \

 --enable-static-openssl --with-openssl-dir=/usr/local/ssl \
--enable-static-libevent --with-libevent-dir=/usr/local/include

I'm not sure why using gcc and linker hardening is not shown in the tutorial. I see there is a ticket that has been completed for enabling gcc hardening by default when building 0.2.3.*. This is  currently an issue yet to be resolved for the tor browser bundle.

You can 
use the tool checksec.sh on the compiled Tor binary to show the extra hardening gained (shown in my last blog post).

Pax flags

Kernel hardening from Grsecurity wouldn't let Tor run. From my syslog:

grsec: From 200.XX.XX.XX: denied RWX mmap of by /home/chroot_tor/tor/tor/bin/tor[tor:21382] uid/euid:9050/9050 gid/egid:9050/9050, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0

Compile paxctl (if using a grsec kern) and read the man page, I'm using these flags:

$ paxctl -v /home/chroot_tor/tor/tor/bin/tor
PaX control v0.5
Copyright 2004,2005,2006,2007 PaX Team

- PaX flags: P-S--m-xE-R- [/home/chroot_tor/tor/tor/bin/tor]
        PAGEEXEC is enabled
        SEGMEXEC is enabled
        MPROTECT is disabled
        RANDEXEC is disabled
        EMUTRAMP is enabled
        RANDMMAP is enabled

Enabling MPROTECT only allows one process to start, set "NumCPUs 1" in torrc when using this option. I found that it hammered my performance (very ram hungry) so I've had to disable this option for now, which is a shame.


       month        rx      |     tx      |    total    |   avg. rate
      Oct '11     87.86 GiB |  107.65 GiB |  195.51 GiB |  612.34 kbit/s
      Nov '11     79.31 GiB |  101.79 GiB |  181.10 GiB |  586.12 kbit/s
      Dec '11     90.09 GiB |  110.20 GiB |  200.29 GiB |  627.29 kbit/s
      Jan '12    154.88 GiB |  187.97 GiB |  342.85 GiB |    1.07 Mbit/s
      Feb '12    141.88 GiB |  179.45 GiB |  321.33 GiB |    1.08 Mbit/s
      Mar '12    191.18 GiB |  223.62 GiB |  414.80 GiB |    1.30 Mbit/s
      Apr '12    216.49 GiB |  249.49 GiB |  465.98 GiB |    1.51 Mbit/s
      May '12    160.70 GiB |  198.33 GiB |  359.04 GiB |    1.12 Mbit/s
      Jun '12    141.58 GiB |  172.10 GiB |  313.68 GiB |    1.02 Mbit/s
      Jul '12    156.97 GiB |  190.04 GiB |  347.01 GiB |    1.09 Mbit/s
      Aug '12    171.87 GiB |  199.50 GiB |  371.37 GiB |    1.16 Mbit/s
      Sep '12    124.30 GiB |  146.02 GiB |  270.33 GiB |  903.34 kbit/s
    estimated    128.35 GiB |  150.77 GiB |  279.12 GiB |

Not entirely Tor relay traffic but the vast majority is =). This graph was generated with vnstat, which unfortunately can't do yearly statistics yet. 


Sign for Tor Weather at https://weather.torproject.org/ To receive email when:
  • Your version of Tor is out of date
  • Your router has low bandwidth capacity
  • Are able to claim a Tor t-shirt (conditions)

Friday, September 7, 2012

Chroot dns2tcp for dns tunnelling

I wanted a lightweight DNS tunnelling daemon that would deploy easily. I found that Dns2tcp was just that and I've had good results with this program when testing.

Dns2tcp is a network tool designed to relay TCP connections through DNS traffic. Encapsulation is done on the TCP level, thus no specific driver is needed (i.e: TUN/TAP).


$ curl http://www.hsc.fr/ressources/outils/dns2tcp/download/dns2tcp-0.5.2.tar.gz | tar xzv
$ cd dns2tcp-0.5.2 
$ ./configure &&  make
$ vim server/dns2tcpd/dns2tcpd.conf
listen =
port = 53
user = nobody
chroot = /tmp
domain = sub.server.com
resources = ssh:
key = secretpassphrase111

$ sudo server/dns2tcpd -f server/dns2tcpd.conf -d 3 -F
08:48:03 : Debug options.c:97   Add resource ssh: port 22
08:48:03 : Debug socket.c:55    Listening on for domain sub.server.com
Starting Server v0.5.2...
08:48:03 : Debug main.c:132     Chroot to /tmp
08:48:03 : Debug main.c:142     Change to user nobody

I've downloaded and compiled dns2tcpd on my Debian server, see the README file that comes with the code for more details. After creating a configuration file the server starts fine, listening on the public ip (nothing can connect to it yet because you run a firewall of course).

chrooting the server

Even though the process drops privileges I'm going to build a chroot jail for it [as something to do because why not? grsecurity has chroot hardenig]. Now to see what libraries the process is using. On another console (because the program above is running in the foreground):

$ sudo su
$ ps aux | grep dns2tcp -m1
nobody   21520  0.0  0.1  41864   824 pts/8    S+   02:01   0:00 server/dns2tcpd -f server/dns2tcpd.conf -d 3 -F

$ lsof -p 21520
dns2tcpd 21520 nobody  cwd    DIR      202,0     4096     12 /tmp
dns2tcpd 21520 nobody  rtd    DIR      202,0     4096     12 /tmp
dns2tcpd 21520 nobody  txt    REG      202,0   138934 165243 /home/user/source/dns2tcp-0.5.2/server/dns2tcpd
dns2tcpd 21520 nobody  mem    REG      202,0    71432  92053 /lib/libresolv-2.11.3.so
dns2tcpd 21520 nobody  mem    REG      202,0    22036  92068 /lib/libnss_dns-2.11.3.so
dns2tcpd 21520 nobody  mem    REG      202,0    42580  92054 /lib/libnss_files-2.11.3.so
dns2tcpd 21520 nobody  mem    REG      202,0    38504  92047 /lib/libnss_nis-2.11.3.so
dns2tcpd 21520 nobody  mem    REG      202,0    79676  92051 /lib/libnsl-2.11.3.so
dns2tcpd 21520 nobody  mem    REG      202,0    30496  92069 /lib/libnss_compat-2.11.3.so
dns2tcpd 21520 nobody  mem    REG      202,0  1319176  92067 /lib/libc-2.11.3.so
dns2tcpd 21520 nobody  mem    REG      202,0   118060  92060 /lib/ld-2.11.3.so
dns2tcpd 21520 nobody    0u   CHR      136,8      0t0     11 /dev/pts/8
dns2tcpd 21520 nobody    1u   CHR      136,8      0t0     11 /dev/pts/8
dns2tcpd 21520 nobody    2u   CHR      136,8      0t0     11 /dev/pts/8
$ kill -9 21520

Building the chroot:

$ mkdir -p /opt/chroot_dns2tcp/dns2tcpd/{etc,tmp,dev,usr,lib}
$ mkdir  /opt/chroot_dns2tcp/dns2tcpd/usr/dns2tcpd/
$ mknod /opt/chroot_dns2tcpd/dns2tcpd/dev/null c 1 3
$ cp /lib/ld-linux.so.2 /lib/libc.so.6 /lib/libnss_dns.so.2 /lib/libnss_files.so.2 /lib/libnss_nis.so.2 /lib/libresolv.so.2 /opt/chroot_dns2tcp/dns2tcpd/lib/
$ cp /home/user/source/dns2tcp-0.5.2/server/dns2tcpd /home/user/source/dns2tcp-0.5.2/server/dns2tcpd.conf /opt/chroot_dns2tcp/dns2tcpd/usr/dns2tcpd/ -v
$ cp /etc/hosts /etc/localtime /opt/chroot_dns2tcp/dns2tcpd/etc/ -v

$ touch /opt/chroot_dns2tcp/dns2tcpd/etc/group

$ echo "nobody:x:65534:65534:nobody:/dev/null:/bin/false" >> /opt/chroot_dns2tcp/dns2tcpd/etc/group

$ touch /opt/chroot_dns2tcp/dns2tcpd/etc/passwd

$ echo "nobody:x:65534:65534:nobody:/dev/null:/bin/false" >> /opt/chroot_dns2tcp/dns2tcpd/etc/passwd

$ touch /opt/chroot_dns2tcpd/daemon.log

$ touch /opt/chroot_dns2tcpd/screen.rc

$ vim /opt/chroot_dns2tcp/dns2tcpd/etc/nsswitch.conf 

passwd:         files
group:          files
shadow:         files
hosts:          files dns
networks:       files
protocols:      db files
services:       db files
ethers:         db files
rpc:            db files
netgroup:       nis


If you use the PaX patch on your Linux kernel you have more hardening options:

$ pax -MRXSPE /opt/chroot_dns2tcp/dns2tcpd/usr/dns2tcpd/dns2tcpd
$ paxctl -v /opt/chroot_dns2tcpd/dns2tcpd/usr/local/dns2tcpd/dns2tcpd
PaX control v0.5
Copyright 2004,2005,2006,2007 PaX Team

- PaX flags: P-S-M-X-E-R- [/opt/chroot_dns2tcpd/dns2tcpd/usr/local/dns2tcpd/dns2tcpd]
        PAGEEXEC is enabled
        SEGMEXEC is enabled
        MPROTECT is enabled
        RANDEXEC is enabled
        EMUTRAMP is enabled
        RANDMMAP is enabled

Using checksec.sh I found that further hardening can still be implemented (listed below). There has been a remote buffer overflow vulnerability in dns2tcp before - CVE-2008-03910.

$ ./checksec.sh --file /opt/chroot_dns2tcpd/dns2tcpd/usr/local/dns2tcpd/dns2tcpd
RELRO           STACK CANARY      NX            PIE             RPATH      RUNPATH      FILE
No RELRO        No canary found   NX enabled    No PIE          No RPATH   No RUNPATH   /opt/chroot_dns2tcpd/dns2tcpd/usr/local/dns2tcpd/dns2tcpd
File permissions:

$ chmod o-rwx /opt/chroot_dns2tcpd/ -R -v
$ chmod -w /opt/chroot_dns2tcpd/dns2tcpd/ -R -v
$ chattr +i /opt/chroot_dns2tcpd/dns2tcpd/* -R -v
$ chattr +a /opt/chroot_dns2tcpd/daemon.log

Starting the server

This command will start the server and log the output. I start the server from inside of screen so I can logout from ssh and leave the process running. Here the screen config set by -c is blank because I can't yet figure out how to stop screen from loading my regular .screenrc on startup.

screen -S dns2tcp -c /opt/chroot_dns2tcpd/screen.rc sh -c "/usr/sbin/chroot /opt/chroot_dns2tcpd/dns2tcpd /usr/local/dns2tcpd/dns2tcpd -f /usr/local/dns2tcpd/dns2tcpd.conf -d1 -F 2>&1 | tee -a  /opt/chroot_dns2tcpd/daemon.log"
02:02:46 : Debug options.c:97   Add resource ssh: port 22
02:02:46 : Debug socket.c:55    Listening on for domain  sub.server.com
Starting Server v0.5.2...
02:02:46 : Debug main.c:132     Chroot to /tmp
09:02:46 : Debug main.c:142     Change to user nobody

When the server starts I get an message in my syslog from grsecurity because I have enabled logging of exec's within chroots. I've also utilised the socket restrictions feature and blocked client sockets for the user nobody.

$ dmesg | grep -m1 dns2tcpd
grsec: From 100.XX.XXX.XXX: exec of /opt/chroot_dns2tcpd/dns2tcpd/usr/local/dns2tcpd/dns2tcpd within chroot by process /opt/chroot_dns2tcpd/dns2tcpd/usr/local/dns2tcpd/dns2tcpd[chroot:18125] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:6016] uid/euid:0/0 gid/egid:0/0


On your local linux machine download and compile the code.

$ vim /home/laptop/.ssh/config
Host dnstunnel
User user
Port 2828
IdentityFile /home/laptop/.ssh/private-key-file
LogLevel  QUIET
DynamicForward localhost:8080

$ dns2tcpc -d 3 -z  sub.server.com -k secretpassphrase111 -r ssh -l 2828
$ ssh dnstunnel

Have now connected to the ssh daemon listening on localhost of the remote server, with all of the traffic being sent/received over dns queries. 

Creating session id: 0x8gd7 address = (compression NOT wanted)
delete_client 0x8gd7 

The dns2tcp daemon will show clients connecting and disconnecting. The IP address shown here belongs to google because as you see above out client is using, one of googles DNS servers. If you are using this behind a captive portal and don't specify a DNS server the client will look in resolv.conf.