manage iptables logs

Iptables default drop and log rules

These rules come after state tracking and all incoming/outgoing rules for specific services.

# Default Incoming traffic log+block
$IPTABLES -N REJECTLOG
$IPTABLES -A REJECTLOG -j LOG --log-level debug --log-tcp-sequence --log-tcp-options --log-ip-options -m limit --limit 3/s --limit-burst 8 --log-prefix "unsolicited "
$IPTABLES -A REJECTLOG -j DROP
# Reject all other incoming traffic:
$IPTABLES -A INPUT -j REJECTLOG

# default all other outgoing traffic log+block:
$IPTABLES -A OUTPUT -j LOG --log-prefix "bad outgoing " --log-tcp-sequence --log-ip-options --log-tcp-options --log-uid -m limit --limit 2/s --limit-burst 4
$IPTABLES -A OUTPUT -j DROP

Iptables and syslogd

On Debian rsyslog will write the logs from iptables to the following locations: Incoming drops will be logged to debug and outgoing drops to messages. The section in rsyslog.conf that deals with this:

# Some "catch-all" log files.
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug

*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages

On CentOS I had to add a line in the rsyslog.conf for debug level messages to even be written anywhere:

*.=debug -/var/log/debug

Move noisy iptables drops elsewhere

I like to log all unsolicited traffic to one place, rsyslog (default syslog on CentOS and Debian) will let us filter alerts like this. Having iptables fill up my system logs is not too useful.

$ vim /etc/rsyslog.d/iptables.incoming.conf
:msg, startswith, "unsolicited" -/var/log/iptables.incoming.log
& ~
$ /etc/init.d/rsyslog restart

The first line means send all messages that start with "unsolicited" to iptables.incoming.log. The next line (the: & ~) will discard anything not matched on the previous line. If this does not work for you try using "contains" instead of "startswith".

Log rotate

Add the new iptables log to be managed by log rotate.

$ vim /etc/logrotate.conf
/var/log/iptables.incoming.log {
  weekly
  create 0660 root root
  rotate 6
  size 2048k
  rotate 1
}

Useful programs to complement this

• Psad (port scan attack detector) offers more logging, lookups on attackers and responding: http://cipherdyne.org/psad/
• fwlogwatch is a useful packet filter / firewall / IDS log analyzer. Can build nice html summary's:  http://fwlogwatch.inside-security.de/
• A port knocking program like knockknock: http://www.thoughtcrime.org/software/knockknock/

Obfuscate ssh traffic with obfsproxy

While travelling around the globe I wanted to ensure that I could always SSH into my server.. one way or another.. even if I had to use an ISP that was using using deep packet inspection to fingerprint and block SSH (changing the port is not enough). This is why I deployed obfsproxy, should I encounter such nasty and invasive filtering of SSH traffic.

Compile some software

Libevent

• Download Libevent (I'm using 2.0.20-stable), verify the file with pgp then untar it.
•  ./configure --prefix=/home/user/tools/libevent2/
• make && make install

obfsproxy

• Download obfsproxy, verify and extract it.
• export libevent_CFLAGS=-I/home/user/tools/libevent2/include
• export libevent_LIBS="-L/home/user/tools/libevent2/lib -levent"
• export LD_LIBRARY_PATH=/home/user/tools/libevent2/lib
• ./autogen.sh && ./configure && make

Setup obfsproxy on the SSH server

• Set your sshd to listen on 127.0.0.1 port 22. Any server/client combination can be used like this.
• Allow Incoming tcp 2222 in your firewall.
• screen obfsproxy --log-min-severity=info obfs2 --shared-secret=password --dest=127.0.0.1:22 server 210.XX.XX.XX:2222

The obfsproxy client

• The client needs to permit outgoing 2222 tcp if you run a local firewall.
• screen
• obfsproxy --log-min-severity=info obfs2 --dest=210.XX.XX.XX:2222 --shared-secret=password client 127.0.0.1:8022
• make another tab in screen, proceed:
• ssh-add /media/mount/key
• ssh user@127.0.0.1 -p 8022

Tight iptables rules for outbound DNS

My Iptables script limits which DNS servers can be used and also restricts which user accounts on my Linux server can do lookups.

I don't see firewall scripts utilising iptables owner matching posted that often so I hope this is a sane way of doing things.. . If I'm taking the time to hand craft each rule then I want my firewall to be as limiting as is possible. Without this malicious users on my system could download files over DNS queries. 

Here are the relative excerpts from said bash script in an example (those are google's dns servers):

SERVER_IP="xx.xx.xx.xx"
DNSSERVER="8.8.4.4 8.8.8.8"
DNSUSERS="root postfix httpd userperson"
IPTABLES=/usr/local/sbin/iptables

### DNS rules
# outgoing requests for these users
for dnsip in $DNSSERVER
  do
   for dnsuser in $DNSUSERS
    do
$IPTABLES -A OUTPUT -p udp -m udp -d $dnsip --dport 53 \ 
--sport 1024:65535 -s $SERVER_IP -m conntrack \
--ctstate NEW,ESTABLISHED --match owner --uid-owner \ 
$dnsuser -j ACCEPT -m comment --comment "DNS out for $dnsuser "
     done
   done
# incoming established
for dnsip in $DNSSERVER
  do
$IPTABLES -A INPUT -p udp -m udp -s $dnsip --sport 53 \ 
--dport 1024:65535 -d $SERVER_IP \ 
-m conntrack --ctstate ESTABLISHED -j ACCEPT \
-m comment --comment "DNS in "
done
# Deny all other DNS and log the UID
$IPTABLES -A OUTPUT -p udp --sport 53 -j LOG --log-prefix "NO DNS for user " --log-uid
$IPTABLES -A OUTPUT -p tcp --sport 53 -j LOG --log-prefix "NO DNS for user " --log-uid
$IPTABLES -A OUTPUT -p udp --dport 53 -j DROP
$IPTABLES -A OUTPUT -p tcp --dport 53 -j DROP

Note that in order to use this matching feature of iptables you must have "owner" match support enabled under the Core Netfilter Configuration options of your kernel. Also during my testing I found I needed separate rules for tcp and udp for reliability, this may have been the old version of iptables that was with debian?

I created a user account with a UID of 12346 for testing the outgoing blocking rules, here is what shows up in my syslog when that user tries to make a request:

kernel: NO DNS for user IN= OUT=eth0 SRC=X.X.X.X DST=8.8.8.8 LEN=75 TOS=0x00 PREC=0x00 TTL=64 ID=28882 DF PROTO=UDP SPT=44654 DPT=53 LEN=55 UID=12346 GID=12346

Thanks to the --log-uid option you can figure out which users are making the requests. This is another option I don't see used all that often (I add --log-uidall outgoing log+drop rules).

Related dns settings:

Set "options rotate" in resolv.conf. There are also some hardening settings available in host.conf (nospoof).

Notes on operating a Tor relay

Chroot Tor

I found the instructions of building a chrooted Tor server (0.2.2.*) on Debian 6.0.6 (Squeeze) to work fine

https://trac.torproject.org/projects/tor/wiki/TheOnionRouter/TorInChroot

Libevent2
Debian stable only has libevent 1 in the repositories, I downloaded and installed 2 from source on my system (./configure && make && make install). I then added the following line to make sure this library could be found:

$ echo "/usr/local/include" >> /etc/ld.so.conf.d/libc.conf

Tor
My compile options are a little different from those in the tutorial:

 $ ./configure --prefix=/tor --with-tor-user=debian-tor --with-tor-group=debian-tor \
--enable-gcc-hardening --enable-linker-hardening \
 --enable-static-openssl --with-openssl-dir=/usr/local/ssl \
--enable-static-libevent --with-libevent-dir=/usr/local/include

I'm not sure why using gcc and linker hardening is not shown in the tutorial. I see there is a ticket that has been completed for enabling gcc hardening by default when building 0.2.3.*. This is  currently an issue yet to be resolved for the tor browser bundle.

You can use the tool checksec.sh on the compiled Tor binary to show the extra hardening gained (shown in my last blog post).

Pax flags

Kernel hardening from Grsecurity wouldn't let Tor run. From my syslog:

grsec: From 200.XX.XX.XX: denied RWX mmap of by /home/chroot_tor/tor/tor/bin/tor[tor:21382] uid/euid:9050/9050 gid/egid:9050/9050, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0

Compile paxctl (if using a grsec kern) and read the man page, I'm using these flags:

$ paxctl -v /home/chroot_tor/tor/tor/bin/tor
PaX control v0.5
Copyright 2004,2005,2006,2007 PaX Team

- PaX flags: P-S--m-xE-R- [/home/chroot_tor/tor/tor/bin/tor]
        PAGEEXEC is enabled
        SEGMEXEC is enabled
        MPROTECT is disabled
        RANDEXEC is disabled
        EMUTRAMP is enabled
        RANDMMAP is enabled

Enabling MPROTECT only allows one process to start, set "NumCPUs 1" in torrc when using this option. I found that it hammered my performance (very ram hungry) so I've had to disable this option for now, which is a shame.

Traffic

       month        rx      |     tx      |    total    |   avg. rate
    ------------------------+-------------+-------------+---------------
      Oct '11     87.86 GiB |  107.65 GiB |  195.51 GiB |  612.34 kbit/s
      Nov '11     79.31 GiB |  101.79 GiB |  181.10 GiB |  586.12 kbit/s
      Dec '11     90.09 GiB |  110.20 GiB |  200.29 GiB |  627.29 kbit/s
      Jan '12    154.88 GiB |  187.97 GiB |  342.85 GiB |    1.07 Mbit/s
      Feb '12    141.88 GiB |  179.45 GiB |  321.33 GiB |    1.08 Mbit/s
      Mar '12    191.18 GiB |  223.62 GiB |  414.80 GiB |    1.30 Mbit/s
      Apr '12    216.49 GiB |  249.49 GiB |  465.98 GiB |    1.51 Mbit/s
      May '12    160.70 GiB |  198.33 GiB |  359.04 GiB |    1.12 Mbit/s
      Jun '12    141.58 GiB |  172.10 GiB |  313.68 GiB |    1.02 Mbit/s
      Jul '12    156.97 GiB |  190.04 GiB |  347.01 GiB |    1.09 Mbit/s
      Aug '12    171.87 GiB |  199.50 GiB |  371.37 GiB |    1.16 Mbit/s
      Sep '12    124.30 GiB |  146.02 GiB |  270.33 GiB |  903.34 kbit/s
    ------------------------+-------------+-------------+---------------
    estimated    128.35 GiB |  150.77 GiB |  279.12 GiB |

Not entirely Tor relay traffic but the vast majority is =). This graph was generated with vnstat, which unfortunately can't do yearly statistics yet. 

Monitoring

Sign for Tor Weather at https://weather.torproject.org/ To receive email when:

• Your version of Tor is out of date
• Your router has low bandwidth capacity
• Are able to claim a Tor t-shirt (conditions)

Chroot dns2tcp for dns tunnelling

I wanted a lightweight DNS tunnelling daemon that would deploy easily. I found that Dns2tcp was just that and I've had good results with this program when testing.

Dns2tcp is a network tool designed to relay TCP connections through DNS traffic. Encapsulation is done on the TCP level, thus no specific driver is needed (i.e: TUN/TAP).
http://www.hsc.fr/ressources/outils/dns2tcp/index.html.en

Server

$ curl http://www.hsc.fr/ressources/outils/dns2tcp/download/dns2tcp-0.5.2.tar.gz | tar xzv
$ cd dns2tcp-0.5.2 
$ ./configure &&  make
$ vim server/dns2tcpd/dns2tcpd.conf
listen = 11.22.22.11
port = 53
user = nobody
chroot = /tmp
domain = sub.server.com
resources = ssh:127.0.0.1:22
key = secretpassphrase111

$ sudo server/dns2tcpd -f server/dns2tcpd.conf -d 3 -F
08:48:03 : Debug options.c:97   Add resource ssh:127.0.0.1 port 22
08:48:03 : Debug socket.c:55    Listening on 50.116.4.114:53 for domain sub.server.com
Starting Server v0.5.2...
08:48:03 : Debug main.c:132     Chroot to /tmp
08:48:03 : Debug main.c:142     Change to user nobody

I've downloaded and compiled dns2tcpd on my Debian server, see the README file that comes with the code for more details. After creating a configuration file the server starts fine, listening on the public ip 11.22.22.11 (nothing can connect to it yet because you run a firewall of course).

chrooting the server

Even though the process drops privileges I'm going to build a chroot jail for it [as something to do because why not? grsecurity has chroot hardenig]. Now to see what libraries the process is using. On another console (because the program above is running in the foreground):

$ sudo su
$ ps aux | grep dns2tcp -m1
nobody   21520  0.0  0.1  41864   824 pts/8    S+   02:01   0:00 server/dns2tcpd -f server/dns2tcpd.conf -d 3 -F

$ lsof -p 21520

COMMAND    PID   USER   FD   TYPE     DEVICE SIZE/OFF   NODE NAME

dns2tcpd 21520 nobody  cwd    DIR      202,0     4096     12 /tmp

dns2tcpd 21520 nobody  rtd    DIR      202,0     4096     12 /tmp

dns2tcpd 21520 nobody  txt    REG      202,0   138934 165243 /home/user/source/dns2tcp-0.5.2/server/dns2tcpd

dns2tcpd 21520 nobody  mem    REG      202,0    71432  92053 /lib/libresolv-2.11.3.so

dns2tcpd 21520 nobody  mem    REG      202,0    22036  92068 /lib/libnss_dns-2.11.3.so

dns2tcpd 21520 nobody  mem    REG      202,0    42580  92054 /lib/libnss_files-2.11.3.so

dns2tcpd 21520 nobody  mem    REG      202,0    38504  92047 /lib/libnss_nis-2.11.3.so

dns2tcpd 21520 nobody  mem    REG      202,0    79676  92051 /lib/libnsl-2.11.3.so

dns2tcpd 21520 nobody  mem    REG      202,0    30496  92069 /lib/libnss_compat-2.11.3.so

dns2tcpd 21520 nobody  mem    REG      202,0  1319176  92067 /lib/libc-2.11.3.so

dns2tcpd 21520 nobody  mem    REG      202,0   118060  92060 /lib/ld-2.11.3.so

dns2tcpd 21520 nobody    0u   CHR      136,8      0t0     11 /dev/pts/8

dns2tcpd 21520 nobody    1u   CHR      136,8      0t0     11 /dev/pts/8

dns2tcpd 21520 nobody    2u   CHR      136,8      0t0     11 /dev/pts/8

$ kill -9 21520

Building the chroot:

$ mkdir -p /opt/chroot_dns2tcp/dns2tcpd/{etc,tmp,dev,usr,lib}

$ mkdir  /opt/chroot_dns2tcp/dns2tcpd/usr/dns2tcpd/

$ mknod /opt/chroot_dns2tcpd/dns2tcpd/dev/null c 1 3
$ cp /lib/ld-linux.so.2 /lib/libc.so.6 /lib/libnss_dns.so.2 /lib/libnss_files.so.2 /lib/libnss_nis.so.2 /lib/libresolv.so.2 /opt/chroot_dns2tcp/dns2tcpd/lib/

$ cp /home/user/source/dns2tcp-0.5.2/server/dns2tcpd /home/user/source/dns2tcp-0.5.2/server/dns2tcpd.conf /opt/chroot_dns2tcp/dns2tcpd/usr/dns2tcpd/ -v
$ cp /etc/hosts /etc/localtime /opt/chroot_dns2tcp/dns2tcpd/etc/ -v

$ touch /opt/chroot_dns2tcp/dns2tcpd/etc/group
$ echo "nobody:x:65534:65534:nobody:/dev/null:/bin/false" >> /opt/chroot_dns2tcp/dns2tcpd/etc/group

$ touch /opt/chroot_dns2tcp/dns2tcpd/etc/passwd
$ echo "nobody:x:65534:65534:nobody:/dev/null:/bin/false" >> /opt/chroot_dns2tcp/dns2tcpd/etc/passwd

$ touch /opt/chroot_dns2tcpd/daemon.log
$ touch /opt/chroot_dns2tcpd/screen.rc

$ vim /opt/chroot_dns2tcp/dns2tcpd/etc/nsswitch.conf 
passwd:         files

group:          files
shadow:         files
hosts:          files dns
networks:       files
protocols:      db files
services:       db files
ethers:         db files
rpc:            db files
netgroup:       nis

Hardening:

If you use the PaX patch on your Linux kernel you have more hardening options:

$ pax -MRXSPE /opt/chroot_dns2tcp/dns2tcpd/usr/dns2tcpd/dns2tcpd

$ paxctl -v /opt/chroot_dns2tcpd/dns2tcpd/usr/local/dns2tcpd/dns2tcpd
PaX control v0.5
Copyright 2004,2005,2006,2007 PaX Team

- PaX flags: P-S-M-X-E-R- [/opt/chroot_dns2tcpd/dns2tcpd/usr/local/dns2tcpd/dns2tcpd]
        PAGEEXEC is enabled
        SEGMEXEC is enabled
        MPROTECT is enabled
        RANDEXEC is enabled
        EMUTRAMP is enabled
        RANDMMAP is enabled

Using checksec.sh I found that further hardening can still be implemented (listed below). There has been a remote buffer overflow vulnerability in dns2tcp before - CVE-2008-03910.

$ ./checksec.sh --file /opt/chroot_dns2tcpd/dns2tcpd/usr/local/dns2tcpd/dns2tcpd
RELRO           STACK CANARY      NX            PIE             RPATH      RUNPATH      FILE
No RELRO        No canary found   NX enabled    No PIE          No RPATH   No RUNPATH   /opt/chroot_dns2tcpd/dns2tcpd/usr/local/dns2tcpd/dns2tcpd

• RELRO hardening: http://tk-blog.blogspot.com/2009/02/relro-not-so-well-known-memory.html
• Stack Canaries: http://en.wikipedia.org/wiki/Buffer_overflow_protection#Canaries
• position-independent executable - http://en.wikipedia.org/wiki/Position-independent_executable

File permissions:

$ chmod o-rwx /opt/chroot_dns2tcpd/ -R -v
$ chmod -w /opt/chroot_dns2tcpd/dns2tcpd/ -R -v
$ chattr +i /opt/chroot_dns2tcpd/dns2tcpd/* -R -v
$ chattr +a /opt/chroot_dns2tcpd/daemon.log

Starting the server

This command will start the server and log the output. I start the server from inside of screen so I can logout from ssh and leave the process running. Here the screen config set by -c is blank because I can't yet figure out how to stop screen from loading my regular .screenrc on startup.

$ screen -S dns2tcp -c /opt/chroot_dns2tcpd/screen.rc sh -c "/usr/sbin/chroot /opt/chroot_dns2tcpd/dns2tcpd /usr/local/dns2tcpd/dns2tcpd -f /usr/local/dns2tcpd/dns2tcpd.conf -d1 -F 2>&1 | tee -a  /opt/chroot_dns2tcpd/daemon.log"
02:02:46 : Debug options.c:97   Add resource ssh:127.0.0.1 port 22
02:02:46 : Debug socket.c:55    Listening on 11.22.22.11:53 for domain  sub.server.com
Starting Server v0.5.2...
02:02:46 : Debug main.c:132     Chroot to /tmp
09:02:46 : Debug main.c:142     Change to user nobody

When the server starts I get an message in my syslog from grsecurity because I have enabled logging of exec's within chroots. I've also utilised the socket restrictions feature and blocked client sockets for the user nobody.

$ dmesg | grep -m1 dns2tcpd

grsec: From 100.XX.XXX.XXX: exec of /opt/chroot_dns2tcpd/dns2tcpd/usr/local/dns2tcpd/dns2tcpd within chroot by process /opt/chroot_dns2tcpd/dns2tcpd/usr/local/dns2tcpd/dns2tcpd[chroot:18125] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:6016] uid/euid:0/0 gid/egid:0/0

Client

On your local linux machine download and compile the code.

$ vim /home/laptop/.ssh/config
Host dnstunnel
User user
Port 2828
HostName 127.0.0.1
IdentityFile /home/laptop/.ssh/private-key-file
LogLevel  QUIET
DynamicForward localhost:8080

$ dns2tcpc -d 3 -z  sub.server.com -k secretpassphrase111 -r ssh -l 2828 8.8.8.8
$ ssh dnstunnel

Have now connected to the ssh daemon listening on localhost of the remote server, with all of the traffic being sent/received over dns queries. 

Creating session id: 0x8gd7 address = 202.180.64.10 (compression NOT wanted)
delete_client 0x8gd7 

The dns2tcp daemon will show clients connecting and disconnecting. The IP address shown here belongs to google because as you see above out client is using 8.8.8.8, one of googles DNS servers. If you are using this behind a captive portal and don't specify a DNS server the client will look in resolv.conf.

Custom Debian kernel with grsecurity on Linode

There are good instructions on the Linode Library here on how to compile and use your own Kernel, but I do things slightly differently using the grsecurity hardening patch and build a .deb (Debian) package of my kernel.

$ wget http://grsecurity.net/test/grsecurity-2.9.1-3.5.2-201208151951.patch
$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.5.2.tar.gz
$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.5.2.tar.gz.sign
$ gpg --verify linux-3.5.2.tar.gz.sign linux-3.5.2.tar.gz
$ tar -xf linux-3.5.2.tar.gz
$ cd linux-3.5.2
$ patch -p1 ../grsecurity-2.9.1-3.5.2-201208151951.patch
$ sudo cp /boot/config-`uname -r`.config
$ make menuconfig

Make any changes to your kernel configuration and then save them. I have disabled loadable module support (not something I require on a server + the security considerations) and a couple of other features (NFS), even so the kernels Linode offer are pretty lean. 

Under grsecurity I have enabled mostly all of the settings without any issues, read about those config options here. Depends on what your requirements are as to how suitable some of these options are.

How I had been compiling:

$ make-kpkg --rootcmd fakeroot kernel_image kernel_headers --initrd --revision=grsec.352
$ cd ..
$ sudo dpkg -i linux-headers-3.5.2-grsec_352_i386.deb linux-image-3.5.2-grsec_352_i386.deb
$ update-initramfs -c -k 3.5.2

Linode build notes:

$ make -j3 bzImage
$ make -j3 modules
$ make
$ make install
$ make modules_install


Check /boot/grub/menu.lst and add the new kernel if it's not there (it should be).

timeout 2

title kernel 3.5.2
root (hd0)
kernel /boot/vmlinuz-3.5.2-grsec root=/dev/xvda ro 
initrd /boot/initrd.img-3.5.2-grsec

Restart the server and all going well it should boot up using the new kernel.

$ uname -a
Linux hostname 3.5.2-grsec #1 SMP Sun Aug 19 01:36:55 PDT 2012 i686 GNU/Linux

Updated: other brief relevant notes:

• Add barrier=0 to your fstab file.
• To stop all the page allocation errors I was getting In sysctl.conf I had to add vm.min_free_kbytes = 5120
• If you get this error when booting:
  close blk: backend at /local/domain/0/backend/vbd/6401/51712
  Then disable "Sanitize kernel stack" in Grsecurity.

Problems with removing the kernel

If you have /var mounted with noexec then when you apt-get remove your-compiled-kern you will get an error message because the post-remove scripts will be unable execute.

If things get really broken you can remove your kernel this ugly way: 

• manually delete the scripts which are stored in /var/lib/dpkg/info/ 
• remove the files for that particular kernel in /boot/ (remember to edit /boot/grub/menu.lst to reflect your changes)
• run: dpkg --purge linux-image-x

CentOS serial console access

I have my old Laptop plugged into my local [headless] server with a RS-232 serial cable for convenient access when things go wrong (booting new kernels and toying with network settings).

Client
On the Laptop I use Minicom for a client, but gtkterm is a good graphical alternative. Run minicom with -s for the first time to configure the serial port:

$ dmesg | grep tty
[    0.000000] console [tty0] enabled
[    1.390457] 00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
$ apt-get install minicom
$ minicom -s

Serial device: /dev/ttyS0
Bps/Par/Bits: 9600 8N1
Hardware Flow Control: Yes
Software Flow Control: no

Server

Found that CentOS 6.3 has no access on the serial port by default.

[root@supermicro ~]# dmesg | grep tty
console [tty1] enabled
console [ttyS0] enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:07: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A

Above we find out what ports we have on the server. Can now edit the /etc/inittab file (below) to attach a getty process to the serial port, this will allow us to login over the serial cable.

id:3:initdefault:
1:23:respawn:/sbin/agetty -h -L ttyS0 9600 linux

Grub
Edit /boot/grub/grub.conf to output grub on the serial port :

default=0
timeout=5
serial --unit=1 --speed=9600
terminal serial console --timeout=10

title CentOS (XXXXXXXXXX.el6.x86_64)
root (hd0,0)

kernel /vmlinuz-XXXXXXXXXX.el6.x86_64 ro root=/dev/mapper/VolGroup-lv_root nomodeset rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=VolGroup/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM vga=795 console=tty1 console=ttyS0,9600

initrd /initramfs-2.6.32-279.2.1.el6.x86_64.img

Now after rebooting the CentOS server I can use the serial cable to select which kernel I want to use, watch it boot and then login - without a monitor or working network connection.