pfSense bridge gateway vmware ovh ip failover ripe

This post is under construction.

This post is about the setup of a pfSense 2.0 firewall in transparent mode to connect and protect virtual machines hosted by OVH to the internet.

OVH provides failover IPs (fr) and RIPE IP Failover block (fr). These IPs and blocks can be re-routed between servers you are renting inside an OVH datacenter in 1min.
This make migration, failover and HA easy and at a low cost, no need to change something in the DNS or in the VM setup.

The setup and the use of these IPs is not easy. Adding a firewall in between don't help.To not increase the complexity of this post, the firewall is configured in transparent mode. This means that interfaces of the firewall are bridged.

What is a bridge ?

A network bridge fits between two networks and repeats Ethernet frames on one side to the other side and vice versa. A bridge works at layer 2 (Ethernet) of the OSI model (IP is layer 3) and ignores the IP settings. All type of Ethernet frames are forwarded by the bridge, even theses that encapsulate packets that are not related to the IP protocol. Bridges are smart and learn MAC addresses on both side to avoid to repeat frames that don't need to cross from one network to the other.

Promiscuous mode

If you are using an hypervisor like VMware ESXi or Xen, you have to enable the promiscuous mode on each interfaces of the bridge to allow it to see all Ethernet frames and guess host on each side.

Here is the procedure for VMware, ask Google if you use another hypervisor.

pfSense and bridging

The bridge connects multiple interfaces. Each of these interfaces can have an IP address, but this is not required. The Bridge itself can be attached to a network interface and have an IP address, but this is not required either. Here the goal is not to combine two networks in one but just to connect them without wasting any IP addresses or increasing complexity. Frames that contains IP packets can be filtered by the firewall rules. Here I'll bridge the WAN and the OPT1 (DMZ) interface. The LAN will be kept out of the bridge.
pfSence can filter at the bridge level, but we want to filter at the member level. Check in the advanced settings of your FW that your setup match this one:

When do you need to allocate an OVH virtual MAC for your VM

I think the main purpose of the virtual MAC at OVH is to avoid MAC address conflict. This could be very annoying if your production server would stop working because someone is installing a new VM using a random MAC address that match your ! In the mean time they can use this to reinforce the security of their network.

Most of the time when you attache a failover IP to a virtual machine then you need to allocate a virtual MAC for this VM. But his is not always true. For example if you are attaching a second IP to an existing VM using the IP Alias approach, you don't need one because this second IP will use the MAC address of the primary address. Also If your VM is behind a router doing some Proxy ARP, the router use its own MAC address to masquerade your IP and the other side of the router it will be happy with any MAC address used by your VM.

This could be counter productive to allocate a MAC address when not needed, the OVH infrastructure could block the traffic when the MAC don't match the IP address. And if their are not doing that today they could do it later.

In the configuration bellow, you need to allocate a virtual MAC address for VMs going in the DMZ and for the firewall itself if you give it an IP address. No need of virtual MAC for VMs going in the LAN.

Giving a routable IP address to the firewall or not ?

Even if the bridge mode allows the firewall to do its job without giving it any IP address, the firewall need one to be managed using a web browser.

For the early setup and for emergency access, I have the use to create a VM using a XUbuntu LiveCD distibution on the LAN side and use the VMware console to manage the firewall. The Firewall LAN side is configured with default address 192.168.1.1 and a working DHCP server, this make the network setup of the VM very easy. Just connect your browser to http://192.168.1.1, use admin / pfsense for login / password and you are in.

Giving a routable IP to the WAN side of your firewall would allow you to remotely manage your firewall and do hide NAT (masquerading) for the VMs connected to the LAN side (if any). This also allows you to upgrade and install new packages.

The challenge

Here is the "virtual" challenge I'll try to solve in this post. I'm using addresses from the private ranges to make my live easier for the demo but address in 10.x.x.x and 172.x.x.x must be viewed as routable address. 192.168.x.x are used for private network, like the LAN.

I have an OVH dedicated server with address 10.99.99.99/254 and its gateway at 10.99.99.254. OVH gives me a RIPE block of 8 addresses: 10.10.10.8/29 with the gateway at 10.10.10.14. I also have 2 single failover IPs 172.16.16.16/32 and 172.17.17.17/32.

The RIPE block can be used in 2 different ways.

• The normal way is to use it as a subnet, with first address of the block reserved for the network and the last one for the broadcast address. Including the gateway address that is already allocated to an OVH router, this make 3 addresses unusable.  If you have a /30 block (4 addresses) this make only one usable address for your host(s).  The advantage of this setup is the simplicity of the network configuration of the hosts themselves.
• The other possibility is to use each address individually with a netmask of  /32. You don't need to split your RIPE block in the OVH manager to works that way (this is what OVH say), just use each address like a single failover IP. When doing this, even if VMs are on the same switch, they are not on the same subnet anymore and traffic between them has to travel up to the default gateway and back. If you want a more direct path, you have to create a route between your VMs (see below). Also I thing this is better to use the RIPE gateway 10.10.10.10.14 than using the gateway of the physical host (10.99.99.254)  because when moving the VMs and the RIPE block to another host, you don't need to change the VMs gateway. The inconvenient of this choice is that you have to configure your host network using the OVH guide (french).

Simple solution for the RIPE block

In this simple setup, I don't give any routable address to the firewall. The early setup must be done with the help of a VM installed on the LAN side.

First we need to enable and setup the WAN and DMZ (OPT1) interfaces to none. We will not attach any IP addresses to these interfaces (this time). We will bridge them. Bridge works one level below the Internet protocol and don't need any IP settings.

Then we are creating the bridge :

And setup the two members of the bridge using ctrl-click. This is the most difficult part :-)

The setup is done. Yes don't be afraid this is enough. Now we have to setup the filter rules. I give you some sample rules, but use your owns.

First I like to be able to ping in any direction, at least for the setup and troubleshooting.

Here are the details :

I want to give unrestricted internet access to VMs in the DMZ. Anyway I'm adding a restriction for the LAN. The DMZ can be accessed by the internet (demilitarized zone) and can be compromised. This is why the DMZ cannot have more access rights to access the LAN than the Internet.

Here are all the details :

Now I suppose I want to host a mail server on 10.10.10.9 and need to let SMTP packets go in..

Here are the details of the SMTP setup.

Now the most important part is the network setup of 10.10.10.9. Nothing difficult  anyway :

IP_ADDR=10.10.10.9
NETMASK=255.255.255.248 (/29)
GATEWAY=10.10.10.14

Network setup for other hosts of the RIPE block are similar.

Add a single failover IP to the DMZ

Now I want to add a host with a single failover IP 172.16.16.16/32 to the DMZ. I just need to setup the filter rules like the SMTP rule above for this new host. Nothing more is required because we are working in bridge mode.
On the other hand, the setup of the host itself is more tricky, not because of the firewall, but because of how failover IP works. Follow the OVH guide (french) to setup your new host using 172.16.16.16 as the failover IP and 10.99.99.254 for the gateway (the one ending in .254).

In this configuration, traffics between 10.10.10.9 and 172.16.16.16 has to go up to the OVH router and then back to the destination. In both case the traffic has to pass through the firewall that must have a rules to let the traffic goes out and in. This is because hosts ignore they are side by side. If you expect to have a lot of traffics and want to speed it up, you have to add routes between the hosts themselves. Don't think that the fact that your traffics between host inside the DMZ goes trough the firewall increase the security. If a pirate is able to hack one of your VM inside the DMZ, he is able to bypass the firewall to attack your other hosts too.

Here are sample command lines for linux and windows. You have to hard-code these routes in the hosts configurations.

On 172.16.16.16, add a route to network 10.10.10.8/29

linux: route add -net 10.10.10.8 netmask 255.255.255.248 dev eth0
windows: route -p ADD 10.10.10.8 MASK 255.255.255.248 172.16.16.16

On 10.10.10.9 add a route to host 172.16.16.16/32

linux: route add -host 172.16.16.16 netmask 255.255.255.255 dev eth0
windows: route -p ADD 172.16.16.16 MASK 255.255.255.255 10.10.10.9

Attach an IP to the WAN side

This makes the firewall remotely manageable and allows to give an internet access to the LAN side.
I choose 10.10.10.13/29 from the RIPE block, in the next section I explain how to use a single failover IP instead.

I go to the WAN interface configuration to setup the IP and its gateway at once.

Now I masquerade the LAN addresses using hide NAT to give Internet access to the LAN side. Don't forget to switch to manual, I don't want to NAT hosts in the DMZ.

The details:

 I also need to check that the LAN has a rule that give access to the internet :

Done !

If you want, you can forward some ports to hosts inside the LAN.
You can reach host in the DMZ from LAN.

Attach a single failover IP to the WAN side

If you don't have a RIPE block or prefer to use a single failover IP instead, then read this section.
This setup cannot be configured easily using the pfSense GUI interface, some tricks are required.
Here is the plan, first get an Internet access using some hand written command lines. Then download and install the Shellcmd package. Finally hard code the command line to make the setup persistent.

First setup the WAN IP without any gateway :

Then setup the default gateway manually. I choose to use the Command Prompt interface of the GUI. Use the two commands :

route add -net 10.99.99.254/32 -iface em0
route add default 10.99.99.254

If you don't know where 10.99.99.254 is coming from, re-read the beginning of this article.
Don't forget to setup a working DNS to have a fully working internet connection. I choose the Google public DNS.

Now the firewall should have internet access and we can install package Shellcmd.

Then hard code the two commands above inside Shellcmd to get the make config persistent.

You can try to reboot, just to see if the config works and setup hide NAT for the LAN side as explained above.

Funny things 

It looks like the OVH gateway works like an universal ARP proxy or Captive portal. I means my OVH gateway replies to any ARP request. This means that for any given a.b.c.d/32 failover IP, I can setup a host or virtual host using a mask /24 (instead of /32) and a.b.c.X where X can be anything not in ( 0, 255 or d ) for my gateway.And it works on my Kimsufi server.

I think this is how OVH setup some (maybe all) of their routers to be able to support migration of  failover IP or block without too much headache.

What is fun is to traceroute some IPs in a.b.c.*  and see that they are not directly attached to the WAN but are behind some routers.

OVH don't say anything about this setup and then this feature is funny but cannot be used on a production server.

Be careful if you use a wrong setup and generate a lot of unexpected ARP requests, OVH monitor ARP request and can warn you to quickly fix the problem or even disable your network link.

Conclusion

For me the use of the bridge is a brilliant idea. The setup is simple. You can add hosts , IPs and RIPE blocks easily to the DMZ.

Long range router : TL-WR1043ND and openwrt

I lost my internet wireless router 2 day ago and my spare one soon after.
I have replaced it with a TP-LINK TL-WR1043ND. It costed only € 54.

I'm astonished by the long range of this router.
All my wireless devices are working everywhere in the house and outside.
They are all working a lot faster, yes faster !
I think that even my neighbors get a better signal than for their own router :-)

It look like its power is 500mw (if you choose the right country code),
most router are between 50 to 200mw only to match the laws.
It has 3 big antennas that help for reception.

I have installed last openwrt (linux) in 3seconds (I'm a bit from Cosica)

Thumb up for openwrt that work very well ! I was using old kamikaze since 2006 on
the previous router.
The luCI web interface provide all the expected stuff.
The original firmware was also very "generous", including an impressive "parental control" screen.

I have setup my SIXXS IPv6 connection using some SSH, VI and command line, but I can see some part of the setup from the web interface.

The performances looks impressive. The WR1043ND can route up to 29Mo/s between LAN and WAN
and the USB allows up to 22Mo/s for read and write.

WBADMIN don't do incremental backup

WBADMIN use Volume Snapshot Service on both side to mimic what we have the use to call incremental backup.

• On the source side (the disk your are backing up), wbadmin use VSS to backup blocks that have changed since the last backup.
• On the target side (where you are backing up), if VSS is enable you can use the previous versions to access old versions of your backup. Network share don't support VSS then you cannot access old versions of your backup, you must backup in different directories. 

Every 14 days or every 14 backups, I don't remember, wbadmin does a full backup. It read all blocks on the source and overwrite all blocks on the destination.

To use the VSS feature, you must use the -vssFull switch.

If you need to do an occasional backup and don't want to disturb your daily backup, you must use -vssCopy switch (or not use -vssFull for old wbadmin than don't know the -vssCopy switch) and the historic will not be updated.

Be careful, each version of wbadmin (on W2K8, Vista, Windows 7) are a bit different and provides different switches.

If you want to use wbadmin in a script to do complex backups and send email reports you should try  MKSBackup instead. MKSBackup is a front end for wbadmin that handle multiple target directory depending the day of the backup and generate a very detailed mail report with an accurate mail subject about he success or the failure.

Xubuntu 12.04 : failed to hibernate session not authorized

Suspend was working, but hibernate was sowing message : "failed to hibernate session not authorized".

I have created file /etc/polkit-1/localauthority/50-local.d/com.ubuntu.enable-hibernate.pkla


[Re-enable hibernate by default]
Identity=unix-user:*
Action=org.freedesktop.upower.hibernate
ResultActive=yes

Now hibernate is working fine.

Xubuntu 12.04 and Nvidia 7300 multi-screen

After installing Xubuntu 12.04, on my old PC with a Nvidia 7300, only the monitor connected to the VGA port was displaying X. The DVI side stayed dark.

I have created a minimalistic /etc/X11/xorg.conf :



Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Unknown"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

And used nvidia-xconfig to customize it using :

# nvidia-xconfig --twinview  --twinview-orientation="LeftOf"

My DVI monitor (the second one) is on the left of my VGA Monitor.

That's it.
Then I have logged out and in and I got both monitors on and windows sliding between them.

For reference here is my working xorg.conf :


# nvidia-xconfig: X configuration file generated by nvidia-xconfig
# nvidia-xconfig:  version 295.40  (buildmeister@swio-display-x86-rhel47-06.nvidia.com)  Thu Apr  5 22:40:54 PDT 2012


Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0" 0 0
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
EndSection

Section "Files"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/psaux"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Keyboard0"
    Driver         "kbd"
EndSection

Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Unknown"
    HorizSync       28.0 - 33.0
    VertRefresh     43.0 - 72.0
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option         "TwinView" "True"
    Option         "MetaModes" "nvidia-auto-select, nvidia-auto-select"
    Option         "TwinViewOrientation" "LeftOf"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

VMware ESXi fastcopy.py to quickly copy .vmdk files

Overview

fastcopy.py is a python script for VMware ESXi 4.0 - 5.x that can be used as a replacement for "cp -R" to copy Virtual Machines.
fastcopy use cp to copy normal files but use vmkfstools to copy .vmk files.
Because vmkfstools is 10 to 20 time faster to copy such files, you will see a big difference.
When copying .iso files you will not notice any difference !

The typical use of fastcopy.py is to copy multiple directories containing VMs to a target directory.
fastcopy.py don't handle recursive copy or simple file copy. Sources and target must be directories.

Usage

For example top copy one VM to a new directory :

./fastcopy.py  /vmfs/volumes/SATA1/windows2008 /vmfs/volumes/NEWSATA

or if you want to copy multiple VMs, just list all VM directories. The last one must be the target one.

./fastcopy.py  /vmfs/volumes/SATA1/windows2008 /vmfs/volumes/SATA1/linux /vmfs/volumes/NEWSATA

Installation

Cut&Paste script below to one of your datastore and don't  forget to make it executable using chmod :

chmod a+x

Here is fastcopy.py

#!/bin/python
# fastcopy.py
# copy VMware virtual machine directories using cp and vmkfstools
# (c) alain.spineux@gmail.com
# released under GPL
# usage: fastcopy.py vm_dir1 vm_dir2 ... target_dir
#
# ATTN: don't copy recursively

import sys, os, subprocess, re

def copy(srcdir, dstdir):
    skipped=0
    for filename in os.listdir(srcdir):
        fullfilename=os.path.join(srcdir, filename)
        if os.path.isdir(fullfilename):
            print fullfilename, 'SKIPPED'
            skipped+=1
            continue
        print filename
        if filename.endswith('-flat.vmdk'):
            # this is the data file, it will be copied by the .vmdk
            continue
        if re.match('.*-s[0-9]{3}.vmdk$', filename):
            # this is part of a sparse file, it will be copied by the .vmdk
            continue

        # dont use vmkfstools for snapshot files
        if filename.endswith('.vmdk') and \
            not re.match('.*-[0-9]{6}-delta.vmdk$', filename) and \
            not re.match('.*-[0-9]{6}.vmdk$', filename):
            args=['vmkfstools', '-i', fullfilename, os.path.join(dstdir, filename) ]
        else:
            args=['cp', fullfilename, os.path.join(dstdir, filename) ]
        # print args
        subprocess.call(args)
    return skipped

if len(sys.argv)<3:
    print 'Usage: fastcopy.py src_dir... target_dir'
    sys.exit(1)

srcdirs=map(lambda x:x.rstrip('/'), sys.argv[1:-1])
dstdir=sys.argv[-1].rstrip('/')

if not os.path.isdir(dstdir):
    print 'dst_dir must be a directories'
    sys.exit(1)

for srcdir in srcdirs:
    if not os.path.isdir(srcdir):
        print 'not a directory:', srcdir
        sys.exit(1)
    targetdir=os.path.join(dstdir, os.path.basename(srcdir))

    if os.path.exists(targetdir):
        print 'target dir already exists:', targetdir
        sys.exit(1)

skipped=0
for srcdir in srcdirs:
    targetdir=os.path.join(dstdir, os.path.basename(srcdir))
    os.mkdir(targetdir)
    skipped+=copy(srcdir, targetdir)

if skipped>0:
    print "SKIPPED:", skipped

Ubuntu 12.04 sometime stuck in boot menu after boot failure

Ubuntu 12.04, when boot fail and maybe even when computer don't shutdown properly (no sure), grub2 stuck in boot menu at next reboot and wait for a user input.
This is very annoying for servers, just after a power cut or something else  !
To avoid this edit file /etc/grub.d/00_header and comment some line to get :

#if [ \${recordfail} = 1 ]; then
#    set timeout=-1
#else
    set timeout=${GRUB_TIMEOUT}
#fi

Develop Linux kernel module using eclipse

Here is how I did on my Ubuntu 12.04.

First I have downloaded and installed Eclipse IDE for C/C++ Linux Developers in /opt/eclipse .
Just download and extract the archive, and simply run /opt/eclipse/eclipse .

The one provided by Ubuntu should work too, but require some more packages, at least eclipse-cdt.

As a module sample, I chose the one from article  A Simple Block Driver for Linux Kernel 2.6.31
Download or Cut&Paste sbd.c and Makefile in directory ~workspace/sdb (I prefer ~/Documents/workspace/sdb).
Be careful if you Cut&Paste, Makefile expect TAB instead of spaces !

Then create a new project in eclipse

And import the sources

Now when sdb.c is open in eclipse, you should see a lot of syntax errors !

To avoid these messages, you must add some paths and symbols to the project :
Right-click on your project and select properties at the bottom, then add the path of your Linux header files:

/usr/src/linux-headers-3.2.0-24-generic/include is where Ubuntu put the headers of my kernel. I found it doing a

# ls /lib/modules/`uname -r`/build

Don't forget the include at the end of both entries.

Then add symbols __KERNEL__ with value 1 and CONFIG_BLOCK without value. Maybe more variables could be required for more complex module.

After this, I still have problem with the macro module_param.
If you find the solution, please tell me !

Ubuntu 12.04 and NoMachine NX

The biggest problem to use Unity (the default Desktop of Ubuntu 12.04) with NoMachine is that Unity use 3D accelerated hardware.
To use Unity with VNC or NoMachine you need to use unity-2d, a lighter version using Metacity instead of the 3D enabled Compiz

The simplest way to make it works is to select Unix and Custom :

and set this command in Custom - Settings

env DESKTOP_SESSION="ubuntu-2d" GDMSESSION="ubuntu-2d" /etc/X11/Xsession "/usr/bin/gnome-session --session=ubuntu-2d"

This line use Xsession instead of starting gnome-session directly. This is to setup some important environment variables like XDG_*. Unfortunately, Xsession looks to be buggy and need some help. This is why I setup the two variables DESKTOP_SESSION and GDMSESSION.

The probleme with this setup is that NoMachine client don't close smoothly and I need to Terminate the session.

To avoid this I edit file /usr/NX/etc/node.cfg and update line :

CommandStartGnome = "env DESKTOP_SESSION=ubuntu-2d GDMSESSION=ubuntu-2d /etc/X11/Xsession '/usr/bin/gnome-session --session=ubuntu-2d'"

And change my NoMachine client settings back to GNOME

Sometime the launcher bar on the left is not refreshed at startup. These icons are over all windows and then are not easy to refresh. To solve this problem I click logout in the "system" menu and move the windows over these icons. When icons are redrew I close the logout windows.

A test

 This is a small test.

A link to an internal page.

Here is a screenshot :

The application tab in Firefox

And here some python code code :

#!/usr/local/bin/python

import string, sys

# If no arguments were given, print a helpful message
if len(sys.argv)==1:
    print 'Usage: celsius temp1 temp2 ...'
    sys.exit(0)

# Loop over the arguments
for i in sys.argv[1:]:
    try: 
        fahrenheit=float(string.atoi(i))
    except string.atoi_error:
 print repr(i), "not a numeric value"
    else:
 celsius=(fahrenheit-32)*5.0/9.0
 print '%i\260F = %i\260C' % (int(fahrenheit), int(celsius+.5))

My previous blog is at ...

My previous blog is at http://blog.magiksys.net
I'm tired to update my drupal setup and don't need all the features, then is switched to blogger to make a try.