How to: route VM traffic based on destination port

Suppose you need to route all outbound traffic to a certain destination port though a different interface (VPN, GRE, you name it). Not something you have to do every day, but if you have found this article you probably know what you are doing at this point. Suppose your virtualization host has a public IP of 111.111.111.111 and the host you want to route the traffic through has a public IP of 222.222.222.222

Set up your GRE tunnel

This section is here for the sake of completeness. If you feel confident setting up your own tunnels or VPNs, just skip ahead.
Let’s make sure the ip_gre module is present in the system and set it to autoload:

lsmod | grep gre
modprobe ip_gre
echo ip_gre >> /etc/modules

Next step, let’s create the tunnel on the VM host and assign a local IP to it:

ip tunnel add gre01 mode gre remote 222.222.222.222 local 111.111.111.111 ttl 255
ip link set gre01 up
ip addr add 10.10.10.1/24 dev gre01

Tunnel creation on the remote host is identical:

ip tunnel add gre01 mode gre remote 111.111.111.111 local 222.222.222.222 ttl 255
ip link set gre01 up
ip addr add 10.10.10.2/24 dev gre01

At this point the tunnel is up and we should be able to ping the remote host from the VM host:

ping 10.10.10.2

Time to mangle some traffic

Create a routing table on the VM host and point it to your remote host. I used number 12 here, that’s arbitrary as long as it doesn’t conflict with existing tables on your system.

ip rule add fwmark 12 table 12
ip route add default via 10.10.10.2 table 12
ip route flush cache

Add the iptables rule that will set the forwarding mark on the VM host:

iptables -t mangle -A OUTPUT -p tcp --dport 8888 -j MARK --set-mark 12

This assumes that your VMs have bridged interfaces. If you’re running a NAT setup, you’ll need to put this into the prerouting chain instead:

iptables -t mangle -A PREROUTING -p tcp --dport 8888 -j MARK --set-mark 12

Now, to make sure the remote host accepts this:

iptables -t nat -A POSTROUTING -o gre01 -j SNAT --to-source 10.10.10.1

Finally, we need to loosen up the reverse path filter on the GRE interface of the VM host. Obviously, substitute gre01 with whatever interface you have configured:

sysctl -w net.ipv4.conf.gre01.rp_filter=2

Now, just to add a masquerade rule on the remote host, and we are done:

iptables -t nat -A POSTROUTING -s 10.10.10.0/24 -j MASQUERADE

Conclusion

We have configured a set of rules that will allow our VMs “use” a different public IP address for their outbound connections depending on the destination port. The important bit is that this is completely transparent for the VMs.

You will probably want to make these changes permanent by configuring persistent rules and interfaces on both hosts. We have also used a trick called loose mode reverse filtering, which you can read up on here: http://www.slashroot.in/linux-kernel-rpfilter-settings-reverse-path-filtering