Suppose you need to route all outbound traffic to a certain destination port though a different interface (VPN, GRE, you name it). Not something you have to do every day, but if you have found this article you probably know what you are doing at this point. Suppose your virtualization host has a public IP of 111.111.111.111 and the host you want to route the traffic through has a public IP of 222.222.222.222
Set up your GRE tunnel
This section is here for the sake of completeness. If you feel confident setting up your own tunnels or VPNs, just skip ahead.
Let’s make sure the ip_gre module is present in the system and set it to autoload:
lsmod | grep gre modprobe ip_gre echo ip_gre >> /etc/modules
Next step, let’s create the tunnel on the VM host and assign a local IP to it:
ip tunnel add gre01 mode gre remote 222.222.222.222 local 111.111.111.111 ttl 255 ip link set gre01 up ip addr add 10.10.10.1/24 dev gre01
Tunnel creation on the remote host is identical:
ip tunnel add gre01 mode gre remote 111.111.111.111 local 222.222.222.222 ttl 255 ip link set gre01 up ip addr add 10.10.10.2/24 dev gre01
At this point the tunnel is up and we should be able to ping the remote host from the VM host:
ping 10.10.10.2
Time to mangle some traffic
Create a routing table on the VM host and point it to your remote host. I used number 12 here, that’s arbitrary as long as it doesn’t conflict with existing tables on your system.
ip rule add fwmark 12 table 12 ip route add default via 10.10.10.2 table 12 ip route flush cache
Add the iptables rule that will set the forwarding mark on the VM host:
iptables -t mangle -A OUTPUT -p tcp --dport 8888 -j MARK --set-mark 12
This assumes that your VMs have bridged interfaces. If you’re running a NAT setup, you’ll need to put this into the prerouting chain instead:
iptables -t mangle -A PREROUTING -p tcp --dport 8888 -j MARK --set-mark 12
Now, to make sure the remote host accepts this:
iptables -t nat -A POSTROUTING -o gre01 -j SNAT --to-source 10.10.10.1
Finally, we need to loosen up the reverse path filter on the GRE interface of the VM host. Obviously, substitute gre01 with whatever interface you have configured:
sysctl -w net.ipv4.conf.gre01.rp_filter=2
Now, just to add a masquerade rule on the remote host, and we are done:
iptables -t nat -A POSTROUTING -s 10.10.10.0/24 -j MASQUERADE
Conclusion
We have configured a set of rules that will allow our VMs “use” a different public IP address for their outbound connections depending on the destination port. The important bit is that this is completely transparent for the VMs.
You will probably want to make these changes permanent by configuring persistent rules and interfaces on both hosts. We have also used a trick called loose mode reverse filtering, which you can read up on here: http://www.slashroot.in/linux-kernel-rpfilter-settings-reverse-path-filtering