Marian Marinov/HackMan

One of the problems when you are migrating a server is actually migrating the traffic from the old IPs to the new IPs.

Since we were doing that fairly often for some of our server I have used one very common trick, which was to use DNAT + SNAT to move the traffic from the old server to the new server, and from the new server back to the client.

However this presents one big issue, all clients are now coming from the IP address of the old server. This is a big problem for systems that authenticate by including the remote IP in connection with the user/password pair.

So I set out to find a way I can move all traffic for whatever TCP service I decide with keeping the source IPs.

The solution was fairly simple and this is what I’ll try to describe here.

So here is our setup:

• server A with IP: 112.0.0.15
• server B with IP: 213.0.0.12

In order to direct all web traffic going to server A we have to use two DNAT rules:

# iptables -t nat -I PREROUTING -j DNAT -d 112.0.0.15 -p tcp --dport 80 --to 213.0.0.12:80
# iptables -t nat -I PREROUTING -j DNAT -d 112.0.0.15 -p tcp --dport 443 --to 213.0.0.12:443

This is all we need to do on sever A. Most of the work is done on server B.

So, after the traffic has reached server B why we have problem?
The problem is because the DNAT from server A only changed the destination IP and so the source of the packet is still the IP of the client and not the IP of server A. So in order for the answering packet to be accepted on the client, the packet should be with source IP, the IP of server A, not the ip of server B. But after the packet is received on server B. The response is send directly to the client and here is where the problems begin.

Here is one of my solutions to this problem. You should build a tunnel between server A and B. I used OpenVPN for that tunnel and used tap devices in order to have full bridge functionality(this is very important since the packets that would be received from server B come from all kinds of IP ranges).
After we have the tunnel setup we should have some private network between A and B, for example:

• server A: 10.0.0.1
• server B: 10.0.0.2

What you should do now, on server B is simple:

1. Mark all the traffic coming from the tunnel interface with some mark, for example 4
2. Restore the marks of all packets as they exit the process and before the first routing decision
3. Move all packets to a separate routing table so their default gateway is now the VPN IP of server A and not the default gw of server B

How are all of these steps done:

1. Creating the routing table:
   # echo '250 forward' >> /etc/iproute2/rt_tables
2. Adding entries to the routing table:
   
# ip r a 213.0.0.12 dev eth0 t forward
# ip r a 127/8 dev lo t forward
# ip r a 0/0 via 10.0.0.1 t forward

3. Make all packets marked with mark 4 to go trough the routing table:
   # ip ru a fwmark 4 t forward
4. Mark all incoming web packets from the VPN with mark 4:
   
# iptables -t mangle -A PREROUTING -j CONNMARK -i tap0 -p tcp --dport 80 --set-mark 4
# iptables -t mangle -A PREROUTING -j CONNMARK -i tap0 -p tcp --dport 443 --set-mark 4

5. Restore all packet marks as they leave the processes:
   # iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

And that should be it.
After this, you should receive all web traffic to server A as it was directly coming to server B.

One Response to “Migrating traffic to a new server”

1. Marian Says:
   November 28th, 2009 at 1:59 am

   This is actually a good idea for man-in-the-middle attacks