Okay, here we go.
As I said, your standard Proxmox setup features everything you need: firewall and routing options.
In my setup I run about 30 nodes (both VM and CT) per server. no HA.
In the network I have the following settings:
- eth0: don't touch
- eth1: don't touch
- vmbr0: this is your public bridge and should be the IP of the server. Adding this bridge to a server will result in a web-facing connection. When using this bridge, I advice to use a dedicated IP. I will elaborate on this later. This bridge should contain the IP settings your DC requires.
- vmbr1: dummy don't touch
- vmbr2: I created this bridge. with settings: ip: 192.168.3.1, subnet: 255.255.255.0. This bridge will provide the connection for the 192.168.3.0/24 network. Any server I want on this network, I give this bridge.
- vmbr3: Same. This bridge serves the 192.168.4.0/24 network.
Continuing. I have set-up several DNS servers. Some serve public, some serve the private network. You will probably want to do the same, setting up private DNS servers to route your internal network. Make sure to
only add a private network to this one.
Consequensively set 2 private dns servers and 2 public dns servers in your proxmox dns settings.
Don't forget to set-up your firewall in proxmox. Never touch the firewall (iptables) on the host machine. They are auto generated and break easily.
Now we have 3 networks. The private networks have no internet facing gateway.
Next we will make sure that servers only connected to the private network can perform updates, pull things from the net when they need to.
For this, I will connect the host machine to the private networks (ips 192.168.3.1 and 192.168.4.1 respectively). For this you'll make an SSH connection to the host. I'm running Proxmox on Debian, so I go and open /etc/network/interfaces.
It looks something like this:
DO NOT create the vmbr2/3/4/... here. you MUST do that in Proxmox.
auto lo
iface lo inet loopback
iface eth0 inet manual
iface eth1 inet manual
auto vmbr1
iface vmbr1 inet manual
bridge_ports dummy0
bridge_stp off
bridge_fd 0
post-up /etc/pve/kvm-networking.sh
auto vmbr0
iface vmbr0 inet static
address 125.133.xxx.xxx
netmask 255.255.255.255
gateway 125.133.xxx.xxx
broadcast 125.133.xxx.xxx
bridge_ports eth0
bridge_stp off
bridge_fd 0
auto vmbr2
iface vmbr2 inet static
address 192.168.3.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
pre-up iptables -N PREROUTING
pre-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 40002 -j DNAT --to-destination 192.168.3.2:22
pre-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 40003 -j DNAT --to-destination 192.168.3.3:22
pre-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 40003 -j DNAT --to-destination 192.168.3.4:22
pre-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 40005 -j DNAT --to-destination 192.168.3.5:22
[...]
pre-up iptables -A PREROUTING -j ACCEPT
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -N POSTROUTING
post-up iptables -t nat -A POSTROUTING -s '192.168.3.0/24' ! -d '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-up iptables -A POSTROUTING -j ACCEPT
pre-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp -m tcp --dport 40002 -j DNAT --to-destination 192.168.3.2:22
pre-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp -m tcp --dport 40003 -j DNAT --to-destination 192.168.3.3:22
pre-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp -m tcp --dport 40004 -j DNAT --to-destination 192.168.3.4:22
pre-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp -m tcp --dport 40005 -j DNAT --to-destination 192.168.3.5:22
[...]
post-down iptables -t nat -D POSTROUTING -s '192.168.3.0/24' ! -d '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -X PREROUTING
post-down iptables -X POSTROUTING
post-down iptables -F -t nat
post-down iptables -F
auto vmbr3
iface vmbr3 inet static
address 192.168.5.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
The only changes you should be making here are additions to the firewall.
Notice the lines:
pre-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 20002 -j DNAT --to-destination 192.168.3.2:22
They
must be matched by their pre-down counterpart! This is a little tweak I made to directly access the servers I want. connecting to the server port 20002 using ssh will send me directly to a server in the private network 192.168.3.2. This is not the best method to do this as it requires a LOT of maintenance to keep all the servers up-to-date and secure. A better way is to use a bastion server (A single virtual server with both public and private facing networks, only accepting SSH and only with a certificate, no root, ... you know the standard ssh security). And to connect to your destination servers through the bastion server.
The following lines:
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -N POSTROUTING
post-up iptables -t nat -A POSTROUTING -s '192.168.3.0/24' ! -d '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-up iptables -A POSTROUTING -j ACCEPT
Make sure traffic from the private network is routed through the host to the internet. Receiving servers will receive the traffic as if it was from your host (which it is in a real sense).
Please match the post-down rules, or you will be screwed after several reboots.
There. That's the basic setup.
Now for adding servers to your proxmox:
On a private network only I use the following workflow:
Each server has a number. I set the number to the network and ip they are on. Example server with IP 192.168.3.2 will be number 302, server with ip 192.168.4.50 will be number 450. This makes management a lot easier.
For the network settings, select the private bridge of your choice, the IPv4 address of your choice. The gateway will be the ip of the bridge, the ip you gave proxmox: 192.168.3.1 / 192.168.4.1 ...
Giving your server 2 or more bridges can confuse the system. Make sure you set-up your routes well and all will be fine.
I hope this little guide will prove fruitful to some.
