[SOLVED] Internal Virtual Network. How to?

fraksken

New Member
Feb 5, 2015
29
3
1
Belgium
remisan.be
Hey,

I am trying to set-up the following configuration:
Dedicated machine hosts ProxMox with 1 eth uplink (e.g. 150.100.2.150)
  • A virtual router / firewall provides IP's to a virtual network (e.g. 192.168.0.x)
    • All servers/containers either receive a virtual IP
    • The servers will receive specific domain names (fqdn) on which they need to be accessible from outside (the internet)
I am completely new to virtualisation and can use some help.
I would do the following:
- add virtual nic (how do I do this?) virteth0
- set up a virtual router and connect the eth0 bridge and the virteth0.
- the virtual router needs route the connections through virteth0
- For this I believe I either need to set up a DNS server OR set the IP's with domain names in the hosts file => on both the dedicated machine and the router
- Additionally I think I'll need to set up routes to the virtual machines/containers in the dedicated machine and router => How do I do this?


I would be endlesly grateful if anybody can help me with this.
I want to avoid using esx and Proxmox seems to be a worthy replacement.
 
Last edited:
Create a bridge thats not connected to an eth device. Install a router distro. My favorite is vyos. Eth0 on the router should go to vmbr0 which should be your internet facing device. Vmbr1 (or whatever number you created in the first step) is your virtual interface. That becomes eth1 on your virtual router. Set up your router with your Nat and DNS settings. That's it. You now have a private network inside your proxmox host.
 
Thanks for your answer.
I applied your recommendations except vyos. I would like a router with a graphical interface, and if possible with ssl termination.
Pfsense seems to be a good option, however, I am running into some configuration problems.

My dedicated machine has 1 physical nic with an IP xxx.xxx.xxx.xxx
I have 4 additional IP's yyy.yyy.yyy.yy1, yyy.yyy.yyy.yy2, ...
For easy management, I want to assign ip yyy.yyy.yyy.yy1 to the router as public IP.
The datacenter where I host my server (OVH) requires me to use a specific configuration.
In short, I need to setup the router without IP configuration and alter the configuration file according to the following scheme:
Contents of the file: /etc/network/interfaces
auto lo eth0
iface lo inet loopback
iface eth0 inet static
address yyy.yyy.yyy.yy1
netmask 255.255.255.255
broadcast yyy.yyy.yyy.yy1
post-up route add xxx.xxx.xxx.254 dev eth0
post-up route add default gw xxx.xxx.xxx.254
pre-down route del xxx.xxx.xxx.254 dev eth0
pre-down route del default gw xxx.xxx.xxx.254

This poses several problems:
- Most routing distro's do not support a netmask of 255.255.255.255.
- Most routing distro's do not support a default gateway outside of the net mask
- Most routing distro's do not allow me to change the network config in cli.

I have tried the following:
- pfsense
- zeroshell
- ipfire
- zentyal
- and some others

I am now thinking of installing a centos with routing software on top. However up to now I could not find anything that I can use. I need a comprehensive GUI which allows the minimum of the following:
- custom configuration of network configuration script
- DMZ
- DNS forwarding
- routing
- Firewall (iptables should do)
- dhcp server for virtual network

If possible, the following features would be awesome:
- logging
- graphics
- ssl termination
- vpn

I need help :(
 
Last edited:
okay, I found a work around.
I set up a gateway between the WAN and the router. I had no problem setting up the network on the gateway and just forwards all traffic.

problem solved.
 
Hi pirateghost, what is the best way to differentiate between the proxmox host IP and the virtual router public IP? At the moment I have vmbr0 with the IP for the Promox host and I have a different IP configured on the router bound to the same bridge. I'm trying to set up a Mikrotik VM as the firewall/router (needed for other custom services hence this choice) and wanted to know if it is possible to firewall the proxmox host access through the Mikrotik or if I will need to lock it down via iptables separately? I can keep the two public IPs on if need be, but wanted to see if there was a better way to do it...
 
Last edited:
Hi there.
I seem to have a similar problem.
I have been using IPFire as firewall for virtual ethernet on a test server for a long time.

Now I try to replicate the setup on a rented server from online.net following along with their PVE KVM VM installation instructions:
https://documentation.online.net/en...istribution-proxmox/distribution-proxmox-ve-4

Using a Failover IP:
https://documentation.online.net/en...over/ip-failover#configuration_on_your_server

Virtual MAC Addresses:
https://documentation.online.net/en/dedicated-server/network/ip-failover/virtual-mac

[edit: So I sorted out using IPFire as the Firewall for my virtual network ('green' interface) by NOT having the Virtual MAC on the access Bridge :-( and giving red0 Netmask of 255.2556.0.0 i.e. x.y.0.0/16 ]

so with eth0 enslaved to vmbr0

and vm100 nic0 on vmbr0 as interface red0

I find that installing IPFire with the red0 interface configured with the Failover IP and virtual MAC provided by online.net does not work.

The online.net documentation says to use a netmask of 255.255.255.255 but IPFire will not allow this.

@ fraksken
I like the sound of your solution, because it sounds like I could setup a network gateway on each Host Machine
( IP, Gateway, Netmask, MAC etc are unique to each server / rack / Datacentre / Hosting Provider)
thus abstracting away those differences and allowing to migrate VM images between PVE hosts without having to change any of their network setup.

Can you describe how you achieved this please?

Any further suggestions on how to get IPFire (or other Virtual Firewall/Gateway/Router working )
will be very gratefully received!

Thanks in advance.
:)
 
Last edited:
Hey Auser,

I had to read the post again to be reminded what I was trying to do at that time.
I can't recommend the solution I found above as it will be slow when it doesn't need to be.
Proxmox itself provides with sufficient routing options, running an additional router is really not required.
I can advice to create a new virtual bridge for your private lan. Whenever you create a new machine, add the LAN bridge to it as a network interface and give it a legal ip in the range what you configured in your bridge.
This will enable networking within proxmox. Any server that needs to get public access you can connect to your public bridge.
If a server needs both public and private (a reverse proxy for example) you give it both bridges and configure the routes on the host.
it's the easiest way to work.
I really really do not advice to setup a VM or CT just for the sake of a router.

if you want me to clarify any of the above, just reply, I can get configuration files etc in a day or 2. (I'm very busy sorry)

as to why IPfire won't allow a 255.255.255.255 netmask, because it's not good practice. it defines the privater network. Setting 255.255.255.0 for example (which may work) will send any broadcasting from your servers to neighbouring servers. They can be listening and collecting valuable information, or they may be disturbed by your traffic. etc etc ... it'd be a security risk.

I will try to make a comprehensible guide this weekend on how I managed to get networking working in proxmox with a minimum of resources used.
 
Last edited:
Thankyou for that rapid and thoughtful answer fraksken. :)

That would be absolutely superb if you could make a guide! :)

I was going to go with IPFire solution because it should 'just work' ... been using it on my LAN and PVE test boxes for years, but it didn't work on online.net host. :-(

Decided to also ask on the the community forums over at online.net, in case anyone there can offer suggestions for gotchas with their specific Failover IP / Virtual MAC setup, and used this thread/your posts as inspiration. ;-)

https://community.online.net/t/prox...ilover-ip-virtual-mac-and-ipfire-in-a-vm/3112

Thanks again!

(In the meantime will go look at vyos again ... I really liked it before but haven't installed it recently, as project seemed to have slowed right down)

[edit:] That was worthwhile- I got it going with vyos no problem, and am heartened to see a new Beta version of vyos is in testing which is updated to a Debian Jessie base. :)
 
Last edited:
Okay, here we go.
As I said, your standard Proxmox setup features everything you need: firewall and routing options.
In my setup I run about 30 nodes (both VM and CT) per server. no HA.
In the network I have the following settings:
- eth0: don't touch
- eth1: don't touch
- vmbr0: this is your public bridge and should be the IP of the server. Adding this bridge to a server will result in a web-facing connection. When using this bridge, I advice to use a dedicated IP. I will elaborate on this later. This bridge should contain the IP settings your DC requires.
- vmbr1: dummy don't touch
- vmbr2: I created this bridge. with settings: ip: 192.168.3.1, subnet: 255.255.255.0. This bridge will provide the connection for the 192.168.3.0/24 network. Any server I want on this network, I give this bridge.
- vmbr3: Same. This bridge serves the 192.168.4.0/24 network.

Continuing. I have set-up several DNS servers. Some serve public, some serve the private network. You will probably want to do the same, setting up private DNS servers to route your internal network. Make sure to only add a private network to this one.

Consequensively set 2 private dns servers and 2 public dns servers in your proxmox dns settings.

Don't forget to set-up your firewall in proxmox. Never touch the firewall (iptables) on the host machine. They are auto generated and break easily.

Now we have 3 networks. The private networks have no internet facing gateway.

Next we will make sure that servers only connected to the private network can perform updates, pull things from the net when they need to.
For this, I will connect the host machine to the private networks (ips 192.168.3.1 and 192.168.4.1 respectively). For this you'll make an SSH connection to the host. I'm running Proxmox on Debian, so I go and open /etc/network/interfaces.
It looks something like this:
DO NOT create the vmbr2/3/4/... here. you MUST do that in Proxmox.
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto vmbr1
iface vmbr1 inet manual
bridge_ports dummy0
bridge_stp off
bridge_fd 0
post-up /etc/pve/kvm-networking.sh

auto vmbr0
iface vmbr0 inet static
address 125.133.xxx.xxx
netmask 255.255.255.255
gateway 125.133.xxx.xxx
broadcast 125.133.xxx.xxx
bridge_ports eth0
bridge_stp off
bridge_fd 0

auto vmbr2
iface vmbr2 inet static
address 192.168.3.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
pre-up iptables -N PREROUTING
pre-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 40002 -j DNAT --to-destination 192.168.3.2:22
pre-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 40003 -j DNAT --to-destination 192.168.3.3:22
pre-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 40003 -j DNAT --to-destination 192.168.3.4:22
pre-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 40005 -j DNAT --to-destination 192.168.3.5:22
[...]
pre-up iptables -A PREROUTING -j ACCEPT
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -N POSTROUTING
post-up iptables -t nat -A POSTROUTING -s '192.168.3.0/24' ! -d '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-up iptables -A POSTROUTING -j ACCEPT
pre-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp -m tcp --dport 40002 -j DNAT --to-destination 192.168.3.2:22
pre-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp -m tcp --dport 40003 -j DNAT --to-destination 192.168.3.3:22
pre-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp -m tcp --dport 40004 -j DNAT --to-destination 192.168.3.4:22
pre-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp -m tcp --dport 40005 -j DNAT --to-destination 192.168.3.5:22
[...]
post-down iptables -t nat -D POSTROUTING -s '192.168.3.0/24' ! -d '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -X PREROUTING
post-down iptables -X POSTROUTING
post-down iptables -F -t nat
post-down iptables -F

auto vmbr3
iface vmbr3 inet static
address 192.168.5.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
The only changes you should be making here are additions to the firewall.
Notice the lines:
pre-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 20002 -j DNAT --to-destination 192.168.3.2:22
They must be matched by their pre-down counterpart! This is a little tweak I made to directly access the servers I want. connecting to the server port 20002 using ssh will send me directly to a server in the private network 192.168.3.2. This is not the best method to do this as it requires a LOT of maintenance to keep all the servers up-to-date and secure. A better way is to use a bastion server (A single virtual server with both public and private facing networks, only accepting SSH and only with a certificate, no root, ... you know the standard ssh security). And to connect to your destination servers through the bastion server.

The following lines:
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -N POSTROUTING
post-up iptables -t nat -A POSTROUTING -s '192.168.3.0/24' ! -d '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-up iptables -A POSTROUTING -j ACCEPT
Make sure traffic from the private network is routed through the host to the internet. Receiving servers will receive the traffic as if it was from your host (which it is in a real sense).
Please match the post-down rules, or you will be screwed after several reboots.

There. That's the basic setup.
Now for adding servers to your proxmox:
On a private network only I use the following workflow:
Each server has a number. I set the number to the network and ip they are on. Example server with IP 192.168.3.2 will be number 302, server with ip 192.168.4.50 will be number 450. This makes management a lot easier.
For the network settings, select the private bridge of your choice, the IPv4 address of your choice. The gateway will be the ip of the bridge, the ip you gave proxmox: 192.168.3.1 / 192.168.4.1 ...

Giving your server 2 or more bridges can confuse the system. Make sure you set-up your routes well and all will be fine.

I hope this little guide will prove fruitful to some.
 
As a note, I would suggest to set the following servers public:
  • reverse proxy (for all web access)
  • bastion (for all SSH connections)
  • public DNS servers
  • Mail relay
All the rest of the servers can be private. Access to admin pages can be extra secured by setting up the reverse proxy to require a client side certificate for certain urls. A client side certificate is infinite times more secure then a mere password.
 
  • Like
Reactions: Thorvi
Thankyou Thanyou Thankyou @fraksen
for making such a clear and useful post. :)

That is good advise to use A bastion server and ssh password disable.
I am trying out Nethserver at the moment.

If only the forum had a Tuturial section: this would be an ideal post for all newbies to learn from.
 
hm, so if I have read all the posts correctly, you basically abandoned the idea of having a firewall/router installed inside a VM and instead went and only used what is built into proxmox right?

I'm trying to setup a firewall distro inside a VM to protect my ProxMox and all other VMs but am having trouble wrapping my head around how to do this. The best thing to find would be a network diagram exemplifying how everything is connected or any other pointer towards this goal.
 
Yes. I would not advice to use resources for a VM or container to do the routing when everything is in Proxmox already.

I am not sure how you are planning to secure your box. Proxmox needs to be able to ssh to the public IP and you require access to port 8006 to manage your host.
What might be a possibility is to set up firewall rules, in proxmox to only allow port 22 from the subnet you have and the public IP. This will lock you out of shell access also. Proxmox allows you to get into the shell from the management console. Or you could add a rule to allow port 22 from a specific IP (your home IP).
To secure the webinterface, I would advice to setup a reverse proxy (as a container or as an external machine) that allows you access to the web interface. Block all access from elsewhere.

Proxmox has a lot of routing and firewalling options. There's really no need to install a routing / firewall OS to regulate that.
While proxmox does not provide DHCP (to my knowledge, it's been a while), you can simply designate the IP's yourself, as long as you connect the systems to the right bridge.

Let me know if you need more info than what I've written above.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!