What's the preferred counterpart for "InternalNetwork" of VirtualBox?

[ams_tschoening]

Hi all,

I'm in the process of migrating from VirtualBox and phpVirtualBox to Proxmox and most of my VMs are simply bridged to the public uplink of the host. I'm planning to change that to a suggested routed configuration, as my new hoster will be Hetzner providing a /28 subnet, which will only be used by my VMs in future.

Though, I have two special private networks as well: One of those was a bureau of my company in the past, has been cancelled and virtualized pretty much as-is. This means there's a firewall providing publicly available OVPN and some hosts behind that firewall. None of those hosts needs to be publicly available, but each needs to reach the outside world, e.g. to download updates for software. that is currently easily reached by making the firewall the gateway of all of the clients. Additionally, the clients need to see each other of course.

The second private network shared by some VMs is for strictly internal communication of those VMs as well. While each of those VMs is additionally bridged to the public interface of the host, they need a private, internal network on their own as well.

Using VirtualBox, both private networks were implemented by the concept of an internal network, simply by using different names. The firewall of described network 1 simply has multiple adapters and one of those is that private network.

Internal networks are created automatically as needed. There is no central configuration. Every internal network is identified simply by its name. Once there is more than one active virtual network card with the same internal network ID, the Oracle VM VirtualBox support driver will automatically wire the cards and act as a network switch. The Oracle VM VirtualBox support driver implements a complete Ethernet switch and supports both broadcast/multicast frames and promiscuous mode.

The closest I found for Proxmox was either masquerading/NAT or using a VLAN, with the latter reading more likely what I need. In my described network 1, the firewall would simply be publicly exposed using the routed config and besides that there are only really private/internal networks. So masquerading/NAT doesn't seem to make too much sense here. OTOH, the docs for the VLAN-approach read exactly like what I need:

So it is possible to have multiple networks (4096) in a physical network, each independent of the other ones.

So, should I use two different VLANs implementing the former internal networks? Or am I misunderstanding something about the purpose of those VLANs regarding VM-guests?

If to use VLAN, which one? "VLAN awareness on the Linux bridge" and "Open vSwitch VLAN" read the most promising, so I should simply start with the first one?

Thanks!

 

[Dunuin]

Why not just use some Linux bridges? You can attach to the VMs as much virtio NICs as you like. If you create a Linux bridge and don't bridge it to a physical NIC you got some kind of isolated virtual switch. Give every VM a virtio NIC attached to that Linux bridge and they can communicate with each other.
And if you want a isolated network with internet access you could for example create a Linux bridge behind a OPNsense VM that works as the router/gateway/firewall.

You could use VLANs too, but it isn't really neccessary if your subnets don't need to leave the proxmox host.

 

[ams_tschoening]

With a Linux bridge you mean that vmbrX thing which can be created in the Proxmox web-UI as well? vmbr1 in the attached screenshot? Didn't know that one doesn't need to bridge that to an actual NIC. I wonder why this isn't covered in the docs and what's the difference to VLAN is in the end.

And if you want a isolated network with internet access you could for example create a Linux bridge behind a OPNsense VM that works as the router/gateway/firewall.

Could you please be more detailed what I would need to configure where? That "behind" thing confuses me, because in the attached web-UI, the bridge isn't bound to a concrete VM. I only know the other way around, binding VMs by their virtualized NICs to bridges.

 

[Dunuin]

With a Linux bridge you mean that vmbrX thing which can be created in the Proxmox web-UI as well? vmbr1 in the attached screenshot? Didn't know that one doesn't need to bridge that to an actual NIC.

Yes thats what I mean.

I wonder why this isn't covered in the docs and what's the difference to VLAN is in the end.

VLAN is a complete other thing. A bridge is just like a physical switch but done in software (so it is layer 2). VLAN is layer 3 and can also be used on top of a bridge (if you enable the "vlan aware" checkbox of that bridge). So it also would be possible to run multiple isolated VLANs on top of a isolated bridge.
And its covered in the docs. Did you read the network section of the documentation? There are 2 examples "3.3.5. Routed Configuration" and "3.3.6. Masquerading (NAT) with iptables" which use a bridge that isn't bridged to a physical NIC so the VMs are on an isolated internal network and the host is doing the routing.

Could you please be more detailed what I would need to configure where? That "behind" thing confuses me, because in the attached web-UI, the bridge isn't bound to a concrete VM. I only know the other way around, binding VMs by their virtualized NICs to bridges.

Every VM/LXC is bridged to a bridge (only a PCI passthroughed physical NIC would be an exception). If you create a new VM and create a virtual NIC for it you always need to select a bridge where the virtual NIC will be connected to. But not every bridge need to be bridged to a NIC.

A basic OPNsense/pfsense router setup would look like this:

Here the host got 3 bridges (or 2 bridges + VLAN) but only one physical NIC. "VM-LAN" is a isolated network that is behind the pfsense VM. The pfsense VM is routing between "VM-LAN" and "VMBR1-NET" and the proxmox host ist routing between the internet and the "VMBR1-NET".

 

[ams_tschoening]

For the routing to work, each VM behind pfsense needs to have configured 192.168.5.1 as gateway on OS-level as well, correct? Or is there some additional "magic" on the level of the virtualized NICs?

Additionally, what is 192.168.5.2 for? Only that Proxmox can communicate with the VMs directly without passing the firewall?

In the end, this seems to be pretty much the setup I have in VBOX as well, so I guess I'll really simply use additional "private" bridges. Would be good if those could be named better, like is possible with "InternalNetwork" of VBOX, though. Something like vmbr1 is a bit less readable than "internal_network_for_purpose_xy", like is possible in VBOX.

 

[Dunuin]

For the routing to work, each VM behind pfsense needs to have configured 192.168.5.1 as gateway on OS-level as well, correct? Or is there some additional "magic" on the level of the virtualized NICs?

Yes, 192.168.5.1 would be the gateway.

Additionally, what is 192.168.5.2 for? Only that Proxmox can communicate with the VMs directly without passing the firewall?

I think so.