Well..., there are multiple ways to work with separated networks and/or VLANs in PVE. At least a) "everything is trunked, pick you VLAN by setting the tag by yourself", b) "there is no VLAN, you only get untagged traffic on this (your VM) interface" and c) SDN.
Maybe SDN is mis-listed here. I am not sure because I do not use it
From a) and b) I chose b) because for me it looks simpler and does fit better to my brain. For c) my clusters are too small to be tempting and I am too old ;-)
But I have one or two dozen networks. And one or two dozen VLANs. Remember those two things represent two independent layers of the network stack. Usually I use a one-to-one relationship to glue them together. But this is a choice of mine, not a requirement.
My approach is to give a VM (or a container) one (or more) NIC which delivers
exactly one network to them. VM NICs are connected to PVE "classic" bridges. So I need one bridge per IP network. Or one bridge per VLAN - which in my case is
exactly the same.
The following is an excerpt of my /etc/network/interfaces and it establishes two bridges
with an IP address and a third bridge
without an assigned IP address. You only need an IP address on a bridge if PVE itself needs to access that network directly. All of my traffic goes through a router and there is only one default gateway known by this PVE. (The other bridges do not have an IP address. I picked the listed subset for the sake of this demonstration.)
Code:
iface eno1 inet manual
iface enp4s0 inet manual
auto vmbr3
iface vmbr3 inet static
address 10.3.16.8/16
gateway 10.3.12.254
bridge-ports enp4s0
bridge-stp off
bridge-fd 0
# san
auto vmbr2
iface vmbr2 inet manual
bridge-ports eno1.2
bridge-stp off
bridge-fd 0
#dmz
auto vmbr11
iface vmbr11 inet static
address 10.11.16.8/16
bridge-ports eno1.11
bridge-stp off
bridge-fd 0
#adm
Note that vmbr3 is not bound to a VLAN but to an separate NIC. Untagged. I want this traffic separated physically from the other main trunk, which contains several networks. (Sidenode: corosync is configured to use one ring on these two pyhsical NICs each.)
The two other bridges use the same physical NIC, separated by VLAN tags. vmbr2 is called "dmz" and PVE is NOT reachable directly from there. On the other hand vmbr11 has an IP address and this means each VM inside that network may talk to this PVE. Without routing!
PVE can talk to each network it has a bridge with an IP address in. Without routing.
For everything else data is being sent to the default gateway. That's it. At least for this topology which is a simple as it can be.
vmbr2 may have (IP-) traffic of any kind. PVE does
not see it. PVE can not get accessed and not attacked from there. For VMs connected to that bridge there is no exit through PVE. There must be (and there is, of course) another router inside that network which handles traffic in/out.
Probably one could write books about the different ways to connect PVE and some VMs to the Internet ;-)
Compare also:
https://pve.proxmox.com/wiki/Network_Configuration#_default_configuration_using_a_bridge ff.
