Proxmox on Management network and vms on own network not working

[edwinherren]

Hi everybody

Not sure if this is an OPNsense issue or Proxmox or both but I hope someone can help me. I did search, read and try numerous posts/suggestions/solutions here and elsewehere but none helped. I'm not even sure if Im going about this the right way.....

My setup: OPNsense on dedicated/standalone server and two proxmox machines, all connected to a managed switch. I have a management network, LAN, and a DMZ network. Each network separated by vlans. OPNsense, and the Proxmox server's web UI are on the same management network and can only be accessed from the LAN. Which is what I want. Only the LAN can access devices on all other networks but other devices cannot acces the LAN and each other. All physical devices work fine, even when using vlans. The problem are the VMs.

I have tried all combinations with and without vlans but have failed on all counts. VMs only work when assigned to the network the proxmox host belongs to: i.e. the management network. The idea is not to have them on the management interface/network but their own network/s.
Does anybody have an idea what could be my problem? Any help would be highly appreciated.

NOTE: Please, I know how to create bridges, vlans, networks, etc. But this is my first time having a management network and I think thats where I am getting lost.

Thank you
Edwin

 

[supermicro_server]

Hello,
what does "VMs only work when assigned to the network the proxmox host belongs to" mean?
No access to internet?
No access to physical network?
No ping to the gateway?

Thanks

 

[edwinherren]

Hi supermicro_server and thanx for your response.
When I create a vm and assign it an IP on the same subnet as the Proxmox host then it works - network and internet acess.
Otherwise the vms created on different subnet have no internet access, cannot be pinged from anywhere. No access to physical network.
Thanx again

 

[supermicro_server]

Can you check if the "Autostart" funcion is checked on the network interface?
Please past here this command:

ethtool <interface_name>

thank you

 

[louie1961]

Please post the contents of your /etc/network/interfaces file here for us to review. Have you made the bridge VLAN aware? Are you connecting your Promox nodes to the managed switch on a trunk port? Your Opensense box must connect to the switch on a trunk port (i.e., a tagged port), and your VLAN aware NIC on your Proxmox nodes must also be connected to a trunked (i.e., untagged) port as well. AND if you are using a VLAN aware WAP, that must connect to ta trunled port as well.

 

[edwinherren]

Can you check if the "Autostart" funcion is checked on the network interface?
Please past here this command:

ethtool <interface_name>

thank you

Hi supermicro_server
Yes the network interface is on autostart

 

[edwinherren]

Please post the contents of your /etc/network/interfaces file here for us to review. Have you made the bridge VLAN aware? Are you connecting your Promox nodes to the managed switch on a trunk port? Your Opensense box must connect to the switch on a trunk port (i.e., a tagged port), and your VLAN aware NIC on your Proxmox nodes must also be connected to a trunked (i.e., untagged) port as well. AND if you are using a VLAN aware WAP, that must connect to ta trunled port as well.

Hi louie1961
Thanx
Here's the current /etc/network/interfaces settings:
auto lo
iface lo inet loopback

auto enp0s31f6
iface enp0s31f6 inet manual

auto vmbr0
iface vmbr0 inet manual
bridge-ports enp0s31f6
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094

auto vmbr0.10
iface vmbr0.10 inet static
address 10.62.10.100/24
gateway 10.62.10.1

auto vmbr1
iface vmbr1 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094

auto vmbr1.20
iface vmbr1.20 inet static
address 10.62.20.100/24

source /etc/network/interfaces.d/*


And yes, all connections from Opnsense box to switch connected via trunk/tagged ports. The interface is vlan aware.
All connected devices with above /etc/network/interfaces config file work. The Proxmox boxes web UI are accessible. It's only the vms and containers on these boxes that I cannot make work.

The Proxmox boxes are on vlan10 and they are accessible. The vms and cts I want on vlan20, which is on a different subnet to the Proxmox box they reside on.

Thanx a lot

 

[sw-omit]

Have you added your VM's to the vmbr1 bridge and set them to VLAN20?
If so:

iface vmbr1 inet manual
bridge-ports none

The Bridge doesn't have any physical network adaptors connected to it, so it doesn't go to the switch.
Try setting the VM's to vmbr0 and then VLAN20, see if that works.

Alternatively: If you set the proxmox-management IP to the VLAN20 and it's range (so swap the enp0s31f6 from vmbr0 to vmbr1), can you then access your server from the LAN to this new lan-IP? This would be to confirm that the switch-settings are correct at least (but do be sure to have console-access to your server to revert it back in /etc/network/interfaces if it does not)

 

[louie1961]

I see a couple of errors. Not sure if this is all of the problems or not. Also did you create this file manually? The spacing looks all wrong, but it may just be the way you pasted it into browser, not sure.

auto vmbr0
iface vmbr0 inet manual
bridge-ports enp0s31f6
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094

This should read as follows. Note the bolded items. Change manual to static so you don't have to manually intervene to bring the bridge up. Also, you can't use VLAN IDs 4093 or 4094. You must stop at 4092. Don't ask me why, I really don't remember. But I am sure this is the case.

auto vmbr0
iface vmbr0 inet static
bridge-ports enp0s31f6
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4092

auto vmbr1
iface vmbr1 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094

auto vmbr1.20
iface vmbr1.20 inet static
address 10.62.20.100/24

Change iface vmbr1 inet to STATIC first of all, otherwise you have to manually start it every time you boot. and second, why no ethernet port on this? Is this just an internal to Proxmox network? If that's the case its probably OK. If you are trying to put your management interface on VLAN 20 then this entire section needs to be removed. Also the IP address conflicts with VLAN 10. Each VLAN needs a unique IP address range.

Here is an example of my /etc/network/interfaces for you to maybe copy/emulate. My management VLAN is 10. All of my VMs are on other/different VLANs, each VLAN having a unique IP range

auto lo
iface lo inet loopback
iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4092

auto vmbr0.10
iface vmbr0.10 inet static
address 10.10.10.2/24
gateway 10.10.10.1

My VLANs are defined in pfSense, not in Proxmox, I have different IP ranges for every VLAN (10.10.10.1/24, 10.10.20.1/24, 10.10.30.1/24, etc.), and each VLAN has its own DHCP server.

 

[louie1961]

Oh and all my VMs and CTs use vmbr0 and not vmbr0.10 Maybe that's where you were getting crossed up?

 

[sw-omit]

Also the IP address conflicts with VLAN 10. Each VLAN needs a unique IP address range.

Not sure where you're seeing a conflict btw,
vlan 10 has a 10.62.10.X/24 range
vlan 20 has a 10.62.20.X/24 range

And yeah, I also use vmbr0 for the traffic with a vlan-tag, or more precisely an SDN vlan-zone VNET so I don't have to remember to set vlan's, can separate permissions and have the different vlan's have a nice nametag (with only untagged traffic going on the vmbr0), but wanted to get it set up with manual vlan's first before suggesting that route

 

[louie1961]

my bad. I read them as being the same