[SOLVED] [SOLVED] VMs not communicating on VLAN

[JordansGhost]

Im a newbie to proxmox and Linux but not to networking - just a little out of practice. This is probably a simple solution but non of the examples I find online or in the forums seem to work for me.

Setup:
1) Latest Unifi network running 3 VLANs - Native VLAN 1 (192.168.150.0/24), VM VLAN (192.168.100.0/24) and client VLAN (192.168.101.0/24).
2) all proxmox devices connected to a flex mini switch on a single NIC for each of my two proxmox hosts (second NIC is available but I planned to use it for ISCSI)
3) switch port set to Native VLAN as a default and "allow all" so it goes into trunk mode
4) Proxmox host 1 is on 192.168.100.62 and host 2 is on 192.168.100.61. My test VM is static assigned 192.168.100.11

I have tried Linux VLANs, SD WAN VLANs - nothing seems to get me anywhere. The nearest I can get is to get a response from host IP with the following config but the VM will not respond to a ping

/etc/network/interfaces

auto lo
iface lo inet loopback


iface enp0s31f6 inet manual


iface enx00e06c390cfe inet manual


auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp0s31f6
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094


auto vmbr0
iface vmbr0 inet static
        address 192.168.100.62/24
        gateway 192.168.100.1
        bridge-ports enp0s31f6.100
        bridge-stp off
        bridge-fd 0


source /etc/network/interfaces.d/*

And this is the setup in my /etc/sysctl.conf - some of these options were added because DHCP and DHCPv6 isnt working either. My DHCP server is external to proxmox running on the router at 192.168.100.1



and this is the latest attempt at my VM network config. Its a windows 2022 server.

 

[JordansGhost]

I solved the issue. Here are detail instructions for anyone who has a similar issue.

First thing well do is change the network interface names to something easier to work with

From the CLI run

ip a

Output will look something like:



This will provide the name and mac address of each interface - you can ignore lo etc a you are only interested in the physical interfaces. In my case one example was:

Name: enx00e06c390cfe
Mac: 00:e0:6c:39:0c:fe

Now execute the following commands

cd /etc/udev/rules.d
nano 10-network.rules

Note this will likely create a new file. Add something similar to the following, replacing the mac address and name with the correct data for your system.

SUBSYSTEM=="net",ACTION=="add", ATTR{address}=="48:4d:7e:e3:7d:ef",NAME="eth0lan"
SUBSYSTEM=="net",ACTION=="add", ATTR{address}=="00:e0:6c:39:0c:fe",NAME="eth1dmz"

Exit and save the file

Now open your network interfaces file. You will see lines like this which contain the names of your interfaces. Change the name to match the new names you put into the subsystem command. In my case:

iface eno1 inet manual

Becomes

iface eth0lan inet manual

Note: manual just means don’t auto assign an ip to the interface and this is needed as the ip is on the bridge or sub interface of the bridge. Alternative options are static / dhcp but that is not used here.

Next find your section that looks like this:



And basically change the file to something that looks like below. It basically removes the ip from the bridge as the bridge cannot sit in a vlan and be a trunk for all vlans at the same time. Also Note here that the management IP VLAN/interface is a totally separate concept from getting your VMs into a VLAN.

Here we will create a sub interface in VLAN 100 (in my case) and set its IP as static. Then we will remove the IP and gateway from the bridge and set it to manual so it doesn’t pick up an ip from the native vlan.

My file now looks like this.

auto lo
iface lo inet loopback
 
iface eth0lan inet manual
 
iface eth1dmz inet manual
 
auto vmbr0
iface vmbr0 inet manual
        bridge-ports eth0lan
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
 
auto vmbr0.100
iface vmbr0.100 inet static
address 192.168.100.62/24
gateway 192.168.100.1
 
source
/etc/network/interfaces.d/*

Next we save the file and reboot the system. As long as you have set the port on your physical switch to be a trunk port then the proxmox server should be pingable.

Now to get VMs working. The concept here is that they DO NOT connect to the sub interface you have created. They need to connect to a bridge. And to make them connect on the correct VLAN you simply change the VLAN tag under hardware > network for the relevant VM.



Once this was set then outbound ping started working to the gateway and internet access was working too.

FINAL NOTES:

1. In my case this got DHCP and outbound ping were working just fine. Inbound I still had to troubleshoot which turned out to be windows firewall blocking inbound ICMP echo. You can change these settings in the advanced firewall mmc to get this working.

2) I reversed out some of the config I had been told to add for DHCP to work in the /etc/sysctl.conf file so it now looks like this (note some of this is needed in a multi node cluster to be able to ping between VMs n ipv6):



3) finally I wanted to add in IPv6 connectivity as I am behind CGNAT on ipv4 and I have a IPv6 delegation from my ISP so I added that into the config and it now looks like this and is working just fine. I also added the post command to redirect the port so I can just type the ip to get to the management interface.

auto lo
iface lo inet loopback

iface eth0lan inet manual

iface eth1dmz inet manual

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eth0lan
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

iface vmbr0 inet6 manual
        accept_ra 2

auto vmbr0.100
iface vmbr0.100 inet static
        address 192.168.100.60/24
        gateway 192.168.100.1
        post-up iptables -t nat -A PREROUTING -p tcp -d 192.168.100.60 --dport 443 -j REDIRECT --to-ports 8006

iface vmbr0.100 inet6 static
        address {add_your_ipv6_address_here}/64
        gateway {add_your_ipv6_default_gateway_here}
        dns-nameservers {add_your_ipv6_dns_server_here}
        post-up ip6tables -t nat -A PREROUTING -p tcp -d {add_your_ipv6_address_here} --dport 443 -j REDIRECT --to-ports 8006

source /etc/network/interfaces.d/*