Proxmox in vlan setup - Subnet issues

N

nabbyBlue

Member
May 18, 2024
10
0
6
Hi I appolagise if i am in the wrong place. I was torn between here and installation/configuration. I have seen a lot of posts about changing the ip address of your proxmox setup and to be honest the web gui and cli both seem to work fine. I had an ip address of 192.168.1.99 when i setup the server. When I setup vlans on my home network I setup the ethernet port to the server dedicated lan. then I changed the IP address to 192.168.20.99 after moving the ethernet to the new port and reboots the server becomes unreachable. I have checked the cli and its all still online. I have tried the changes via both web gui and cli I can even see the machine on my router on the new subnet but still unreachable under the new IP address.

I also tried to just change my IP address but keeping it on the original subnet. that works fine so as far as I can see i am making the changes correctly its just as soon as I move it to a new subnet I have the issues. I have turned off all firewall rules to rule that out too.

Im not sure if I have to make the system vlan aware? I know that sounds silly but my understanding is that setting is if I want my virtual machines on differing subnets? in this case I dont.

any suggestions would be great. thanks
 
I use VLANs in my home lab, but when you set up Proxmox for the first time, you can't make it VLAN aware (at least I haven't figured out a way). So I go through the standard install, then from the console (no network required) I will edit /etc/network/interfaces and set up my network that way. I usually will put one interface as VLAN aware, and assign the Proxmox management interface an IP address on my management VLAN. I usually make my second NIC port as not VLAN aware, and assign it a static IP on my non-routed storage VLAN. All my workloads get two virtual NICs, one on vmbr0 assigned to whatever VLAN they are on, and one on vmbr1 which is my dedicated storage network. Any NFS shares are accessed over the dedicated network, as it exists on both my NAS devices too. This way there is no storage traffic contention with regular traffic and nothing has to go through pfSense for routing. In your case, if you don't have a dedicate VLAN, you can leave that out. Just copy my settings for vmbr0 and adjust the name of the NIC to match wherever is in your machine. If you are not sure, execute "ip a" from the terminal before editing /etc/network/interfaces. I usually just reboot at this point and can access the management interface from the new IP address.

Also bear in mind that you have to set up your switch ports correctly if you are using VLANs in Proxmox. The switch port feeding a VLAN aware interface has to be tagged/trunk port. Switch ports feeding a non-VLAN aware interface need to be untagged/access ports. You can configure this in the management interface for your switch.


Code: 
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp46s0 inet manual

iface enp46s0d1 inet manual

auto vmbr0
iface vmbr0 inet static
        bridge-ports enp46s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-100

auto vmbr0.10
iface vmbr0.100 inet static
        address 192.168.10.2/24
        gateway 192.168.10.1

auto vmbr1
iface vmbr1 inet static
        address 192.168.3.4/24
        bridge-ports enp46s0d1
        bridge-stp off
        bridge-fd 0


source /etc/network/interfaces.d/*
 
Good morning. Thanks for the reply. I dont have a extensive setup and my limited understanding leads me to believe that vlan aware is for when you have multiple virtual machines across multiple subnets? I have 2 vi's both on the same subnet as the host. As for my router setup. Whilst I have a switch already setup.and working across 3 vlans.
The proxmox machine is currently plugged into one of the ports on the router that has been allocated to the correct vlan. I also have a raspberry PI running on that vlan so my assumptions are its setup correctly. Its just proxmox doesn't want to work when the subnet is changed.

I am happy to be corrected if my assumptions are incorrect.
 
The first thing to do is to have your bridge interface to be VLAN-aware. Then, the etherport port that the physical network interface listens to must be trunked, i.e. let through all of your VLANs.

The configuration shown above is slightly incorrect, because it should read 'iface vmbr0.10 inet static' , but otherwise it would work, UNLESS you try to access the web UI on 192.168.3.4 from a differrent subnet!

If you see the IP on your router, then I would think that you have a routing problem like the one created by the setup shown above.

This is easy to create. Even, if in the configuration above, you would have create gateway 192.168.3.1, it would not work because you would create two gateways when there can only be one. WIthout a gateway, you will have return packets from 192.168.3.4 dropped because of split routes.

If you need two routes, one specific for when you access an IP in the second network, the setup has to be a little more complex, like mine:


Code: 
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eth1
iface eth1 inet manual
#10 Gbit/s Trunk

auto eth2
iface eth2 inet manual
        bridge-access 10
#10 Gbit/s Hulk Direct Access LAN only

auto eth0
iface eth0 inet manual
        accept_ra 0
        autoconf 0
        pre-up echo 0 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra
        post-up echo 0 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra
#Unused

auto vmbr0
iface vmbr0 inet static
        address 192.168.6.5/24
        bridge-ports eth1 eth2
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        bridge-mcsnoop 0
        post-up ip rule add from 192.168.6.5 table 60
        post-up ip route add 192.168.6.0/24 dev vmbr0 table 60
        post-up ip route add default via 192.168.6.1 table 60
        post-down ip route del default via 192.168.6.1 table 60
        post-down ip route del 192.168.6.0/24 dev vmbr0 table 60
        post-down ip rule del from 192.168.6.5 table 60
#Network bridge

auto lan0
iface lan0 inet static
        address 192.168.10.5/24
        gateway 192.168.10.1
        vlan-id 10
        vlan-raw-device vmbr0
        accept_ra 2
        autoconf 1
        pre-up echo 2 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra
        post-up echo 2 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra
#pve "vlan"

That setup has a few more tricks up its sleeve, like: IPv6, bridging two interfaces eth1 and eth2, the latter of which is used to directly link a machine that gets on VLAN 10 and some less obvious ones.

Also, it does not use vmbr0.10, but explicitely creates a virtual lan0 on VLAN 10. That way, "physical" interfaces and bridges are kept separate.

The special routing for 192.168.6.5 is neccessary, because otherwise, TCP connections addressed to 192.168.6.5 would have a split route over 192.168.10.1 so that the answer packets get nuked in the router.

Before you get into this, I would create only ONE IP on your PVE instance and try if it works.

You can also try if your VMs connected to your bridge (and having the correct VLAN assigned) work. Since they probably have only one leg into your network, they will probably work.
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back