Hi!
I am not certain if this even possible to do. I have a PVE Cluster with 3 nodes as a homelab. I am now moving it to a 10Gbps connectivity using a Mikrotik switch and three 2x 10Gbps cards.
I was able to switch Ceph cluster network which is on its own dedicated port, as well as VLANs used by my guests to the new NICs, but I cannot switch the PVE Admin / Ceph client network. For reference:
* VLAN10 - 10.0.0.0/24 - Admin, Ceph client network
* VLAN20 - 10.163.20.0/24 - Trusted systems
* VLAN21 - 10.163.21.0/24 - Untrusted systems
* VLAN100 - ISP
* VLAN101 - Ceph cluster
I've tried changing `bridge-ports` for `vmbr10` to `enp3s0f0` however the node is not able to communicate on that VLAN when this is done. Rebooting does not help. I've tried doing this through Web UI and via SSH. All other connections work. I also confirmed I'm receiving VLAN10 traffic so the switch is configured correctly.
Is there something else I am missing, or can this simply not be done and I need to wipe each node individually and re-install?
Here is my `interfaces` file from one of the nodes:
I am not certain if this even possible to do. I have a PVE Cluster with 3 nodes as a homelab. I am now moving it to a 10Gbps connectivity using a Mikrotik switch and three 2x 10Gbps cards.
I was able to switch Ceph cluster network which is on its own dedicated port, as well as VLANs used by my guests to the new NICs, but I cannot switch the PVE Admin / Ceph client network. For reference:
* VLAN10 - 10.0.0.0/24 - Admin, Ceph client network
* VLAN20 - 10.163.20.0/24 - Trusted systems
* VLAN21 - 10.163.21.0/24 - Untrusted systems
* VLAN100 - ISP
* VLAN101 - Ceph cluster
I've tried changing `bridge-ports` for `vmbr10` to `enp3s0f0` however the node is not able to communicate on that VLAN when this is done. Rebooting does not help. I've tried doing this through Web UI and via SSH. All other connections work. I also confirmed I'm receiving VLAN10 traffic so the switch is configured correctly.
Is there something else I am missing, or can this simply not be done and I need to wipe each node individually and re-install?
Here is my `interfaces` file from one of the nodes:
Code:
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!
auto lo
iface lo inet loopback
iface eno1 inet manual
#office-sw 0/15
iface eno2 inet manual
#office-sw 0/16
iface enp3s0f0 inet manual
#pve-switch-1 sfp+1
iface enp3s0f1 inet manual
#pve-switch-1 sfp+4
auto vmbr10
iface vmbr10 inet static
address 10.0.0.41/24
gateway 10.0.0.1
bridge-ports eno1.10
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#bridge-ports enp3s0f0.10
#VLAN 10 Admin
auto vmbr100
iface vmbr100 inet manual
bridge-ports eno1.100
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#VLAN 100 ISP
auto vmbr101
iface vmbr101 inet static
address 10.0.101.41/24
bridge-ports enp3s0f1.101
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#VLAN 101 Ceph Cluster
auto vmbr20
iface vmbr20 inet static
address 10.163.20.41/24
bridge-ports enp3s0f0.20
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#VLAN 20 Trusted
auto vmbr21
iface vmbr21 inet static
address 10.163.21.41/24
bridge-ports enp3s0f0.21
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#VLAN 21 Untrusted