[SOLVED] node disconnect from cluster when in vlan

rafafell

Well-Known Member
Sep 24, 2016
61
2
48
36
Hi,

When I put the ip of the node directly in the physical interface or in the vmbrX everything works fine, but when I pass this ip to vmbr.100 (see below) I lose the connection with the other nodes in the cluster and when I restart the node (despite the web interface loading ) I can't log into it

** I don't have direct access to the switch that connects these machines (I have to ask for adjustments if necessary - the less the better), but at first there are 1 ports forming a vlan for each node (tagged) and 1 port for the public ip (untagged)

Could anyone tell me why this happens? what am I doing wrong?

best

Code:
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface enp2s0 inet manual


auto enp1s0
iface enp1s0 inet static
        address xxx.xxx.xxx.xxw/27
        gateway xxx.xxx.xxx.xxy
#ip-public

iface vmbr0 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
 
auto vmbr1.100
iface vmbr1.100 inet static
        address 192.168.0.13/23
#ip-cluster


Code:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 04:d9:f5:06:da:9c brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.xxx.xxw/27 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 2801:88:cede:20:6d9:f5ff:fe06:da9c/64 scope global dynamic mngtmpaddr
       valid_lft 86250sec preferred_lft 43050sec
    inet6 fe80::6d9:f5ff:fe06:da9c/64 scope link
       valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UP group default qlen 1000
    link/ether 50:3e:aa:10:62:fe brd ff:ff:ff:ff:ff:ff
4: wlp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 84:fd:d1:5c:9d:88 brd ff:ff:ff:ff:ff:ff
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 50:3e:aa:10:62:fe brd ff:ff:ff:ff:ff:ff
    inet6 fe80::523e:aaff:fe10:62fe/64 scope link
       valid_lft forever preferred_lft forever
6: vmbr1.100@vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 50:3e:aa:10:62:fe brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.13/23 scope global vmbr1.100
       valid_lft forever preferred_lft forever
    inet6 fe80::523e:aaff:fe10:62fe/64 scope link
       valid_lft forever preferred_lft forever

Code:
proxmox-ve: 7.2-1 (running kernel: 5.15.35-1-pve)
pve-manager: 7.2-3 (running version: 7.2-3/c743d6c1)
pve-kernel-5.15: 7.2-3
pve-kernel-helper: 7.2-3
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.35-1-pve: 5.15.35-2
pve-kernel-5.13.19-6-pve: 5.13.19-15
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-6
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-1
libpve-network-perl: 0.7.1
libpve-storage-perl: 7.2-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
openvswitch-switch: 2.15.0+ds1-2+deb11u1
proxmox-backup-client: 2.1.8-1
proxmox-backup-file-restore: 2.1.8-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-10
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-1
pve-qemu-kvm: 6.2.0-5
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1
 
Last edited:
When I put the ip of the node directly in the physical interface or in the vmbrX everything works fine, but when I pass this ip to vmbr.100

* if you configure it on the physical interface/vmbr0 the packets get send out w/o VLAN tag (usually meaning that the switch appends the 'default', 'untagged' VLAN when sending them on)
* if you configure it on vmbr.100 they get sent out with a VLAN 100

if it works without explicit vlan - why bother changing the config?
depending on the switch config it might be that the VLAN 100 is the default VLAN for the port?
 
Hi @Stoiko Ivanov ,

my scenario is as follows:

- 1 node with two NICs (one exclusively for public IPs - untagged/access; and another for private ips - tagged/trunk)

- 5 node with one NIC (tagged/trunk)

** I don't have direct access to the switch that connects these machines (I have to ask for adjustments if necessary - the less the better), but at first there are 6 ports forming a vlan (tagged) and 1 port for the public ip (untagged)

- subnet 192.168.0.0/23 - ip-cluster (lan pfsense)
- subnet 192.168.4.1/23 - ip-vms (lan pfsense)
- subnet 192.168.2.0/30 - wan pfsense (via nat with public-ip interface)

the cluster ips are 192.168.0.10-15. When I link the cluster ips directly on the NIC interface or on vmbrX your subnet works fine. I can pass ips and internet link to the vms of all nodes, but I can't pass internet link to cluster nodes

with this subnet 192.168.4.0/23 works only on pfsense local node

My goals (which I'm not achieving):

(*) use the same NIC to pass the two subnets (192.168.0.0/23 and 192.168.4.0/23) - I thought of using vlan for this

(*) pass internet link for all cluster ips (192.168.0.10-15)

I tried linux bridge, OVS bridge and SDN but I didn't get it.

How do I use 6 ports (tagged/trunk) that form a vlan on a switch where I don't have direct access. I think the physical switch is not allowing the tag to pass, does it make sense?

Could you give me some guidance on how I can achieve the above goals? (I've been trying to move forward for a few days without success)
 
I thought of using vlan for this
usually you need to have the vlans also configured on the switch - I really would suggest you talk this through with your Housing/Network provider

* There are many options and it usually depends very much on their setup - you will need their cooperation

I hope this helps!
 
  • Like
Reactions: rafafell

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!