[SOLVED] node disconnect from cluster when in vlan

[rafafell]

Hi,

When I put the ip of the node directly in the physical interface or in the vmbrX everything works fine, but when I pass this ip to vmbr.100 (see below) I lose the connection with the other nodes in the cluster and when I restart the node (despite the web interface loading ) I can't log into it

** I don't have direct access to the switch that connects these machines (I have to ask for adjustments if necessary - the less the better), but at first there are 1 ports forming a vlan for each node (tagged) and 1 port for the public ip (untagged)

Could anyone tell me why this happens? what am I doing wrong?

best

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface enp2s0 inet manual


auto enp1s0
iface enp1s0 inet static
        address xxx.xxx.xxx.xxw/27
        gateway xxx.xxx.xxx.xxy
#ip-public

iface vmbr0 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
 
auto vmbr1.100
iface vmbr1.100 inet static
        address 192.168.0.13/23
#ip-cluster

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 04:d9:f5:06:da:9c brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.xxx.xxw/27 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 2801:88:cede:20:6d9:f5ff:fe06:da9c/64 scope global dynamic mngtmpaddr
       valid_lft 86250sec preferred_lft 43050sec
    inet6 fe80::6d9:f5ff:fe06:da9c/64 scope link
       valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UP group default qlen 1000
    link/ether 50:3e:aa:10:62:fe brd ff:ff:ff:ff:ff:ff
4: wlp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 84:fd:d1:5c:9d:88 brd ff:ff:ff:ff:ff:ff
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 50:3e:aa:10:62:fe brd ff:ff:ff:ff:ff:ff
    inet6 fe80::523e:aaff:fe10:62fe/64 scope link
       valid_lft forever preferred_lft forever
6: vmbr1.100@vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 50:3e:aa:10:62:fe brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.13/23 scope global vmbr1.100
       valid_lft forever preferred_lft forever
    inet6 fe80::523e:aaff:fe10:62fe/64 scope link
       valid_lft forever preferred_lft forever

proxmox-ve: 7.2-1 (running kernel: 5.15.35-1-pve)
pve-manager: 7.2-3 (running version: 7.2-3/c743d6c1)
pve-kernel-5.15: 7.2-3
pve-kernel-helper: 7.2-3
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.35-1-pve: 5.15.35-2
pve-kernel-5.13.19-6-pve: 5.13.19-15
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-6
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-1
libpve-network-perl: 0.7.1
libpve-storage-perl: 7.2-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
openvswitch-switch: 2.15.0+ds1-2+deb11u1
proxmox-backup-client: 2.1.8-1
proxmox-backup-file-restore: 2.1.8-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-10
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-1
pve-qemu-kvm: 6.2.0-5
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

 

[Stoiko Ivanov]

When I put the ip of the node directly in the physical interface or in the vmbrX everything works fine, but when I pass this ip to vmbr.100

* if you configure it on the physical interface/vmbr0 the packets get send out w/o VLAN tag (usually meaning that the switch appends the 'default', 'untagged' VLAN when sending them on)
* if you configure it on vmbr.100 they get sent out with a VLAN 100

if it works without explicit vlan - why bother changing the config?
depending on the switch config it might be that the VLAN 100 is the default VLAN for the port?

 

[rafafell]

Hi @Stoiko Ivanov ,

my scenario is as follows:

- 1 node with two NICs (one exclusively for public IPs - untagged/access; and another for private ips - tagged/trunk)

- 5 node with one NIC (tagged/trunk)

** I don't have direct access to the switch that connects these machines (I have to ask for adjustments if necessary - the less the better), but at first there are 6 ports forming a vlan (tagged) and 1 port for the public ip (untagged)

- subnet 192.168.0.0/23 - ip-cluster (lan pfsense)
- subnet 192.168.4.1/23 - ip-vms (lan pfsense)
- subnet 192.168.2.0/30 - wan pfsense (via nat with public-ip interface)

the cluster ips are 192.168.0.10-15. When I link the cluster ips directly on the NIC interface or on vmbrX your subnet works fine. I can pass ips and internet link to the vms of all nodes, but I can't pass internet link to cluster nodes

with this subnet 192.168.4.0/23 works only on pfsense local node

My goals (which I'm not achieving):

(*) use the same NIC to pass the two subnets (192.168.0.0/23 and 192.168.4.0/23) - I thought of using vlan for this

(*) pass internet link for all cluster ips (192.168.0.10-15)

I tried linux bridge, OVS bridge and SDN but I didn't get it.

How do I use 6 ports (tagged/trunk) that form a vlan on a switch where I don't have direct access. I think the physical switch is not allowing the tag to pass, does it make sense?

Could you give me some guidance on how I can achieve the above goals? (I've been trying to move forward for a few days without success)

 

[Stoiko Ivanov]

I thought of using vlan for this

usually you need to have the vlans also configured on the switch - I really would suggest you talk this through with your Housing/Network provider

* There are many options and it usually depends very much on their setup - you will need their cooperation

I hope this helps!

 

[rafafell]

Sorted out! Really were the switch ports that formed the vlan were configured with trunk/tagged for just one tag that they had not informed me.

see also: https://forum.proxmox.com/threads/vlan-tagging.79830/