Corosync - nsscrypto: Incorrect packet size

D

Dudeplayz

Member
Proxmox Subscriber
Jan 23, 2020
22
1
8
27
Hello,
after upgrading from pve6 to pve7, the syslog of both servers is spammed with:
Code: 
corosync[8001]: [KNET  ] nsscrypto: Incorrect packet size.

After some googleing I haven't found a solution yet.
Hopfefully someone can help.

1636535131008.png

Edit (Solution):
Related corosync github issue
The reason was the running corosync service on my QDevice (RPi), which is not needed in this setup. Only corosync-qnetd is needed to supply a vote to the cluster. The problem already existed before the upgrade to pve7. An update in knet changed the filtering of invalid packets, which results in the spammed Syslog.
 
Last edited:
Bash: 
root@pve2:~# pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-4-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.4: 6.4-5
pve-kernel-5.0: 6.0-11
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.4.128-1-pve: 5.4.128-2
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-10
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-12
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.11-1
proxmox-backup-file-restore: 2.0.11-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.1-1
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-4
pve-firmware: 3.3-2
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-14
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

Bash: 
root@pve2:~# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.41.10
    ring1_addr: 192.168.40.11
  }
  node {
    name: pve2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.41.20
    ring1_addr: 192.168.40.21
  }
}

quorum {
  device {
    model: net
    net {
      algorithm: ffsplit
      host: 192.168.41.30
      tls: on
    }
    votes: 1
  }
  provider: corosync_votequorum
}

totem {
  cluster_name: PVE-Cluster
  config_version: 5
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  secauth: on
  version: 2
}
 
Bash: 
root@pve2:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP group default qlen 1000
    link/ether ac:1f:6b:ad:01:ac brd ff:ff:ff:ff:ff:ff
    altname enp196s0
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond1 state DOWN group default qlen 1000
    link/ether ac:1f:6b:ad:01:ac brd ff:ff:ff:ff:ff:ff permaddr ac:1f:6b:ad:01:ad
    altname enp197s0
4: enp129s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether ac:1f:6b:ac:99:de brd ff:ff:ff:ff:ff:ff
5: enp129s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether ac:1f:6b:ac:99:de brd ff:ff:ff:ff:ff:ff permaddr ac:1f:6b:ac:99:df
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether ac:1f:6b:ac:99:de brd ff:ff:ff:ff:ff:ff
7: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether ac:1f:6b:ad:01:ac brd ff:ff:ff:ff:ff:ff
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:ac:99:de brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ae1f:6bff:feac:99de/64 scope link
       valid_lft forever preferred_lft forever
9: vmbr0.40@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:ac:99:de brd ff:ff:ff:ff:ff:ff
    inet 192.168.40.21/24 brd 192.168.40.255 scope global vmbr0.40
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:feac:99de/64 scope link
       valid_lft forever preferred_lft forever
10: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:ad:01:ac brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ae1f:6bff:fead:1ac/64 scope link
       valid_lft forever preferred_lft forever
11: vmbr1.41@vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:ad:01:ac brd ff:ff:ff:ff:ff:ff
    inet 192.168.41.20/24 brd 192.168.41.255 scope global vmbr1.41
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:fead:1ac/64 scope link
       valid_lft forever preferred_lft forever
12: tap202i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr202i0 state UNKNOWN group default qlen 1000
    link/ether ae:b6:ae:93:86:d4 brd ff:ff:ff:ff:ff:ff
13: fwbr202i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 4a:3f:ca:01:d6:c2 brd ff:ff:ff:ff:ff:ff
14: fwpr202p0@fwln202i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 9a:81:25:81:22:7b brd ff:ff:ff:ff:ff:ff
15: fwln202i0@fwpr202p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr202i0 state UP group default qlen 1000
    link/ether 2e:b8:9b:2d:19:2c brd ff:ff:ff:ff:ff:ff
28: tap214i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr214i0 state UNKNOWN group default qlen 1000
    link/ether 8a:61:7d:4f:c6:6a brd ff:ff:ff:ff:ff:ff
29: fwbr214i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:a8:e3:75:64:83 brd ff:ff:ff:ff:ff:ff
30: fwpr214p0@fwln214i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 8a:c1:a1:8f:5b:de brd ff:ff:ff:ff:ff:ff
31: fwln214i0@fwpr214p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr214i0 state UP group default qlen 1000
    link/ether 3a:c5:3e:cc:e1:9c brd ff:ff:ff:ff:ff:ff
32: veth100i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000
    link/ether fe:ec:43:43:9b:63 brd ff:ff:ff:ff:ff:ff link-netnsid 2
33: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d2:01:aa:eb:bf:68 brd ff:ff:ff:ff:ff:ff
34: fwpr100p0@fwln100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fa:db:60:d5:93:91 brd ff:ff:ff:ff:ff:ff
35: fwln100i0@fwpr100p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000
    link/ether 7e:c0:ea:fb:d3:c7 brd ff:ff:ff:ff:ff:ff
36: tap216i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr216i0 state UNKNOWN group default qlen 1000
    link/ether 3a:a4:ba:56:e8:76 brd ff:ff:ff:ff:ff:ff
37: fwbr216i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f6:c6:6b:bf:8e:ee brd ff:ff:ff:ff:ff:ff
38: fwpr216p0@fwln216i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether ce:df:f5:07:4b:5d brd ff:ff:ff:ff:ff:ff
39: fwln216i0@fwpr216p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr216i0 state UP group default qlen 1000
    link/ether ca:50:96:f6:d7:1f brd ff:ff:ff:ff:ff:ff
44: veth103i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr103i0 state UP group default qlen 1000
    link/ether fe:de:8f:7c:1a:9f brd ff:ff:ff:ff:ff:ff link-netnsid 4
45: fwbr103i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 26:76:f4:d4:0c:2f brd ff:ff:ff:ff:ff:ff
46: fwpr103p0@fwln103i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 92:06:e4:bf:fa:a7 brd ff:ff:ff:ff:ff:ff
47: fwln103i0@fwpr103p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr103i0 state UP group default qlen 1000
    link/ether ae:96:3d:ec:21:f3 brd ff:ff:ff:ff:ff:ff
48: veth101i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr101i0 state UP group default qlen 1000
    link/ether fe:91:96:a8:5c:7f brd ff:ff:ff:ff:ff:ff link-netnsid 5
49: fwbr101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ae:5b:af:a3:69:89 brd ff:ff:ff:ff:ff:ff
50: fwpr101p0@fwln101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether d2:0a:da:b5:81:84 brd ff:ff:ff:ff:ff:ff
51: fwln101i0@fwpr101p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr101i0 state UP group default qlen 1000
    link/ether 76:61:f1:ac:b1:f0 brd ff:ff:ff:ff:ff:ff
52: tap211i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr211i0 state UNKNOWN group default qlen 1000
    link/ether ba:35:bd:53:49:be brd ff:ff:ff:ff:ff:ff
53: fwbr211i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0e:45:23:82:fc:3a brd ff:ff:ff:ff:ff:ff
54: fwpr211p0@fwln211i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 1e:bd:b0:22:09:d8 brd ff:ff:ff:ff:ff:ff
55: fwln211i0@fwpr211p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr211i0 state UP group default qlen 1000
    link/ether d6:de:df:f3:7d:12 brd ff:ff:ff:ff:ff:ff
64: veth105i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr105i0 state UP group default qlen 1000
    link/ether fe:41:2f:a7:9f:de brd ff:ff:ff:ff:ff:ff link-netnsid 6
65: fwbr105i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b2:51:8c:ef:77:6b brd ff:ff:ff:ff:ff:ff
66: fwpr105p0@fwln105i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether ee:94:e9:cb:92:14 brd ff:ff:ff:ff:ff:ff
67: fwln105i0@fwpr105p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr105i0 state UP group default qlen 1000
    link/ether 6a:f9:10:3e:24:8c brd ff:ff:ff:ff:ff:ff
80: tap219i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr219i0 state UNKNOWN group default qlen 1000
    link/ether 92:e1:27:67:56:70 brd ff:ff:ff:ff:ff:ff
81: fwbr219i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b2:72:dd:d3:6d:cc brd ff:ff:ff:ff:ff:ff
82: fwpr219p0@fwln219i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 8a:a8:d3:53:1c:a4 brd ff:ff:ff:ff:ff:ff
83: fwln219i0@fwpr219p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr219i0 state UP group default qlen 1000
    link/ether 82:c1:79:d0:f8:7a brd ff:ff:ff:ff:ff:ff
100: tap206i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr206i0 state UNKNOWN group default qlen 1000
    link/ether a6:3e:75:e6:cc:aa brd ff:ff:ff:ff:ff:ff
101: fwbr206i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 72:8c:d7:20:a2:6f brd ff:ff:ff:ff:ff:ff
102: fwpr206p0@fwln206i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether b6:64:1e:a8:d9:7b brd ff:ff:ff:ff:ff:ff
103: fwln206i0@fwpr206p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr206i0 state UP group default qlen 1000
    link/ether ee:50:82:c2:ef:ac brd ff:ff:ff:ff:ff:ff
128: veth109i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr109i0 state UP group default qlen 1000
    link/ether fe:bf:aa:7a:b2:01 brd ff:ff:ff:ff:ff:ff link-netnsid 3
129: fwbr109i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d2:2a:12:e5:95:47 brd ff:ff:ff:ff:ff:ff
130: fwpr109p0@fwln109i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 32:b1:67:94:c2:a2 brd ff:ff:ff:ff:ff:ff
131: fwln109i0@fwpr109p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr109i0 state UP group default qlen 1000
    link/ether b2:7e:c8:99:ea:f6 brd ff:ff:ff:ff:ff:ff
132: veth102i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr102i0 state UP group default qlen 1000
    link/ether fe:ca:e5:c8:41:7e brd ff:ff:ff:ff:ff:ff link-netnsid 0
133: fwbr102i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:14:4a:d0:bc:b7 brd ff:ff:ff:ff:ff:ff
134: fwpr102p0@fwln102i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 2e:77:59:0e:91:53 brd ff:ff:ff:ff:ff:ff
135: fwln102i0@fwpr102p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr102i0 state UP group default qlen 1000
    link/ether 56:94:81:87:24:9b brd ff:ff:ff:ff:ff:ff
172: tap200i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr200i0 state UNKNOWN group default qlen 1000
    link/ether 86:22:74:4c:2e:94 brd ff:ff:ff:ff:ff:ff
173: fwbr200i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 96:e8:d3:dd:94:24 brd ff:ff:ff:ff:ff:ff
174: fwpr200p0@fwln200i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 42:70:dd:c8:98:74 brd ff:ff:ff:ff:ff:ff
175: fwln200i0@fwpr200p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr200i0 state UP group default qlen 1000
    link/ether 26:c9:eb:67:55:fa brd ff:ff:ff:ff:ff:ff
176: tap212i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr212i0 state UNKNOWN group default qlen 1000
    link/ether a6:d1:d4:02:e0:31 brd ff:ff:ff:ff:ff:ff
177: fwbr212i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether be:19:78:2a:0e:ed brd ff:ff:ff:ff:ff:ff
178: fwpr212p0@fwln212i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 62:9e:3d:ea:eb:01 brd ff:ff:ff:ff:ff:ff
179: fwln212i0@fwpr212p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr212i0 state UP group default qlen 1000
    link/ether ea:41:ed:66:4d:3b brd ff:ff:ff:ff:ff:ff
 
could you post the output of those commands / contents of those files for both nodes please?
 
Second node:

Bash: 
root@pve1:~# pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-4-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.4: 6.4-5
pve-kernel-5.0: 6.0-11
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.4.128-1-pve: 5.4.128-2
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-10
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-12
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.11-1
proxmox-backup-file-restore: 2.0.11-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.1-1
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-4
pve-firmware: 3.3-2
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-14
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

Bash: 
root@pve1:~# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.41.10
    ring1_addr: 192.168.40.11
  }
  node {
    name: pve2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.41.20
    ring1_addr: 192.168.40.21
  }
}

quorum {
  device {
    model: net
    net {
      algorithm: ffsplit
      host: 192.168.41.30
      tls: on
    }
    votes: 1
  }
  provider: corosync_votequorum
}

totem {
  cluster_name: PVE-Cluster
  config_version: 5
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  secauth: on
  version: 2
}
 
Bash: 
root@pve1:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP group default qlen 1000
    link/ether 0a:e5:fd:31:10:a5 brd ff:ff:ff:ff:ff:ff permaddr ac:1f:6b:ad:2a:74
    altname enp4s0
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond1 state DOWN group default qlen 1000
    link/ether 0a:e5:fd:31:10:a5 brd ff:ff:ff:ff:ff:ff permaddr ac:1f:6b:ad:2a:75
    altname enp5s0
4: enp99s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 92:f0:af:8e:58:e9 brd ff:ff:ff:ff:ff:ff permaddr ac:1f:6b:ac:90:26
5: enp99s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 92:f0:af:8e:58:e9 brd ff:ff:ff:ff:ff:ff permaddr ac:1f:6b:ac:90:27
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 92:f0:af:8e:58:e9 brd ff:ff:ff:ff:ff:ff
7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 92:f0:af:8e:58:e9 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::90f0:afff:fe8e:58e9/64 scope link
       valid_lft forever preferred_lft forever
8: vmbr0.40@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 92:f0:af:8e:58:e9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.40.11/24 scope global vmbr0.40
       valid_lft forever preferred_lft forever
    inet6 fe80::90f0:afff:fe8e:58e9/64 scope link
       valid_lft forever preferred_lft forever
9: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether 0a:e5:fd:31:10:a5 brd ff:ff:ff:ff:ff:ff
10: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:e5:fd:31:10:a5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8e5:fdff:fe31:10a5/64 scope link
       valid_lft forever preferred_lft forever
11: vmbr1.41@vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:e5:fd:31:10:a5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.41.10/24 scope global vmbr1.41
       valid_lft forever preferred_lft forever
    inet6 fe80::8e5:fdff:fe31:10a5/64 scope link
       valid_lft forever preferred_lft forever
12: tap201i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr201i0 state UNKNOWN group default qlen 1000
    link/ether 82:b4:d1:65:08:c9 brd ff:ff:ff:ff:ff:ff
13: fwbr201i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d6:41:6d:76:5d:c7 brd ff:ff:ff:ff:ff:ff
14: fwpr201p0@fwln201i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 26:4f:33:c6:f3:51 brd ff:ff:ff:ff:ff:ff
15: fwln201i0@fwpr201p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr201i0 state UP group default qlen 1000
    link/ether 2e:a8:aa:e0:7b:4d brd ff:ff:ff:ff:ff:ff
16: tap205i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr205i0 state UNKNOWN group default qlen 1000
    link/ether 56:cf:57:7e:34:60 brd ff:ff:ff:ff:ff:ff
17: fwbr205i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ce:52:63:6c:5b:ee brd ff:ff:ff:ff:ff:ff
18: fwpr205p0@fwln205i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 7e:88:cf:07:a3:0d brd ff:ff:ff:ff:ff:ff
19: fwln205i0@fwpr205p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr205i0 state UP group default qlen 1000
    link/ether be:e4:10:a3:f7:d5 brd ff:ff:ff:ff:ff:ff
20: veth104i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr104i0 state UP group default qlen 1000
    link/ether fe:20:8a:7d:a2:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0
21: fwbr104i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 2e:7f:32:2f:80:42 brd ff:ff:ff:ff:ff:ff
22: fwpr104p0@fwln104i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether c6:da:3b:b9:4c:8e brd ff:ff:ff:ff:ff:ff
23: fwln104i0@fwpr104p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr104i0 state UP group default qlen 1000
    link/ether 92:e2:73:29:c4:d1 brd ff:ff:ff:ff:ff:ff
24: veth106i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr106i0 state UP group default qlen 1000
    link/ether fe:a9:1d:33:3b:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 1
25: fwbr106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 42:08:7a:a0:7a:50 brd ff:ff:ff:ff:ff:ff
26: fwpr106p0@fwln106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether ca:43:d4:21:00:8f brd ff:ff:ff:ff:ff:ff
27: fwln106i0@fwpr106p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr106i0 state UP group default qlen 1000
    link/ether 0e:db:77:ac:08:e6 brd ff:ff:ff:ff:ff:ff
28: veth107i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr107i0 state UP group default qlen 1000
    link/ether fe:6a:e2:66:58:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 2
29: fwbr107i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 4e:81:c9:cf:d1:ed brd ff:ff:ff:ff:ff:ff
30: fwpr107p0@fwln107i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether a2:53:ab:4f:fa:ed brd ff:ff:ff:ff:ff:ff
31: fwln107i0@fwpr107p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr107i0 state UP group default qlen 1000
    link/ether 9e:94:a0:75:56:34 brd ff:ff:ff:ff:ff:ff
32: veth108i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr108i0 state UP group default qlen 1000
    link/ether fe:4e:8e:4c:e8:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 3
33: fwbr108i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 16:01:41:5c:75:d9 brd ff:ff:ff:ff:ff:ff
34: fwpr108p0@fwln108i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 52:b9:85:65:28:c8 brd ff:ff:ff:ff:ff:ff
35: fwln108i0@fwpr108p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr108i0 state UP group default qlen 1000
    link/ether 4e:00:4d:84:5d:1b brd ff:ff:ff:ff:ff:ff
36: veth111i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr111i0 state UP group default qlen 1000
    link/ether fe:c1:d4:fd:d8:37 brd ff:ff:ff:ff:ff:ff link-netnsid 4
37: fwbr111i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d6:06:d1:60:ff:97 brd ff:ff:ff:ff:ff:ff
38: fwpr111p0@fwln111i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 46:c1:75:83:d7:62 brd ff:ff:ff:ff:ff:ff
39: fwln111i0@fwpr111p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr111i0 state UP group default qlen 1000
    link/ether ba:52:bc:99:e0:5c brd ff:ff:ff:ff:ff:ff
40: veth112i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr112i0 state UP group default qlen 1000
    link/ether fe:69:d9:63:14:d1 brd ff:ff:ff:ff:ff:ff link-netnsid 5
41: fwbr112i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 46:28:93:bf:cb:13 brd ff:ff:ff:ff:ff:ff
42: fwpr112p0@fwln112i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fa:57:46:92:07:e5 brd ff:ff:ff:ff:ff:ff
43: fwln112i0@fwpr112p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr112i0 state UP group default qlen 1000
    link/ether aa:12:42:e5:2b:83 brd ff:ff:ff:ff:ff:ff
45: veth115i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr115i0 state UP group default qlen 1000
    link/ether fe:58:ec:ae:7a:9d brd ff:ff:ff:ff:ff:ff link-netnsid 7
46: fwbr115i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 4a:f8:6d:75:4c:57 brd ff:ff:ff:ff:ff:ff
47: fwpr115p0@fwln115i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 62:81:82:54:95:fb brd ff:ff:ff:ff:ff:ff
48: fwln115i0@fwpr115p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr115i0 state UP group default qlen 1000
    link/ether 66:d8:01:69:b9:49 brd ff:ff:ff:ff:ff:ff
49: tap140i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr140i0 state UNKNOWN group default qlen 1000
    link/ether 6e:b6:37:1a:5b:ba brd ff:ff:ff:ff:ff:ff
50: fwbr140i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 96:6e:2a:96:46:78 brd ff:ff:ff:ff:ff:ff
51: fwpr140p0@fwln140i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether ce:32:72:4f:5c:45 brd ff:ff:ff:ff:ff:ff
52: fwln140i0@fwpr140p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr140i0 state UP group default qlen 1000
    link/ether 36:61:de:d0:bd:66 brd ff:ff:ff:ff:ff:ff
57: tap207i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr207i0 state UNKNOWN group default qlen 1000
    link/ether 7a:10:c9:e5:81:e7 brd ff:ff:ff:ff:ff:ff
58: fwbr207i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:c4:1f:4e:21:bf brd ff:ff:ff:ff:ff:ff
59: fwpr207p0@fwln207i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 42:1c:b4:7f:3d:56 brd ff:ff:ff:ff:ff:ff
60: fwln207i0@fwpr207p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr207i0 state UP group default qlen 1000
    link/ether 72:9e:a7:4c:02:20 brd ff:ff:ff:ff:ff:ff
61: veth110i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr110i0 state UP group default qlen 1000
    link/ether fe:ff:29:8f:ca:e9 brd ff:ff:ff:ff:ff:ff link-netnsid 8
62: fwbr110i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 46:07:9e:12:70:ae brd ff:ff:ff:ff:ff:ff
63: fwpr110p0@fwln110i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 3e:be:cf:94:22:37 brd ff:ff:ff:ff:ff:ff
64: fwln110i0@fwpr110p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr110i0 state UP group default qlen 1000
    link/ether aa:92:20:96:80:4f brd ff:ff:ff:ff:ff:ff
65: veth113i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:27:41:82:83:63 brd ff:ff:ff:ff:ff:ff link-netnsid 9
66: veth116i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr116i0 state UP group default qlen 1000
    link/ether fe:b6:5e:b9:79:2a brd ff:ff:ff:ff:ff:ff link-netnsid 6
67: fwbr116i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6e:73:ab:e4:c1:40 brd ff:ff:ff:ff:ff:ff
68: fwpr116p0@fwln116i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether a2:08:19:21:a0:4c brd ff:ff:ff:ff:ff:ff
69: fwln116i0@fwpr116p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr116i0 state UP group default qlen 1000
    link/ether e6:db:20:c8:45:75 brd ff:ff:ff:ff:ff:ff
70: veth117i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr117i0 state UP group default qlen 1000
    link/ether fe:e5:cd:8a:59:2b brd ff:ff:ff:ff:ff:ff link-netnsid 10
71: fwbr117i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a2:db:e0:ae:4a:f5 brd ff:ff:ff:ff:ff:ff
72: fwpr117p0@fwln117i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 3a:2d:e1:d3:44:31 brd ff:ff:ff:ff:ff:ff
73: fwln117i0@fwpr117p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr117i0 state UP group default qlen 1000
    link/ether e2:94:a4:86:27:87 brd ff:ff:ff:ff:ff:ff
78: tap155i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr155i0 state UNKNOWN group default qlen 1000
    link/ether fe:30:3c:a8:7a:65 brd ff:ff:ff:ff:ff:ff
79: fwbr155i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether c6:a8:63:55:66:87 brd ff:ff:ff:ff:ff:ff
80: fwpr155p0@fwln155i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 2e:a0:8c:36:ff:84 brd ff:ff:ff:ff:ff:ff
81: fwln155i0@fwpr155p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr155i0 state UP group default qlen 1000
    link/ether 72:7d:b6:c8:a4:d4 brd ff:ff:ff:ff:ff:ff
82: tap217i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr217i0 state UNKNOWN group default qlen 1000
    link/ether c6:f9:68:be:3e:a1 brd ff:ff:ff:ff:ff:ff
83: fwbr217i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 4a:56:7b:98:ce:4e brd ff:ff:ff:ff:ff:ff
84: fwpr217p0@fwln217i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 2a:05:cb:3f:15:d6 brd ff:ff:ff:ff:ff:ff
85: fwln217i0@fwpr217p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr217i0 state UP group default qlen 1000
    link/ether 76:8f:5b:33:c1:2b brd ff:ff:ff:ff:ff:ff
 
and the configs in /etc/corosync/corosync.conf (on both nodes)? could you also provide (you can cut down on duplicate log spam) journalctl -b -u corosync from both nodes, especially the parts printed directly after the last startup and any things related to config reloading would be interesting.
 
Bash: 
root@pve1:~# cat /etc/corosync/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.41.10
    ring1_addr: 192.168.40.11
  }
  node {
    name: pve2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.41.20
    ring1_addr: 192.168.40.21
  }
}

quorum {
  device {
    model: net
    net {
      algorithm: ffsplit
      host: 192.168.41.30
      tls: on
    }
    votes: 1
  }
  provider: corosync_votequorum
}

totem {
  cluster_name: PVE-Cluster
  config_version: 5
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  secauth: on
  version: 2
}

Bash: 
root@pve2:~# cat /etc/corosync/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.41.10
    ring1_addr: 192.168.40.11
  }
  node {
    name: pve2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.41.20
    ring1_addr: 192.168.40.21
  }
}

quorum {
  device {
    model: net
    net {
      algorithm: ffsplit
      host: 192.168.41.30
      tls: on
    }
    votes: 1
  }
  provider: corosync_votequorum
}

totem {
  cluster_name: PVE-Cluster
  config_version: 5
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  secauth: on
  version: 2
}
 
could you restart corosync on both nodes? does the message persist? it it does, could you set debug to on in corosync.conf and again restart corosync and grab the startup log messages from both nodes? thanks
 
also, is there any sort of firewall/VPN/network middle box between the nodes that might interfere with (some subset of) the corosync traffic? because of the consistent stream of messages it's most likely the ping/pong messages that trigger it, but it can't be all of them else your knet links would go down and your cluster would fall apart ;)
 
Here is the log after restarting corosync with debug information.

It is now also always logging corosync[3787721]: [KNET ] rx: Unable to decrypt/auth packet.
 

Attachments

  • corosync-reverse.log
    92.3 KB · Views: 2
fabian said:
also, is there any sort of firewall/VPN/network middle box between the nodes that might interfere with (some subset of) the corosync traffic? because of the consistent stream of messages it's most likely the ping/pong messages that trigger it, but it can't be all of them else your knet links would go down and your cluster would fall apart ;)
Click to expand...
They hang on the same switch in the same IP-Net. So it shouldn't pass through the router. Still it is confusing, why this first appeared after pve upgrade. Before everything was normal. Maybe it's a problem with the separate QDevice?
 
no, the qdevice is not operating at the knet layer at all.. did you restart corosync on both nodes?
 
and could you post the network config (/etc/network/interfaces) from both nodes? something is mangling packets here, the question is who/what/where.. my next best guess would be to run tcpdump and monitor traffic on both nodes and links
 
Bash: 
root@pve1:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto enp99s0f0
iface enp99s0f0 inet manual

auto enp99s0f1
iface enp99s0f1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp99s0f0 enp99s0f1
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp99s0f0
#10Gb Uplink - 1Gb Active Backup

auto bond1
iface bond1 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
#Corosync LACP

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#Uplink VLAN Bridge

auto vmbr0.40
iface vmbr0.40 inet static
        address 192.168.40.11/24
        gateway 192.168.40.1

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#Corosync VLAN Bridge

auto vmbr1.41
iface vmbr1.41 inet static
        address 192.168.41.10/24
#Corosync

Bash: 
root@pve2:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto enp129s0f0
iface enp129s0f0 inet manual

auto enp129s0f1
iface enp129s0f1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp129s0f0 enp129s0f1
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp129s0f0
#10Gb Uplink Active Backup

auto bond1
iface bond1 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
#Corosync LACP

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#Uplink VLAN Bridge

auto vmbr0.40
iface vmbr0.40 inet static
        address 192.168.40.21/24
        gateway 192.168.40.1

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#Corosync VLAN Bridge

auto vmbr1.41
iface vmbr1.41 inet static
        address 192.168.41.20/24
#Corosync
 
fabian said:
no, the qdevice is not operating at the knet layer at all.. did you restart corosync on both nodes?
Click to expand...
Yes before I restarted the service on both nodes. The debug log I only tested and one. Here is the log of the second one after the restart with debugging on.no, the qdevice is not operating at the knet layer at all.. did you restart corosync on both nodes?
 

Attachments

  • corosync-reverse.log
    18.9 KB · Views: 2
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back