Cluster corosync timeout

[showiproute]

Hello everyone,

is it possible to define a timout when a server should reboot if there is no quorum ?
I am experience the problem that two PVE servers are being on the same switch and if the switch will be rebooted both PVE servers will crash and reboot.

 

[Moayad]

Can you share what happened in the syslog when a node rebooted? This might help us identify what triggered the reboots.

Have you enabled the HA on those nodes?

 

[Moayad]

Can you also post the output of the following command?

last reboot -n 10

 

[showiproute]

I checked journalctl for that - hope that is sufficient as there is no syslog at /var/log

Oct 11 16:04:26 proxmox1 nut-driver@powerwalker[2924084]: No supported devices found. Please check your device availability with 'lsusb'

Oct 11 16:04:26 proxmox1 nut-driver@powerwalker[2924084]: and make sure you have an up-to-date version of NUT. If this does not help,

Oct 11 16:04:26 proxmox1 nut-driver@powerwalker[2924084]: try running the driver with at least 'subdriver', 'vendorid' and 'productid'

Oct 11 16:04:26 proxmox1 nut-driver@powerwalker[2924084]: options specified. Please refer to the man page for details about these options

Oct 11 16:04:26 proxmox1 nut-driver@powerwalker[2924084]: (man 8 nutdrv_qx).

Oct 11 16:04:26 proxmox1 nut-driver@powerwalker[2924083]: Driver failed to start (exit status=1)

Oct 11 16:04:28 proxmox1 corosync-qdevice[3192]: Connect timeout

Oct 11 16:04:31 proxmox1 nut-driver@powerwalker[2924121]: Network UPS Tools - Generic Q* USB/Serial driver 0.32 (2.8.0)

Oct 11 16:04:31 proxmox1 nut-driver@powerwalker[2924121]: USB communication driver (libusb 1.0) 0.43

Oct 11 16:04:31 proxmox1 nut-driver@powerwalker[2924121]: libusb1: Could not open any HID devices: insufficient permissions on everything

Oct 11 16:04:31 proxmox1 nut-driver@powerwalker[2924121]: No supported devices found. Please check your device availability with 'lsusb'

Oct 11 16:04:31 proxmox1 nut-driver@powerwalker[2924121]: and make sure you have an up-to-date version of NUT. If this does not help,

Oct 11 16:04:31 proxmox1 nut-driver@powerwalker[2924121]: try running the driver with at least 'subdriver', 'vendorid' and 'productid'

Oct 11 16:04:31 proxmox1 nut-driver@powerwalker[2924121]: options specified. Please refer to the man page for details about these options

Oct 11 16:04:31 proxmox1 nut-driver@powerwalker[2924121]: (man 8 nutdrv_qx).

Oct 11 16:04:31 proxmox1 nut-driver@powerwalker[2924083]: Driver failed to start (exit status=1)

Oct 11 16:04:31 proxmox1 corosync-qdevice[3192]: Can't connect to qnetd host. (-5986): Network address not available (in use?)

Oct 11 16:04:36 proxmox1 pvestatd[3204]: PBS: error fetching datastores - 500 Can't connect to proxmox2.hks.lan:8007 (Temporary failure in name resolution)

Oct 11 16:04:36 proxmox1 corosync-qdevice[3192]: Connect timeout

Oct 11 16:04:37 proxmox1 nut-driver@powerwalker[2924176]: Network UPS Tools - Generic Q* USB/Serial driver 0.32 (2.8.0)

Oct 11 16:04:37 proxmox1 nut-driver@powerwalker[2924176]: USB communication driver (libusb 1.0) 0.43

Oct 11 16:04:37 proxmox1 nut-driver@powerwalker[2924176]: libusb1: Could not open any HID devices: insufficient permissions on everything

Oct 11 16:04:37 proxmox1 nut-driver@powerwalker[2924176]: No supported devices found. Please check your device availability with 'lsusb'

Oct 11 16:04:37 proxmox1 nut-driver@powerwalker[2924176]: and make sure you have an up-to-date version of NUT. If this does not help,

Oct 11 16:04:37 proxmox1 nut-driver@powerwalker[2924176]: try running the driver with at least 'subdriver', 'vendorid' and 'productid'

Oct 11 16:04:37 proxmox1 nut-driver@powerwalker[2924176]: options specified. Please refer to the man page for details about these options

Oct 11 16:04:37 proxmox1 nut-driver@powerwalker[2924176]: (man 8 nutdrv_qx).

Oct 11 16:04:37 proxmox1 nut-driver@powerwalker[2924083]: Driver failed to start (exit status=1)

Oct 11 16:04:37 proxmox1 nut-driver@powerwalker[2924083]: Network UPS Tools - UPS driver controller 2.8.0

Oct 11 16:04:37 proxmox1 systemd[1]: nut-driver@powerwalker.service: Control process exited, code=exited, status=1/FAILURE

Oct 11 16:04:37 proxmox1 systemd[1]: nut-driver@powerwalker.service: Failed with result 'exit-code'.

Oct 11 16:04:37 proxmox1 systemd[1]: Failed to start nut-driver@powerwalker.service - Network UPS Tools - device driver for NUT device 'powerwalker'.

Oct 11 16:04:39 proxmox1 corosync-qdevice[3192]: Can't connect to qnetd host. (-5986): Network address not available (in use?)

Oct 11 16:04:44 proxmox1 corosync-qdevice[3192]: Connect timeout

Oct 11 16:04:47 proxmox1 corosync-qdevice[3192]: Can't connect to qnetd host. (-5986): Network address not available (in use?)

Oct 11 16:04:48 proxmox1 pvestatd[3204]: Tuxis: error fetching datastores - 500 Can't connect to pbs001.tuxis.nl:8007 (Temporary failure in name resolution)

Oct 11 16:04:49 proxmox1 pvestatd[3204]: status update time (46.757 seconds)

Oct 11 16:04:52 proxmox1 systemd[1]: nut-driver@powerwalker.service: Scheduled restart job, restart counter is at 56329.

Oct 11 16:04:52 proxmox1 systemd[1]: Stopped nut-driver@powerwalker.service - Network UPS Tools - device driver for NUT device 'powerwalker'.

Oct 11 16:04:52 proxmox1 systemd[1]: Starting nut-driver@powerwalker.service - Network UPS Tools - device driver for NUT device 'powerwalker'...

Oct 11 16:04:52 proxmox1 nut-driver@powerwalker[2924309]: Network UPS Tools - Generic Q* USB/Serial driver 0.32 (2.8.0)

Oct 11 16:04:52 proxmox1 nut-driver@powerwalker[2924309]: USB communication driver (libusb 1.0) 0.43

Oct 11 16:04:52 proxmox1 nut-driver@powerwalker[2924309]: libusb1: Could not open any HID devices: insufficient permissions on everything

Oct 11 16:04:52 proxmox1 nut-driver@powerwalker[2924309]: No supported devices found. Please check your device availability with 'lsusb'

Oct 11 16:04:52 proxmox1 nut-driver@powerwalker[2924309]: and make sure you have an up-to-date version of NUT. If this does not help,

Oct 11 16:04:52 proxmox1 nut-driver@powerwalker[2924309]: try running the driver with at least 'subdriver', 'vendorid' and 'productid'

Oct 11 16:04:52 proxmox1 nut-driver@powerwalker[2924309]: options specified. Please refer to the man page for details about these options

Oct 11 16:04:52 proxmox1 nut-driver@powerwalker[2924309]: (man 8 nutdrv_qx).

Oct 11 16:04:52 proxmox1 nut-driver@powerwalker[2924308]: Driver failed to start (exit status=1)

Oct 11 16:04:52 proxmox1 corosync-qdevice[3192]: Connect timeout

Oct 11 16:04:55 proxmox1 corosync-qdevice[3192]: Can't connect to qnetd host. (-5986): Network address not available (in use?)

Oct 11 16:04:56 proxmox1 watchdog-mux[2312]: client watchdog expired - disable watchdog updates

Oct 11 16:04:57 proxmox1 nut-driver@powerwalker[2924360]: Network UPS Tools - Generic Q* USB/Serial driver 0.32 (2.8.0)

Oct 11 16:04:57 proxmox1 nut-driver@powerwalker[2924360]: USB communication driver (libusb 1.0) 0.43

Oct 11 16:04:57 proxmox1 nut-driver@powerwalker[2924360]: libusb1: Could not open any HID devices: insufficient permissions on everything

Oct 11 16:04:57 proxmox1 nut-driver@powerwalker[2924360]: No supported devices found. Please check your device availability with 'lsusb'

Oct 11 16:04:57 proxmox1 nut-driver@powerwalker[2924360]: and make sure you have an up-to-date version of NUT. If this does not help,

Oct 11 16:04:57 proxmox1 nut-driver@powerwalker[2924360]: try running the driver with at least 'subdriver', 'vendorid' and 'productid'

Oct 11 16:04:57 proxmox1 nut-driver@powerwalker[2924360]: options specified. Please refer to the man page for details about these options

Oct 11 16:04:57 proxmox1 nut-driver@powerwalker[2924360]: (man 8 nutdrv_qx).

Oct 11 16:04:57 proxmox1 nut-driver@powerwalker[2924308]: Driver failed to start (exit status=1)

Yes I do have some HA VMs and container running on my two nodes.
Can that cause this issue?

 

[Moayad]

Thank you for the logs!

Can that cause this issue?

The losing quorum can trigger the HA manager to initiate a node reboot, which is what seems to be happening, since HA requires stable network communication to ensure that resources are available.

If you have only one ring_x in your corosync config, please consider adding a second ring_x for redundancy [0]

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy

 

[showiproute]

Or I may disable HA for the VMs/containers, right?