Proxmox hanging

[Hyien]

Hi, I'm currently running pve 7.2-7.

After updating the following packages:
libpve-rs-perl:amd64 0.6.2 -> 0.7.1
pve-qemu-kvm:amd64 6.2.0-11 -> 7.0.0-2
libpve-storage-perl:amd64 7.2-7 -> 7.2-8
libproxmox-rs-perl:amd64 0.1.2 -> 0.2.0
qemu-server:amd64 7.2-3 -> 7.2-4,
pve-edk2-firmware:amd64 3.20210831-2 -> 3.20220526-1

I noticed my clusters hanging. Every proxmox command would hang and require
a hard reboot to fix. After rebooting, it runs for a bit, then hangs again.

 

[Docop2]

Have you read all code change for the beta package ? As updating is to try out and find errors and post log related to certain commit. If you only need a working server, just install direct iso.

 

[Hyien]

beta package? i'm also getting the update on the enterprise repo.

 

[dcsapak]

can you check the syslog (via /var/log/syslog or journalctl) to see if there are any errors or related messages there when it hangs?
'hanging' of commands usually happens when there is disk io that is blocking (so storage related)

 

[Hyien]

don't see any storage errors. i see a bunch of cluster related errors.
pmxcfs[1551]: [status] notice: cpg_send_message retry 100
pmxcfs[1551]: [status] notice: cpg_send_message retried 100 times
pmxcfs[1551]: [status] crit: cpg_send_message failed: 6
pve-firewall[1577]: firewall update time (10.016 seconds)
pmxcfs[1551]: [status] notice: cpg_send_message retry 10
corosync[1557]: [KNET ] rx: host: 1 link: 0 is up
corosync[1557]: [KNET ] rx: host: 2 link: 0 is up
corosync[1557]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
corosync[1557]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
pmxcfs[1551]: [status] notice: cpg_send_message retry 20

 

[dcsapak]

then check your network & firewall settings

 

[Hyien]

there aren't any fw restrictions.
i have nodes in different subnets in the same cluster.
would this cause any issue?

 

[dcsapak]

without more info & logs to your setup it's hard to say...

 

[Hyien]

is UDP 5404, 5405 all that's required for clustering?

 

[dcsapak]

for corosync yes, though if you use the webui/api the nodes must be able to connect via https (port 8006) to each other, for some things ssh is required and if you use spice, port 3128 is required

though that may not be the problem, throughput and latency can also be problem

i have nodes in different subnets in the same cluster.
would this cause any issue?

possibly, never tried that to be honest

 

[Hyien]

what are acceptable ping times for corosync to work properly?

 

[Neobin]

what are acceptable ping times for corosync to work properly?

https://pve.proxmox.com/wiki/Cluster_Manager#_cluster_network