freezing Linux VMs and sporadically connection issues regarding a node

Sebi-S

Sebi-S

Active Member
Proxmox Subscriber
Apr 19, 2022
62
10
28
Germany
Hello,

We have a problem affecting VMs on the 7-node cluster. Linux VMs freeze frequently and in a reproducible manner following migrations to other nodes.
The symptom is that the VM is shown as ‘running’ in Proxmox, but no longer responds within the VM itself; the services become inactive and the machine cannot be operated. External monitoring of the running services reports an error.
The problem can be resolved by performing a hard reset on the VM.
At the same time, node 7 sporadically experiences connection issues with the other six nodes. On the node’s GUI, the other nodes are shown as unreachable, whilst an active HTTPS connection to the node exists (meaning it is indeed online).

All nodes are running Proxmox version 9.2.10 and kernel version 7.0.14-11-pve; the guests are running Debian 11 with the linux-image-5.10.0-46-amd64 kernel and Ubuntu 24.04 Server with the 6.8.0-137-generic kernel.

Any ideas as to what might be causing this?
 
when you say the VM freezes, do you mean it isnt responding to network traffic or local (console) access?

if its console access, is it a specific VM or all VMs on the nodes?
If its network access, check your raw interfaces for faults. ifconfig reports these.
 
JohnnieKF said:
I would start by checking your corosync is in order on the problematic node:
corosync-cfgtool -s
journalctl -u corosync
Click to expand...


It looks like this:
# corosync-cfgtool -s
Local node ID 7, transport knet
LINK ID 0 udp
addr = 192.168.20.208
status:
nodeid: 1: connected
nodeid: 2: connected
nodeid: 3: connected
nodeid: 4: connected
nodeid: 5: connected
nodeid: 6: connected
nodeid: 7: localhost

Even more interesting is the journal:
at about 5:29 I've experienced a migration and freezing of a machine.
Aug 18 05:16:15 pve07 corosync[3141]: [TOTEM ] Retransmit List: 4d7a
Aug 18 05:16:47 pve07 corosync[3141]: [KNET ] link: host: 6 link: 0 is down
Aug 18 05:16:47 pve07 corosync[3141]: [KNET ] link: host: 4 link: 0 is down
Aug 18 05:16:47 pve07 corosync[3141]: [KNET ] link: host: 1 link: 0 is down
Aug 18 05:16:47 pve07 corosync[3141]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 1)
Aug 18 05:16:47 pve07 corosync[3141]: [KNET ] host: host: 6 has no active links
Aug 18 05:16:47 pve07 corosync[3141]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Aug 18 05:16:47 pve07 corosync[3141]: [KNET ] host: host: 4 has no active links
Aug 18 05:16:47 pve07 corosync[3141]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Aug 18 05:16:47 pve07 corosync[3141]: [KNET ] host: host: 1 has no active links
Aug 18 05:16:47 pve07 corosync[3141]: [KNET ] link: Resetting MTU for link 0 because host 1 joined
Aug 18 05:16:47 pve07 corosync[3141]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Aug 18 05:16:47 pve07 corosync[3141]: [KNET ] pmtud: Global data MTU changed to: 8837
Aug 18 05:16:48 pve07 corosync[3141]: [KNET ] link: Resetting MTU for link 0 because host 6 joined
Aug 18 05:16:48 pve07 corosync[3141]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 1)
Aug 18 05:16:48 pve07 corosync[3141]: [KNET ] pmtud: Global data MTU changed to: 8837
Aug 18 05:16:49 pve07 corosync[3141]: [KNET ] link: Resetting MTU for link 0 because host 4 joined
Aug 18 05:16:49 pve07 corosync[3141]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Aug 18 05:16:49 pve07 corosync[3141]: [KNET ] pmtud: Global data MTU changed to: 8837
Aug 18 05:17:16 pve07 corosync[3141]: [TOTEM ] Retransmit List: 5147
Aug 18 05:19:02 pve07 corosync[3141]: [TOTEM ] Retransmit List: 57fa
Aug 18 05:19:18 pve07 corosync[3141]: [TOTEM ] Retransmit List: 58cf
Aug 18 05:19:18 pve07 corosync[3141]: [TOTEM ] Retransmit List: 58d0
Aug 18 05:19:18 pve07 corosync[3141]: [TOTEM ] Retransmit List: 58d1
Aug 18 05:19:18 pve07 corosync[3141]: [TOTEM ] Retransmit List: 58d2
Aug 18 05:28:55 pve07 corosync[3141]: [KNET ] link: host: 4 link: 0 is down
Aug 18 05:28:55 pve07 corosync[3141]: [KNET ] link: host: 1 link: 0 is down
Aug 18 05:28:55 pve07 corosync[3141]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Aug 18 05:28:55 pve07 corosync[3141]: [KNET ] host: host: 4 has no active links
Aug 18 05:28:55 pve07 corosync[3141]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Aug 18 05:28:55 pve07 corosync[3141]: [KNET ] host: host: 1 has no active links
Aug 18 05:28:58 pve07 corosync[3141]: [KNET ] link: host: 6 link: 0 is down
Aug 18 05:28:58 pve07 corosync[3141]: [KNET ] link: host: 5 link: 0 is down
Aug 18 05:28:58 pve07 corosync[3141]: [KNET ] link: host: 3 link: 0 is down
Aug 18 05:28:58 pve07 corosync[3141]: [KNET ] link: host: 2 link: 0 is down
Aug 18 05:28:58 pve07 corosync[3141]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 1)
Aug 18 05:28:58 pve07 corosync[3141]: [KNET ] host: host: 6 has no active links
Aug 18 05:28:58 pve07 corosync[3141]: [KNET ] host: host: 5 (passive) best link: 0 (pri: 1)
Aug 18 05:28:58 pve07 corosync[3141]: [KNET ] host: host: 5 has no active links
Aug 18 05:28:58 pve07 corosync[3141]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Aug 18 05:28:58 pve07 corosync[3141]: [KNET ] host: host: 3 has no active links
Aug 18 05:28:58 pve07 corosync[3141]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Aug 18 05:28:58 pve07 corosync[3141]: [KNET ] host: host: 2 has no active links
Aug 18 05:29:00 pve07 corosync[3141]: [TOTEM ] Token has not been received in 4687 ms
Aug 18 05:29:01 pve07 corosync[3141]: [TOTEM ] A processor failed, forming new configuration: token timed out (6250ms), waiting 7500ms for consensus.
Aug 18 05:29:09 pve07 corosync[3141]: [QUORUM] Sync members[1]: 7
Aug 18 05:29:09 pve07 corosync[3141]: [QUORUM] Sync left[6]: 1 2 3 4 5 6
Aug 18 05:29:09 pve07 corosync[3141]: [TOTEM ] A new membership (7.1a28) was formed. Members left: 1 2 3 4 5 6
Aug 18 05:29:09 pve07 corosync[3141]: [TOTEM ] Failed to receive the leave message. failed: 1 2 3 4 5 6
Aug 18 05:29:09 pve07 corosync[3141]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Aug 18 05:29:09 pve07 corosync[3141]: [QUORUM] Members[1]: 7
Aug 18 05:29:09 pve07 corosync[3141]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 18 05:29:49 pve07 corosync[3141]: [KNET ] link: Resetting MTU for link 0 because host 4 joined
Aug 18 05:29:49 pve07 corosync[3141]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Aug 18 05:29:49 pve07 corosync[3141]: [KNET ] pmtud: Global data MTU changed to: 8837
Aug 18 05:29:49 pve07 corosync[3141]: [KNET ] link: Resetting MTU for link 0 because host 6 joined
Aug 18 05:29:49 pve07 corosync[3141]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 1)
Aug 18 05:29:49 pve07 corosync[3141]: [KNET ] pmtud: Global data MTU changed to: 8837
Aug 18 05:29:49 pve07 corosync[3141]: [KNET ] link: Resetting MTU for link 0 because host 1 joined
Aug 18 05:29:49 pve07 corosync[3141]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Aug 18 05:29:49 pve07 corosync[3141]: [KNET ] link: Resetting MTU for link 0 because host 3 joined
Aug 18 05:29:49 pve07 corosync[3141]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Aug 18 05:29:49 pve07 corosync[3141]: [QUORUM] Sync members[7]: 1 2 3 4 5 6 7
Aug 18 05:29:49 pve07 corosync[3141]: [QUORUM] Sync joined[6]: 1 2 3 4 5 6
Aug 18 05:29:49 pve07 corosync[3141]: [TOTEM ] A new membership (1.1a2c) was formed. Members joined: 1 2 3 4 5 6
Aug 18 05:29:49 pve07 corosync[3141]: [QUORUM] This node is within the primary component and will provide service.
Aug 18 05:29:49 pve07 corosync[3141]: [QUORUM] Members[7]: 1 2 3 4 5 6 7
Aug 18 05:29:49 pve07 corosync[3141]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 18 05:29:50 pve07 corosync[3141]: [KNET ] pmtud: Global data MTU changed to: 8837
Aug 18 05:29:50 pve07 corosync[3141]: [KNET ] rx: host: 2 link: 0 is up
Aug 18 05:29:50 pve07 corosync[3141]: [KNET ] link: Resetting MTU for link 0 because host 2 joined
Aug 18 05:29:50 pve07 corosync[3141]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Aug 18 05:29:50 pve07 corosync[3141]: [KNET ] rx: host: 5 link: 0 is up
Aug 18 05:29:50 pve07 corosync[3141]: [KNET ] link: Resetting MTU for link 0 because host 5 joined
Aug 18 05:29:50 pve07 corosync[3141]: [KNET ] host: host: 5 (passive) best link: 0 (pri: 1)
Aug 18 05:29:50 pve07 corosync[3141]: [KNET ] pmtud: Global data MTU changed to: 8837
 
alexskysilk said:
when you say the VM freezes, do you mean it isnt responding to network traffic or local (console) access?

if its console access, is it a specific VM or all VMs on the nodes?
If its network access, check your raw interfaces for faults. ifconfig reports these.
Click to expand...

"Freezing" in this context means that the machine is not responding to either network traffic or the console.

I haven’t checked the physical layer yet. The node is connected via a network card with two ports to two different switches (MLAG).

And no – it’s only one machine at a time. The VM administrator was able to predict this: “I’ve now started the VM on the node/migrated it to the node; the VM runs for a few minutes and then freezes...”

So yes, checking my switch ports and the switch configuration will also be another step for me.
 
Looks like a network issue on pve07, since it lost all 6 peers in three seconds.

It's also worth checking the other nodes corosync logs around that time to confirm they only lost connection to node 7.
 
Hi,
regarding the freezing VM, please share the output of pveversion -v, qm config ID with the numerical ID of the VM and qm status ID --verbose at the time it's frozen.

Is the VM maybe using the host CPU model and do your nodes have different physical CPUs? If yes, you need to switch to a non-host model instead for live-migration to work properly.
 
You must log in or register to reply here.
Top Bottom
Back