VM processes stall when rebooting adjacent node

C

crlt

New Member
Jul 19, 2026
3
1
3
I am experiencing an issue where processes in virtual machines experience stalls (<10s) during the reboot of an adjacent node. So the HA VM hosted on pve1 will stall when I reboot pve3. It seems to impact sensitive things such as leader elections in Kubernetes processes. It also only appears to impact VMs where Ceph is the storage backend as migrating a VM to use local storage alleviates the issue.

Setup (verified cabling is correct and each bond is functional):
  • 3x PVE 9.2.4 Nodes w/ Ceph 19.2.4, total 6 interfaces (3 networks)
  • Interface 1&2: Active/Backup Bond for LAN (1Gbps)
  • Interface 3&4: Active/Backup Bond for Storage / Ceph Public (10Gbps)
  • Interface 5&6: Full Mesh (OVS RSTP) Cluster / Ceph Private (40Gbps)
  • Ceph pools are 3/2

I set the ceph flags (nodeep-scrub noout norebalance norecover noscrub) and the pve maintenance mode before a reboot, logs from that minute:
Code: 
Jul 19 12:20:32 pve3 corosync[1961]:   [CFG   ] Node 1 was shut down by sysadmin
Jul 19 12:20:32 pve3 pmxcfs[1730]: [dcdb] notice: members: 2/1808, 3/1730
Jul 19 12:20:32 pve3 pmxcfs[1730]: [dcdb] notice: starting data syncronisation
Jul 19 12:20:32 pve3 pmxcfs[1730]: [status] notice: members: 2/1808, 3/1730
Jul 19 12:20:32 pve3 pmxcfs[1730]: [status] notice: starting data syncronisation
Jul 19 12:20:32 pve3 pmxcfs[1730]: [dcdb] notice: received sync request (epoch 2/1808/00000004)
Jul 19 12:20:32 pve3 pmxcfs[1730]: [status] notice: received sync request (epoch 2/1808/00000004)
Jul 19 12:20:32 pve3 corosync[1961]:   [QUORUM] Sync members[2]: 2 3
Jul 19 12:20:32 pve3 corosync[1961]:   [QUORUM] Sync left[1]: 1
Jul 19 12:20:32 pve3 corosync[1961]:   [TOTEM ] A new membership (2.9d2) was formed. Members left: 1
Jul 19 12:20:32 pve3 corosync[1961]:   [QUORUM] Members[2]: 2 3
Jul 19 12:20:32 pve3 corosync[1961]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jul 19 12:20:32 pve3 pmxcfs[1730]: [dcdb] notice: received all states
Jul 19 12:20:32 pve3 pmxcfs[1730]: [dcdb] notice: leader is 2/1808
Jul 19 12:20:32 pve3 pmxcfs[1730]: [dcdb] notice: synced members: 2/1808, 3/1730
Jul 19 12:20:32 pve3 pmxcfs[1730]: [dcdb] notice: all data is up to date
Jul 19 12:20:32 pve3 pmxcfs[1730]: [status] notice: received all states
Jul 19 12:20:32 pve3 pmxcfs[1730]: [status] notice: all data is up to date
Jul 19 12:20:33 pve3 corosync[1961]:   [KNET  ] link: host: 1 link: 2 is down
Jul 19 12:20:33 pve3 corosync[1961]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 50)
Jul 19 12:20:33 pve3 corosync[1961]:   [KNET  ] link: host: 1 link: 0 is down
Jul 19 12:20:33 pve3 corosync[1961]:   [KNET  ] link: host: 1 link: 1 is down
Jul 19 12:20:33 pve3 corosync[1961]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 50)
Jul 19 12:20:33 pve3 corosync[1961]:   [KNET  ] host: host: 1 has no active links
Jul 19 12:20:33 pve3 corosync[1961]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 50)
Jul 19 12:20:33 pve3 corosync[1961]:   [KNET  ] host: host: 1 has no active links
Jul 19 12:20:34 pve3 kernel: libceph: osd9 (1)10.10.11.101:6827 socket closed (con state OPEN)
Jul 19 12:20:34 pve3 kernel: libceph: osd9 (1)10.10.11.101:6827 socket closed (con state V1_BANNER)
Jul 19 12:20:34 pve3 kernel: libceph: osd9 (1)10.10.11.101:6827 socket closed (con state V1_BANNER)
Jul 19 12:20:34 pve3 kernel: libceph: osd9 (1)10.10.11.101:6827 socket closed (con state V1_BANNER)
Jul 19 12:20:35 pve3 kernel: libceph: osd9 (1)10.10.11.101:6827 socket closed (con state V1_BANNER)
Jul 19 12:20:39 pve3 kernel: mlx4_en: enp1s0: Link Down
Jul 19 12:20:39 pve3 kernel: mlx4_core 0000:01:00.0 mlx4_0: Port: 1 Link ACTIVE
Jul 19 12:20:40 pve3 pvestatd[3713]: got timeout
Jul 19 12:20:42 pve3 pvestatd[3713]: status update time (6.711 seconds)
Jul 19 12:20:42 pve3 pveproxy[71329]: worker exit
Jul 19 12:20:42 pve3 pveproxy[4706]: worker 71329 finished
Jul 19 12:20:42 pve3 pveproxy[4706]: starting 1 worker(s)
Jul 19 12:20:42 pve3 pveproxy[4706]: worker 77417 started
Jul 19 12:20:42 pve3 kernel: libceph (036w7z3f-0b37-998c-9412-bf8b01c3db21 e18413): osd0 down
Jul 19 12:20:42 pve3 kernel: libceph (036w7z3f-0b37-998c-9412-bf8b01c3db21 e18413): osd3 down
Jul 19 12:20:42 pve3 kernel: libceph (036w7z3f-0b37-998c-9412-bf8b01c3db21 e18413): osd6 down
Jul 19 12:20:42 pve3 kernel: libceph (036w7z3f-0b37-998c-9412-bf8b01c3db21 e18413): osd9 down
Jul 19 12:20:42 pve3 ceph-mgr[1889]: DEBUG:notification_queue:processing queue: 7
Jul 19 12:20:42 pve3 ceph-mgr[1889]: DEBUG:notification_queue:processing queue: 11
Jul 19 12:20:42 pve3 ceph-mgr[1889]: DEBUG:notification_queue:processing queue: 38
Jul 19 12:20:43 pve3 ceph-mgr[1889]: DEBUG:notification_queue:processing queue: 1
Jul 19 12:20:48 pve3 ceph-mgr[1889]: DEBUG:notification_queue:processing queue: 1

Ceph Crush Map Tunables:
Code: 
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54

Network Interface for Full Mesh (cluster and ceph private network):
Code: 
auto enp1s0
iface enp1s0 inet manual
    ovs_type OVSPort
    ovs_bridge vmbr2
    ovs_mtu 9000
    ovs_options other_config:rstp-port-admin-edge=false other_config:rstp-port-mcheck=true other_config:rstp-path-cost=150 vlan_mode=native-untagged other_config:rstp-port-auto-edge=false other_config:rstp-enable=true

auto enp1s0d1
iface enp1s0d1 inet manual
    ovs_type OVSPort
    ovs_bridge vmbr2
    ovs_mtu 9000
    ovs_options other_config:rstp-port-auto-edge=false other_config:rstp-port-admin-edge=false other_config:rstp-path-cost=150 vlan_mode=native-untagged other_config:rstp-port-mcheck=true other_config:rstp-enable=true

auto vmbr2
iface vmbr2 inet static
    address 10.10.12.103/24
    ovs_type OVSBridge
    ovs_ports enp1s0 enp1s0d1
    ovs_mtu 9000
    up ovs-vsctl set Bridge ${IFACE} rstp_enable=true other_config:rstp-priority=32768 other_config:rstp-forward-delay=4 other_config:rstp-max-age=6
    post-up sleep 10
 
Last edited:
I have noticed very similar behavior when Ceph nodes are rebooted. I have noticed that reboot or poweroff command does not give enough time for OSDs to cleanly stop. In my case OSD require about 12 seconds to stop as they have about 10GB of RAM in use.

This situation can be easily avoided by manually stopping OSDs, either in the GUI or by using systemctl. After OSDs are stopped, reboot or poweroff command can be easily issued.

Next time try stopping OSDs first before giving reboot command.
 
  • Like
Reactions: leesteken and crlt
robertlukan said:
I have noticed very similar behavior when Ceph nodes are rebooted. I have noticed that reboot or poweroff command does not give enough time for OSDs to cleanly stop. In my case OSD require about 12 seconds to stop as they have about 10GB of RAM in use.

This situation can be easily avoided by manually stopping OSDs, either in the GUI or by using systemctl. After OSDs are stopped, reboot or poweroff command can be easily issued.

Next time try stopping OSDs first before giving reboot command.
Click to expand...

Thank you! I added stopping OSDs into the procedure and I did not experience the issue when I rebooted a node just now. I will continue doing this during the next set of updates/reboots.
 
  • Like
Reactions: robertlukan
I am actually surprised that Proxmox shutdown procedure does not handle this situation differently = gracefully, by introducing a delay/dependency into bringing down network interface only after OSD processes are stopped.

I have experienced this situation in a bad way recently, as one of the admins issued poweroff command on a Ceph node. This process worked very well a few months ago as OSDs were less busy(less RAM used), but this time the same command caused an outage.
 
Myself as well, when I originally searched this issue I couldn't find much so I assumed it was something on my end but if it seems to be more widespread maybe this thread will gather some attention from the maintainers?
 
You must log in or register to reply here.
Top Bottom
Back