VMs hung after backup

I have this issue as well on several of my VM's. Sometimes VM's freeze after backup, some VM's show increased load, without actual load on the VM, CPU wait timebecomes very large and the then VM freezes later or in the next backup. suspending and resuming the VM does nothing for me, only a shutdown and restart helps.
I am running ceph and have an ssd and hdd pool. I've only had the issue on systems with multiple disks that include disks form the hdd pool.

Load increase on VM's that did not freeze:
Screenshot 2024-01-21 at 11.05.09.png

Frozen VM:
qm config <ID>
Code: 
balloon: 65536
boot: order=scsi0;net0
cores: 16
cpu: host
memory: 131072
meta: creation-qemu=8.0.2,ctime=1699352342
name:
net0: virtio=76:F2:1E:FA:62:4D,bridge=vmbr991
net1: virtio=CA:FA:24:43:22:CD,bridge=vmbr1010
numa: 1
ostype: l26
scsi0: ceph_ssd:vm-203-disk-0,discard=on,iothread=1,size=2000G
scsi1: ceph_hdd:vm-203-disk-0,iothread=1,size=5000G
scsi2: ceph_hdd:vm-203-disk-1,iothread=1,size=5000G
scsihw: virtio-scsi-single
smbios1: uuid=8c4368d1-4965-4817-99a7-1edcccd72a7c
sockets: 2
vmgenid: 66ba396e-3b83-4ba3-b54b-9694d39eb2b3

qm status <ID> --verbose
Code: 
balloon: 137334095872
balloon_min: 68719476736
ballooninfo:
    actual: 137334095872
    free_mem: 116300972032
    last_update: 1705830548
    major_page_faults: 9044
    max_mem: 137438953472
    mem_swapped_in: 0
    mem_swapped_out: 0
    minor_page_faults: 854250602
    total_mem: 135090749440
blockstat:
    scsi0:
        account_failed: 1
        account_invalid: 1
        failed_flush_operations: 0
        failed_rd_operations: 0
        failed_unmap_operations: 0
        failed_wr_operations: 0
        failed_zone_append_operations: 0
        flush_operations: 3570094
        flush_total_time_ns: 485526215059
        idle_time_ns: 2167562739344
        invalid_flush_operations: 0
        invalid_rd_operations: 0
        invalid_unmap_operations: 0
        invalid_wr_operations: 0
        invalid_zone_append_operations: 0
        rd_bytes: 149557248
        rd_merged: 0
        rd_operations: 2694
        rd_total_time_ns: 4688579081
        timed_stats:
        unmap_bytes: 0
        unmap_merged: 0
        unmap_operations: 0
        unmap_total_time_ns: 0
        wr_bytes: 86216837120
        wr_highest_offset: 2046320816128
        wr_merged: 0
        wr_operations: 6991971
        wr_total_time_ns: 29959237124628
        zone_append_bytes: 0
        zone_append_merged: 0
        zone_append_operations: 0
        zone_append_total_time_ns: 0
    scsi1:
        account_failed: 1
        account_invalid: 1
        failed_flush_operations: 0
        failed_rd_operations: 0
        failed_unmap_operations: 0
        failed_wr_operations: 0
        failed_zone_append_operations: 0
        flush_operations: 0
        flush_total_time_ns: 0
        invalid_flush_operations: 0
        invalid_rd_operations: 0
        invalid_unmap_operations: 0
        invalid_wr_operations: 0
        invalid_zone_append_operations: 0
        rd_bytes: 0
        rd_merged: 0
        rd_operations: 0
        rd_total_time_ns: 0
        timed_stats:
        unmap_bytes: 0
        unmap_merged: 0
        unmap_operations: 0
        unmap_total_time_ns: 0
        wr_bytes: 0
        wr_highest_offset: 0
        wr_merged: 0
        wr_operations: 0
        wr_total_time_ns: 0
        zone_append_bytes: 0
        zone_append_merged: 0
        zone_append_operations: 0
        zone_append_total_time_ns: 0
    scsi2:
        account_failed: 1
        account_invalid: 1
        failed_flush_operations: 0
        failed_rd_operations: 0
        failed_unmap_operations: 0
        failed_wr_operations: 0
        failed_zone_append_operations: 0
        flush_operations: 0
        flush_total_time_ns: 0
        invalid_flush_operations: 0
        invalid_rd_operations: 0
        invalid_unmap_operations: 0
        invalid_wr_operations: 0
        invalid_zone_append_operations: 0
        rd_bytes: 0
        rd_merged: 0
        rd_operations: 0
        rd_total_time_ns: 0
        timed_stats:
        unmap_bytes: 0
        unmap_merged: 0
        unmap_operations: 0
        unmap_total_time_ns: 0
        wr_bytes: 0
        wr_highest_offset: 0
        wr_merged: 0
        wr_operations: 0
        wr_total_time_ns: 0
        zone_append_bytes: 0
        zone_append_merged: 0
        zone_append_operations: 0
        zone_append_total_time_ns: 0
cpus: 32
disk: 0
diskread: 149557248
diskwrite: 86216837120
freemem: 116300972032
maxdisk: 2147483648000
maxmem: 137438953472
mem: 18789777408
name:
netin: 228628941
netout: 90731944
nics:
    tap203i0:
        netin: 168823646
        netout: 50820211
    tap203i1:
        netin: 59805295
        netout: 39911733
pid: 289679
proxmox-support:
    backup-max-workers: 1
    pbs-dirty-bitmap: 1
    pbs-dirty-bitmap-migration: 1
    pbs-dirty-bitmap-savevm: 1
    pbs-library-version: 1.4.1 (UNKNOWN)
    pbs-masterkey: 1
    query-bitmap-info: 1
qmpstatus: running
running-machine: pc-i440fx-8.1+pve0
running-qemu: 8.1.2
shares: 1000
status: running
uptime: 519203
vmid: 203

pveversion -v
Code: 
proxmox-ve: 8.1.0 (running kernel: 6.5.11-7-pve)
pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5: 6.5.11-7
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
pve-kernel-5.4: 6.4-19
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-5.4.195-1-pve: 5.4.195-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph: 18.2.0-pve2
ceph-fuse: 18.2.0-pve2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx7
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.5
libqb0: 1.0.5-1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.2-1
proxmox-backup-file-restore: 3.1.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.3
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-2
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.1.5
pve-qemu-kvm: 8.1.2-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1

Backup log:
Code: 
NFO: Starting Backup of VM 203 (qemu)
INFO: Backup started at 2024-01-21 01:00:10
INFO: status = running
INFO: VM Name:
INFO: include disk 'scsi0' 'ceph_ssd:vm-203-disk-0' 2000G
INFO: include disk 'scsi1' 'ceph_hdd:vm-203-disk-0' 5000G
INFO: include disk 'scsi2' 'ceph_hdd:vm-203-disk-1' 5000G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/backup/dump/vzdump-qemu-203-2024_01_21-01_00_10.vma.zst'
INFO: started backup task '49ef32b2-b8d6-4cd2-b90d-a60ace21e60e'
INFO: resuming VM again
INFO:   0% (11.6 GiB of 11.7 TiB) in 3s, read: 3.9 GiB/s, write: 27.4 MiB/s
INFO:   1% (122.9 GiB of 11.7 TiB) in 27s, read: 4.6 GiB/s, write: 903.2 KiB/s
INFO:   2% (244.3 GiB of 11.7 TiB) in 52s, read: 4.9 GiB/s, write: 129.6 KiB/s
INFO:   3% (361.7 GiB of 11.7 TiB) in 1m 16s, read: 4.9 GiB/s, write: 296.0 KiB/s
INFO:   4% (483.5 GiB of 11.7 TiB) in 1m 41s, read: 4.9 GiB/s, write: 172.6 KiB/s
INFO:   5% (602.1 GiB of 11.7 TiB) in 2m 1s, read: 5.9 GiB/s, write: 165.0 KiB/s
INFO:   6% (726.5 GiB of 11.7 TiB) in 2m 17s, read: 7.8 GiB/s, write: 72.8 KiB/s
INFO:   7% (841.7 GiB of 11.7 TiB) in 2m 40s, read: 5.0 GiB/s, write: 201.0 KiB/s
INFO:   8% (967.5 GiB of 11.7 TiB) in 2m 59s, read: 6.6 GiB/s, write: 71.4 KiB/s
INFO:   9% (1.1 TiB of 11.7 TiB) in 3m 15s, read: 7.2 GiB/s, write: 64.2 KiB/s
INFO:  10% (1.2 TiB of 11.7 TiB) in 3m 41s, read: 4.6 GiB/s, write: 67.4 KiB/s
INFO:  11% (1.3 TiB of 11.7 TiB) in 4m 5s, read: 5.0 GiB/s, write: 71.2 KiB/s
INFO:  12% (1.4 TiB of 11.7 TiB) in 4m 25s, read: 6.0 GiB/s, write: 72.6 KiB/s
INFO:  13% (1.5 TiB of 11.7 TiB) in 4m 46s, read: 5.7 GiB/s, write: 87.0 KiB/s
INFO:  14% (1.6 TiB of 11.7 TiB) in 5m 11s, read: 4.9 GiB/s, write: 130.1 KiB/s
INFO:  15% (1.8 TiB of 11.7 TiB) in 5m 28s, read: 6.9 GiB/s, write: 312.5 KiB/s
INFO:  16% (1.9 TiB of 11.7 TiB) in 5m 51s, read: 5.4 GiB/s, write: 80.2 KiB/s
INFO:  17% (2.0 TiB of 11.7 TiB) in 6m 15s, read: 5.0 GiB/s, write: 173.0 KiB/s
INFO:  18% (2.1 TiB of 11.7 TiB) in 6m 35s, read: 6.2 GiB/s, write: 197.0 KiB/s
INFO:  19% (2.2 TiB of 11.7 TiB) in 6m 54s, read: 6.1 GiB/s, write: 89.7 KiB/s
INFO:  20% (2.3 TiB of 11.7 TiB) in 7m 18s, read: 5.1 GiB/s, write: 59.2 KiB/s
INFO:  21% (2.5 TiB of 11.7 TiB) in 7m 39s, read: 5.7 GiB/s, write: 433.3 KiB/s
INFO:  22% (2.6 TiB of 11.7 TiB) in 8m 5s, read: 4.5 GiB/s, write: 63.2 KiB/s
INFO:  23% (2.7 TiB of 11.7 TiB) in 8m 28s, read: 5.4 GiB/s, write: 77.6 KiB/s
INFO:  24% (2.8 TiB of 11.7 TiB) in 8m 48s, read: 6.0 GiB/s, write: 1.5 MiB/s
INFO:  25% (2.9 TiB of 11.7 TiB) in 9m 9s, read: 5.7 GiB/s, write: 109.0 KiB/s
INFO:  26% (3.1 TiB of 11.7 TiB) in 9m 31s, read: 5.7 GiB/s, write: 69.5 KiB/s
INFO:  27% (3.2 TiB of 11.7 TiB) in 9m 46s, read: 7.4 GiB/s, write: 112.5 KiB/s
INFO:  28% (3.3 TiB of 11.7 TiB) in 10m 7s, read: 5.8 GiB/s, write: 93.9 KiB/s
INFO:  29% (3.4 TiB of 11.7 TiB) in 10m 32s, read: 4.7 GiB/s, write: 63.2 KiB/s
INFO:  30% (3.5 TiB of 11.7 TiB) in 10m 51s, read: 6.3 GiB/s, write: 73.3 KiB/s
INFO:  31% (3.6 TiB of 11.7 TiB) in 11m 6s, read: 8.3 GiB/s, write: 72.0 KiB/s
INFO:  32% (3.8 TiB of 11.7 TiB) in 11m 27s, read: 5.5 GiB/s, write: 53.7 KiB/s
INFO:  33% (3.9 TiB of 11.7 TiB) in 11m 45s, read: 6.9 GiB/s, write: 74.0 KiB/s
INFO:  34% (4.0 TiB of 11.7 TiB) in 12m 3s, read: 6.5 GiB/s, write: 1.3 MiB/s
INFO:  35% (4.1 TiB of 11.7 TiB) in 12m 21s, read: 6.7 GiB/s, write: 70.4 KiB/s
INFO:  36% (4.2 TiB of 11.7 TiB) in 12m 46s, read: 4.8 GiB/s, write: 68.2 KiB/s
INFO:  37% (4.3 TiB of 11.7 TiB) in 13m 4s, read: 6.7 GiB/s, write: 70.4 KiB/s
INFO:  38% (4.5 TiB of 11.7 TiB) in 13m 28s, read: 4.8 GiB/s, write: 63.0 KiB/s
INFO:  39% (4.6 TiB of 11.7 TiB) in 13m 51s, read: 5.5 GiB/s, write: 74.3 KiB/s
INFO:  40% (4.7 TiB of 11.7 TiB) in 14m 9s, read: 6.6 GiB/s, write: 84.4 KiB/s
INFO:  41% (4.8 TiB of 11.7 TiB) in 14m 31s, read: 5.4 GiB/s, write: 66.0 KiB/s
INFO:  42% (4.9 TiB of 11.7 TiB) in 14m 52s, read: 5.5 GiB/s, write: 1.7 MiB/s
INFO:  43% (5.0 TiB of 11.7 TiB) in 15m 19s, read: 4.5 GiB/s, write: 349.9 KiB/s
INFO:  44% (5.2 TiB of 11.7 TiB) in 15m 45s, read: 4.7 GiB/s, write: 164.8 KiB/s
INFO:  45% (5.3 TiB of 11.7 TiB) in 16m 11s, read: 4.6 GiB/s, write: 258.3 KiB/s
INFO:  46% (5.4 TiB of 11.7 TiB) in 16m 38s, read: 4.5 GiB/s, write: 68.3 KiB/s
INFO:  47% (5.5 TiB of 11.7 TiB) in 17m 4s, read: 4.6 GiB/s, write: 70.6 KiB/s
INFO:  48% (5.6 TiB of 11.7 TiB) in 17m 29s, read: 4.8 GiB/s, write: 60.8 KiB/s
INFO:  49% (5.7 TiB of 11.7 TiB) in 17m 56s, read: 4.5 GiB/s, write: 158.5 KiB/s
INFO:  50% (5.9 TiB of 11.7 TiB) in 18m 22s, read: 4.6 GiB/s, write: 58.5 KiB/s
INFO:  51% (6.0 TiB of 11.7 TiB) in 18m 47s, read: 4.8 GiB/s, write: 80.0 KiB/s
INFO:  52% (6.1 TiB of 11.7 TiB) in 19m 13s, read: 4.6 GiB/s, write: 68.3 KiB/s
INFO:  53% (6.2 TiB of 11.7 TiB) in 19m 38s, read: 4.9 GiB/s, write: 58.4 KiB/s
INFO:  54% (6.3 TiB of 11.7 TiB) in 20m 4s, read: 4.6 GiB/s, write: 87.8 KiB/s
INFO:  55% (6.4 TiB of 11.7 TiB) in 20m 29s, read: 4.7 GiB/s, write: 78.7 KiB/s
INFO:  56% (6.6 TiB of 11.7 TiB) in 20m 48s, read: 6.3 GiB/s, write: 52.8 KiB/s
INFO:  57% (6.7 TiB of 11.7 TiB) in 21m 12s, read: 5.0 GiB/s, write: 71.5 KiB/s
INFO:  58% (6.8 TiB of 11.7 TiB) in 21m 33s, read: 5.9 GiB/s, write: 206.3 KiB/s
INFO:  59% (6.9 TiB of 11.7 TiB) in 21m 51s, read: 6.5 GiB/s, write: 55.6 KiB/s
INFO:  60% (7.0 TiB of 11.7 TiB) in 22m 19s, read: 4.3 GiB/s, write: 162.0 KiB/s
INFO:  61% (7.1 TiB of 11.7 TiB) in 22m 45s, read: 4.6 GiB/s, write: 65.8 KiB/s
INFO:  62% (7.3 TiB of 11.7 TiB) in 23m 5s, read: 6.3 GiB/s, write: 63.2 KiB/s
INFO:  63% (7.4 TiB of 11.7 TiB) in 23m 26s, read: 5.5 GiB/s, write: 549.3 KiB/s
INFO:  64% (7.5 TiB of 11.7 TiB) in 23m 48s, read: 5.5 GiB/s, write: 80.7 KiB/s
INFO:  65% (7.6 TiB of 11.7 TiB) in 24m 10s, read: 5.6 GiB/s, write: 108.7 KiB/s
INFO:  66% (7.7 TiB of 11.7 TiB) in 24m 30s, read: 5.8 GiB/s, write: 62.8 KiB/s
INFO:  67% (7.9 TiB of 11.7 TiB) in 24m 47s, read: 7.3 GiB/s, write: 93.4 KiB/s
INFO:  68% (8.0 TiB of 11.7 TiB) in 25m 3s, read: 7.5 GiB/s, write: 71.2 KiB/s
INFO:  69% (8.1 TiB of 11.7 TiB) in 25m 19s, read: 7.5 GiB/s, write: 91.0 KiB/s
INFO:  70% (8.2 TiB of 11.7 TiB) in 25m 41s, read: 5.3 GiB/s, write: 65.6 KiB/s
INFO:  71% (8.3 TiB of 11.7 TiB) in 26m 5s, read: 5.2 GiB/s, write: 69.2 KiB/s
INFO:  72% (8.4 TiB of 11.7 TiB) in 26m 24s, read: 6.0 GiB/s, write: 59.2 KiB/s
INFO:  73% (8.6 TiB of 11.7 TiB) in 26m 44s, read: 6.2 GiB/s, write: 70.0 KiB/s
INFO:  74% (8.7 TiB of 11.7 TiB) in 27m, read: 7.3 GiB/s, write: 74.5 KiB/s
INFO:  75% (8.8 TiB of 11.7 TiB) in 27m 16s, read: 7.8 GiB/s, write: 67.5 KiB/s
INFO:  76% (8.9 TiB of 11.7 TiB) in 27m 36s, read: 5.8 GiB/s, write: 67.6 KiB/s
INFO:  77% (9.0 TiB of 11.7 TiB) in 27m 59s, read: 5.3 GiB/s, write: 60.5 KiB/s
INFO:  78% (9.1 TiB of 11.7 TiB) in 28m 16s, read: 7.2 GiB/s, write: 91.5 KiB/s
INFO:  79% (9.3 TiB of 11.7 TiB) in 28m 34s, read: 6.7 GiB/s, write: 70.4 KiB/s
INFO:  80% (9.4 TiB of 11.7 TiB) in 28m 50s, read: 7.4 GiB/s, write: 66.8 KiB/s
INFO:  81% (9.5 TiB of 11.7 TiB) in 29m 7s, read: 7.1 GiB/s, write: 66.8 KiB/s
INFO:  82% (9.6 TiB of 11.7 TiB) in 29m 30s, read: 5.1 GiB/s, write: 68.7 KiB/s
INFO:  83% (9.7 TiB of 11.7 TiB) in 29m 54s, read: 5.1 GiB/s, write: 82.3 KiB/s
INFO:  84% (9.8 TiB of 11.7 TiB) in 31m 10s, read: 1.6 GiB/s, write: 102.3 MiB/s
INFO:  85% (10.0 TiB of 11.7 TiB) in 31m 27s, read: 6.8 GiB/s, write: 125.9 KiB/s
INFO:  86% (10.1 TiB of 11.7 TiB) in 31m 45s, read: 6.6 GiB/s, write: 2.7 MiB/s
INFO:  87% (10.2 TiB of 11.7 TiB) in 32m 3s, read: 6.8 GiB/s, write: 128.4 KiB/s
INFO:  88% (10.3 TiB of 11.7 TiB) in 32m 19s, read: 7.4 GiB/s, write: 67.0 KiB/s
INFO:  89% (10.4 TiB of 11.7 TiB) in 32m 34s, read: 7.8 GiB/s, write: 242.7 KiB/s
INFO:  90% (10.6 TiB of 11.7 TiB) in 32m 52s, read: 7.1 GiB/s, write: 61.1 KiB/s
INFO:  91% (10.7 TiB of 11.7 TiB) in 33m 6s, read: 8.1 GiB/s, write: 138.3 KiB/s
INFO:  92% (10.8 TiB of 11.7 TiB) in 33m 25s, read: 6.2 GiB/s, write: 48.4 MiB/s
INFO:  93% (10.9 TiB of 11.7 TiB) in 33m 44s, read: 6.6 GiB/s, write: 16.4 KiB/s
INFO:  94% (11.0 TiB of 11.7 TiB) in 34m 2s, read: 6.5 GiB/s, write: 13.8 KiB/s
INFO:  95% (11.1 TiB of 11.7 TiB) in 34m 19s, read: 7.1 GiB/s, write: 17.4 KiB/s
INFO:  96% (11.3 TiB of 11.7 TiB) in 34m 40s, read: 5.9 GiB/s, write: 17.9 KiB/s
INFO:  97% (11.4 TiB of 11.7 TiB) in 35m, read: 5.7 GiB/s, write: 42.4 MiB/s
INFO:  98% (11.5 TiB of 11.7 TiB) in 35m 16s, read: 7.6 GiB/s, write: 25.8 KiB/s
INFO:  99% (11.6 TiB of 11.7 TiB) in 35m 35s, read: 6.4 GiB/s, write: 2.5 MiB/s
INFO: 100% (11.7 TiB of 11.7 TiB) in 37m 4s, read: 1.3 GiB/s, write: 112.3 MiB/s
INFO: backup is sparse: 11.70 TiB (99%) total zero data
INFO: transferred 11.72 TiB in 2224 seconds (5.4 GiB/s)
INFO: archive file size: 7.22GB
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=5
INFO: removing backup 'backup:backup/vzdump-qemu-203-2023_12_17-01_26_01.vma.zst'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 203 (00:37:06)
INFO: Backup finished at 2024-01-21 01:37:16
 

Attachments

  • Screenshot 2024-01-21 at 10.38.07.png
    Screenshot 2024-01-21 at 10.38.07.png
    30.4 KB · Views: 3
Last edited:
I tried a few more things, the results may be of interest:

1) I had a VM that was showing higher load and CPU wait after backup. I had recently migrated some of its disks from ceph HDD to ceph SSD. The HDD were still attached but unused. I detached the HDD, and immediately load and wait time want back to normal. This lead my to believe that the issue had something to do with the hdd pool.

2) I removed iothreads=1 only from disks in the hdd pool, kept it for the SSD's. Unfortunately this did not help. I still had issues, and actually it seems to be starting to affect more hosts.
Before a host freezes, you can see threads getting blocked in syslog:
Code: 
Jan 21 14:42:40 portal kernel: [ 9305.892018] RSP: 002b:00007ffda668ca48 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Jan 21 14:42:40 portal kernel: [ 9305.892021] RAX: ffffffffffffffda RBX: 000000000000006b RCX: 00007f742d110b4e
Jan 21 14:42:40 portal kernel: [ 9305.892022] RDX: 00007ffda668cab0 RSI: 00007ffda668cbd0 RDI: 00000000ffffff9c
Jan 21 14:42:40 portal kernel: [ 9305.892023] RBP: 00007ffda668cb80 R08: 0000000000000000 R09: 0000000000000002
Jan 21 14:42:40 portal kernel: [ 9305.892025] R10: 0000000000000100 R11: 0000000000000246 R12: 00007ffda668cbd0
Jan 21 14:42:40 portal kernel: [ 9305.892026] R13: 0000000000000002 R14: 0000000065ad1d9e R15: 0000000000000056
Jan 21 14:44:40 portal kernel: [ 9426.722867] INFO: task apache2:9169 blocked for more than 120 seconds.
Jan 21 14:44:40 portal kernel: [ 9426.722873]       Not tainted 4.19.0-25-amd64 #1 Debian 4.19.289-2
Jan 21 14:44:40 portal kernel: [ 9426.722874] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 21 14:44:40 portal kernel: [ 9426.722877] apache2         D    0  9169   9166 0x00000320
Jan 21 14:44:40 portal kernel: [ 9426.722883] Call Trace:
Jan 21 14:44:40 portal kernel: [ 9426.722898]  __schedule+0x29f/0x840
Jan 21 14:44:40 portal kernel: [ 9426.722906]  ? bit_wait_timeout+0x90/0x90
Jan 21 14:44:40 portal kernel: [ 9426.722908]  schedule+0x28/0x80
Jan 21 14:44:40 portal kernel: [ 9426.722911]  io_schedule+0x12/0x40
Jan 21 14:44:40 portal kernel: [ 9426.722914]  bit_wait_io+0xd/0x50
Jan 21 14:44:40 portal kernel: [ 9426.722917]  __wait_on_bit+0x73/0x90
Jan 21 14:44:40 portal kernel: [ 9426.722921]  out_of_line_wait_on_bit+0x91/0xb0
Jan 21 14:44:40 portal kernel: [ 9426.722930]  ? init_wait_var_entry+0x40/0x40
Jan 21 14:44:40 portal kernel: [ 9426.722987]  __ext4_get_inode_loc+0x3a4/0x430 [ext4]
Jan 21 14:44:40 portal kernel: [ 9426.723011]  __ext4_iget+0x113/0xd70 [ext4]
Jan 21 14:44:40 portal kernel: [ 9426.723036]  ext4_lookup+0x12a/0x220 [ext4]
Jan 21 14:44:40 portal kernel: [ 9426.723042]  __lookup_slow+0x97/0x150
Jan 21 14:44:40 portal kernel: [ 9426.723046]  lookup_slow+0x35/0x50
Jan 21 14:44:40 portal kernel: [ 9426.723049]  walk_component+0x1bf/0x330
Jan 21 14:44:40 portal kernel: [ 9426.723052]  link_path_walk.part.44+0x2b9/0x520
Jan 21 14:44:40 portal kernel: [ 9426.723056]  path_lookupat.isra.48+0x93/0x220
Jan 21 14:44:40 portal kernel: [ 9426.723060]  ? kvm_sched_clock_read+0xd/0x20
Jan 21 14:44:40 portal kernel: [ 9426.723065]  ? ___bpf_prog_run+0x266/0xf00
Jan 21 14:44:40 portal kernel: [ 9426.723068]  filename_lookup.part.62+0xa0/0x170
Jan 21 14:44:40 portal kernel: [ 9426.723073]  ? __check_object_size+0x162/0x180
Jan 21 14:44:40 portal kernel: [ 9426.723079]  ? strncpy_from_user+0x47/0x1a0
Jan 21 14:44:40 portal kernel: [ 9426.723084]  vfs_statx+0x73/0xe0
Jan 21 14:44:40 portal kernel: [ 9426.723088]  __do_sys_newfstatat+0x31/0x70
Jan 21 14:44:40 portal kernel: [ 9426.723094]  ? syscall_trace_enter+0x192/0x2b0
Jan 21 14:44:40 portal kernel: [ 9426.723098]  do_syscall_64+0x53/0x110
Jan 21 14:44:40 portal kernel: [ 9426.723103]  entry_SYSCALL_64_after_hwframe+0x5c/0xc1
Jan 21 14:44:40 portal kernel: [ 9426.723108] RIP: 0033:0x7f742d110b4e
Jan 21 14:44:40 portal kernel: [ 9426.723119] Code: Bad RIP value.
Jan 21 14:44:40 portal kernel: [ 9426.723121] RSP: 002b:00007ffda668cb68 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Jan 21 14:44:40 portal kernel: [ 9426.723124] RAX: ffffffffffffffda RBX: 0000000000000088 RCX: 00007f742d110b4e
Jan 21 14:44:40 portal kernel: [ 9426.723126] RDX: 00007ffda668cbd0 RSI: 00007ffda668ccf0 RDI: 00000000ffffff9c
Jan 21 14:44:40 portal kernel: [ 9426.723128] RBP: 00007ffda668cca0 R08: 0000000000000000 R09: 0000000000000002
Jan 21 14:44:40 portal kernel: [ 9426.723129] R10: 0000000000000100 R11: 0000000000000246 R12: 00007ffda668ccf0
Jan 21 14:44:40 portal kernel: [ 9426.723131] R13: 0000000000000002 R14: 0000000065ad1d93 R15: 000000000000007f

3) During one of hte backups I also fund a new issue, where fs-freeze command from the qemu-agent locked up the vm:
Code: 
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2024-01-21 12:16:49
INFO: status = running
INFO: VM Name: dns1
INFO: include disk 'scsi0' 'ceph_ssd:vm-101-disk-0' 80G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/backup/dump/vzdump-qemu-101-2024_01_21-12_16_49.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
closing with read buffer at /usr/share/perl5/IO/Multiplex.pm line 927.
ERROR: interrupted by signal
INFO: issuing guest-agent 'fs-thaw' command
Could this have the same root cause?
This was reported in several other threads, but there does not seem to be a solution except for disabling FS-Freeze, which is not ideal for backup consistency.
https://forum.proxmox.com/threads/backup-crashes-vm-guest-fsfreeze-freeze-failed-got-timeout.104694/
https://forum.proxmox.com/threads/disable-fs-freeze-on-snapshot-backups.122533/

Note, we also have a cluster that leverage iscsi storage instead of ceph, and there I have not seen any issues. Could this be ceph related?

I will try to turn off all iothreads on my VM's and disable FS freeze and see if I can get through a backup without any hanging VM's.
Is this the recommended workaround for now or should just disabling iothreads be sufficient?
 
Last edited:
@bbn: we also have hdd+ssd pool in ceph, we do have all VM disks on ssd pool. the issue still exists.

As the freezing occurs immediately after creating a backup (see my thread), I like to think that this is caused by a bug rather than slow IO.

There does not seem any recent activity on the mailing list @fiona linked earlier. I hope there is still some activity in regards of finding a solution
 
@mnih: same here. we started having the issues with SSD pools as well. For now I have removed iothreads from all VM's and disabled fs-freeze in qemu-agent. For now, all VM's have survived backup. It would be good if the proxmox team could confirm that this is the recommended workaround until this issue gets fixed. Should both iothreads and fs-freeze be disabled? Or is the issue with fs-freeze caused by the iothreads issue?
Also @fiona mentioned that qm suspend && qm resume should solve the issue, if not there is another issue. It does nothing for me, and it seems to also not help in many other cases, so are we dealing with a single issue or are there multiple issues?
 
Just want to add a +1 here since I'm also affected. We don't use the proxmox backup feature but we do use snapshots. We have at least 1 VM where we can trigger this with a snapshot about 75% of the time consistently. It's a mostly idle VM. Interestingly when running an FIO test while making the snapshot to generate some load the issue could no longer be reproduced.

We use mdadm raid10 on NVME drives. lvmthin storage type.
Disabling iothreads does resolve the issue, but the flag requires a VM reboot to change. We'd rather not reboot hundreds of VMs if we can help it.
I'm not sure when the issue was introduced but it is a bit worrying this also affects the enterprise repo. Perhaps it's possible to rollback/revert to a version that does not have this bug for the time being?

On the forum we see more and more threads/issues popping up regarding this issue. Hoping a solution is close as currently we're worried when using the snapshot feature as it can now randomly make the VMs we snapshot unusable.
 
  • Like
Reactions: UdoB, nielsh and mnih
Thank you for that link @fiona
If I understand correctly, the proposed solution is to 're-enable notifications and kick the virt queue after draining'.
Would this not basically be what the qm suspend and resume workaround you posted does as well?
For a few of us on this thread this does not work. Would this fix help us?
 
Hi,
phs said:
any news? our clusters is plagued with VM freezes
Click to expand...
I literally posted the news just yesterday. If you are affected, try disabling iothread on the guests as a workaround until the fix is available.

bbn said:
Thank you for that link @fiona
If I understand correctly, the proposed solution is to 're-enable notifications and kick the virt queue after draining'.
Would this not basically be what the qm suspend and resume workaround you posted does as well?
For a few of us on this thread this does not work. Would this fix help us?
Click to expand...
It's difficult to tell, but I'd not be surprised if having IO stuck for a long time in the guest might not cause other issues too. You'll have to see if you are still affected with a fixed version and we can investigate further if you are, but I'd guess the fix will help in many cases.
 
fiona said:
It's difficult to tell, but I'd not be surprised if having IO stuck for a long time in the guest might not cause other issues too. You'll have to see if you are still affected with a fixed version and we can investigate further if you are, but I'd guess the fix will help in many cases.
Click to expand...
I feel this is true. We had a guest locking up every night from backups. If I could get to it quickly and suspend/resume, it was OK. If it was many hours later, suspend/resume would free up the VM but it was just so messed up from no disk access for hours, it was unstable and had to be rebooted.

I see the patch was "posted for discussion", can you tell us what that means in terms of timeline to a patch/update? days, weeks, months? Thanks.
 
arubenstein said:
I see the patch was "posted for discussion", can you tell us what that means in terms of timeline to a patch/update? days, weeks, months? Thanks.
Click to expand...
That depends on how quickly other developers will review it, so I really cannot tell.
 
Hello Fiona, there is an update for pve-qemu-kvm:

Code: 
pve-qemu-kvm (8.1.5-2) bookworm; urgency=medium

  * work around for a situation where guest IO might get stuck, if the VM is
    configure  with iothread and VirtIO block/SCSI

 -- Proxmox Support Team <support@proxmox.com>  Fri, 02 Feb 2024 19:41:27 +0100

Is this the update we were waiting for?
 
pigpen said:
Hello Fiona, there is an update for pve-qemu-kvm:

Code: 
pve-qemu-kvm (8.1.5-2) bookworm; urgency=medium

  * work around for a situation where guest IO might get stuck, if the VM is
    configure  with iothread and VirtIO block/SCSI

 -- Proxmox Support Team <support@proxmox.com>  Fri, 02 Feb 2024 19:41:27 +0100

Is this the update we were waiting for?
Click to expand...
Yes, just wanted to post it here too :) It's available on the no-subscription repository now and will be made available on the enterprise repository in a few days if no other issues pop up.
 
  • Like
Reactions: fbett, phs, nielsh and 3 others
fiona said:
Yes, just wanted to post it here too :) It's available on the no-subscription repository now and will be made available on the enterprise repository in a few days if no other issues pop up.
Click to expand...
Hello Fiona, did any issues pop up, yet? If not, I'll install the update today!
 
Hi,
Max2048 said:
Hey @fiona quick question. Can this patch also be pushed to PVE 7 branch? The initial issue (not the high CPU usage) with the stuck VMs at high I/O usage also exists in PVE 7 which is still supported according to the Support Lifecycle. This would be greatly appreciated!!!
Click to expand...
the patch you mention addresses a different issue, which was introduced in QEMU 8.1 (see the commits mentioned in the patch) and is not yet present in the QEMU version 7.2 used by Proxmox VE 7.

Issue addressed by the patch: VirtIO host notifiers potentially becoming inactive after a drain (which is used during backup/snapshot/etc. to ensure there is no IO while QEMU changes its internal block layer) when iothread is used.

Issue in the thread you linked: guest write times out, because backup target is too slow and when it later completes, the VirtIO-win guest driver doesn't gracefully handle it. Not related to iothread.
 
  • Like
Reactions: Max2048
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back