[SOLVED] Vm freeze during backup

[Maksimus]

The backup started according to the schedule every 2 hours starts, before that everything was fine for several months. Somewhere in the 75% of the backup, they found out that the vm was hanging, waited for the end of the backup, the vm didn’t hang down, it all worked.
Vm based on windows server 2019

()
INFO: starting new backup job: vzdump 101 --prune-backups 'keep-daily=30,keep-last=36' --storage PB022-DS4 --mailnotification failure --quiet 1 --notes-template '{{cluster}}, {{guestname}}, {{node}}, {{vmid}}' --mailto help@fruitcloud.net --mode snapshot
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2023-03-02 17:00:03
INFO: status = running
INFO: VM Name: BaltFresh
INFO: include disk 'virtio0' 'disk2:101/vm-101-disk-0.qcow2' 170G
INFO: include disk 'virtio1' 'disk2:101/vm-101-disk-1.qcow2' 120G
INFO: exclude disk 'virtio2' 'disk2:101/vm-101-disk-2.qcow2' (backup=no)
INFO: include disk 'virtio3' 'disk2:101/vm-101-disk-3.qcow2' 200G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/101/2023-03-02T14:00:03Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '53e4c9b1-c0ba-46b7-b716-1c6c95ca4732'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: OK (27.7 GiB of 170.0 GiB dirty)
INFO: virtio1: dirty-bitmap status: OK (14.0 GiB of 120.0 GiB dirty)
INFO: virtio3: dirty-bitmap status: OK (128.0 MiB of 200.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 41.8 GiB dirty of 490.0 GiB total
INFO:   2% (1.1 GiB of 41.8 GiB) in 3s, read: 373.3 MiB/s, write: 373.3 MiB/s
INFO:   3% (1.4 GiB of 41.8 GiB) in 37s, read: 9.9 MiB/s, write: 9.8 MiB/s
INFO:   5% (2.2 GiB of 41.8 GiB) in 40s, read: 264.0 MiB/s, write: 262.7 MiB/s
INFO:   7% (3.0 GiB of 41.8 GiB) in 43s, read: 281.3 MiB/s, write: 278.7 MiB/s
INFO:   8% (3.6 GiB of 41.8 GiB) in 57s, read: 40.0 MiB/s, write: 40.0 MiB/s
INFO:  10% (4.6 GiB of 41.8 GiB) in 1m, read: 345.3 MiB/s, write: 344.0 MiB/s
INFO:  11% (5.0 GiB of 41.8 GiB) in 1m 3s, read: 136.0 MiB/s, write: 136.0 MiB/s
INFO:  12% (5.3 GiB of 41.8 GiB) in 1m 36s, read: 8.6 MiB/s, write: 8.6 MiB/s
INFO:  14% (6.1 GiB of 41.8 GiB) in 1m 39s, read: 300.0 MiB/s, write: 297.3 MiB/s
INFO:  17% (7.2 GiB of 41.8 GiB) in 1m 42s, read: 354.7 MiB/s, write: 350.7 MiB/s
INFO:  20% (8.5 GiB of 41.8 GiB) in 1m 45s, read: 452.0 MiB/s, write: 452.0 MiB/s
INFO:  23% (9.8 GiB of 41.8 GiB) in 1m 48s, read: 440.0 MiB/s, write: 440.0 MiB/s
INFO:  25% (10.9 GiB of 41.8 GiB) in 1m 51s, read: 369.3 MiB/s, write: 369.3 MiB/s
INFO:  28% (11.8 GiB of 41.8 GiB) in 1m 54s, read: 333.3 MiB/s, write: 333.3 MiB/s
INFO:  30% (12.7 GiB of 41.8 GiB) in 1m 57s, read: 285.3 MiB/s, write: 285.3 MiB/s
INFO:  32% (13.6 GiB of 41.8 GiB) in 2m, read: 321.3 MiB/s, write: 321.3 MiB/s
INFO:  35% (14.9 GiB of 41.8 GiB) in 2m 3s, read: 438.7 MiB/s, write: 438.7 MiB/s
INFO:  37% (15.8 GiB of 41.8 GiB) in 2m 6s, read: 320.0 MiB/s, write: 320.0 MiB/s
INFO:  40% (17.1 GiB of 41.8 GiB) in 2m 9s, read: 424.0 MiB/s, write: 421.3 MiB/s
INFO:  42% (17.8 GiB of 41.8 GiB) in 2m 12s, read: 252.0 MiB/s, write: 248.0 MiB/s
INFO:  43% (18.1 GiB of 41.8 GiB) in 3m 55s, read: 2.4 MiB/s, write: 2.3 MiB/s
INFO:  44% (18.4 GiB of 41.8 GiB) in 4m 1s, read: 58.0 MiB/s, write: 56.0 MiB/s
INFO:  45% (19.0 GiB of 41.8 GiB) in 4m 7s, read: 104.0 MiB/s, write: 101.3 MiB/s
INFO:  46% (19.3 GiB of 41.8 GiB) in 4m 10s, read: 81.3 MiB/s, write: 80.0 MiB/s
INFO:  47% (19.7 GiB of 41.8 GiB) in 4m 16s, read: 70.0 MiB/s, write: 69.3 MiB/s
INFO:  48% (20.2 GiB of 41.8 GiB) in 4m 27s, read: 52.7 MiB/s, write: 51.6 MiB/s
INFO:  49% (20.7 GiB of 41.8 GiB) in 4m 30s, read: 149.3 MiB/s, write: 144.0 MiB/s
INFO:  50% (21.0 GiB of 41.8 GiB) in 4m 39s, read: 42.2 MiB/s, write: 41.8 MiB/s
INFO:  51% (21.4 GiB of 41.8 GiB) in 4m 42s, read: 113.3 MiB/s, write: 112.0 MiB/s
INFO:  52% (21.8 GiB of 41.8 GiB) in 4m 52s, read: 47.2 MiB/s, write: 46.8 MiB/s
INFO:  53% (22.2 GiB of 41.8 GiB) in 4m 57s, read: 77.6 MiB/s, write: 76.0 MiB/s
INFO:  54% (22.6 GiB of 41.8 GiB) in 5m 11s, read: 31.7 MiB/s, write: 30.9 MiB/s
INFO:  55% (23.0 GiB of 41.8 GiB) in 5m 18s, read: 58.3 MiB/s, write: 57.7 MiB/s
INFO:  56% (23.5 GiB of 41.8 GiB) in 5m 26s, read: 52.0 MiB/s, write: 49.5 MiB/s
INFO:  57% (23.9 GiB of 41.8 GiB) in 5m 40s, read: 30.6 MiB/s, write: 30.0 MiB/s
INFO:  58% (24.3 GiB of 41.8 GiB) in 6m 14s, read: 12.7 MiB/s, write: 12.2 MiB/s
INFO:  59% (24.7 GiB of 41.8 GiB) in 6m 29s, read: 28.8 MiB/s, write: 28.5 MiB/s
INFO:  60% (25.1 GiB of 41.8 GiB) in 6m 48s, read: 23.4 MiB/s, write: 23.2 MiB/s
INFO:  61% (25.6 GiB of 41.8 GiB) in 7m 15s, read: 17.6 MiB/s, write: 17.5 MiB/s
INFO:  62% (26.1 GiB of 41.8 GiB) in 7m 27s, read: 44.7 MiB/s, write: 43.3 MiB/s
INFO:  63% (26.7 GiB of 41.8 GiB) in 7m 30s, read: 180.0 MiB/s, write: 180.0 MiB/s
INFO:  64% (27.0 GiB of 41.8 GiB) in 7m 33s, read: 100.0 MiB/s, write: 98.7 MiB/s
INFO:  65% (27.3 GiB of 41.8 GiB) in 7m 37s, read: 84.0 MiB/s, write: 84.0 MiB/s
INFO:  66% (27.6 GiB of 41.8 GiB) in 7m 41s, read: 93.0 MiB/s, write: 93.0 MiB/s
INFO:  67% (28.2 GiB of 41.8 GiB) in 7m 44s, read: 186.7 MiB/s, write: 185.3 MiB/s
INFO:  68% (28.6 GiB of 41.8 GiB) in 7m 48s, read: 95.0 MiB/s, write: 94.0 MiB/s
INFO:  69% (29.1 GiB of 41.8 GiB) in 7m 51s, read: 180.0 MiB/s, write: 173.3 MiB/s
INFO:  70% (29.5 GiB of 41.8 GiB) in 7m 54s, read: 137.3 MiB/s, write: 134.7 MiB/s
INFO:  71% (29.8 GiB of 41.8 GiB) in 7m 59s, read: 65.6 MiB/s, write: 64.8 MiB/s
INFO:  72% (30.1 GiB of 41.8 GiB) in 8m 2s, read: 112.0 MiB/s, write: 110.7 MiB/s
INFO:  73% (30.6 GiB of 41.8 GiB) in 8m 6s, read: 120.0 MiB/s, write: 120.0 MiB/s
INFO:  74% (31.0 GiB of 41.8 GiB) in 8m 9s, read: 122.7 MiB/s, write: 120.0 MiB/s
INFO:  75% (31.5 GiB of 41.8 GiB) in 8m 12s, read: 170.7 MiB/s, write: 168.0 MiB/s
INFO:  76% (31.9 GiB of 41.8 GiB) in 8m 15s, read: 150.7 MiB/s, write: 148.0 MiB/s
INFO:  77% (32.2 GiB of 41.8 GiB) in 8m 18s, read: 102.7 MiB/s, write: 101.3 MiB/s
INFO:  78% (32.8 GiB of 41.8 GiB) in 8m 25s, read: 78.3 MiB/s, write: 77.1 MiB/s
INFO:  79% (33.2 GiB of 41.8 GiB) in 8m 28s, read: 149.3 MiB/s, write: 146.7 MiB/s
INFO:  80% (33.7 GiB of 41.8 GiB) in 8m 36s, read: 61.5 MiB/s, write: 61.0 MiB/s
INFO:  81% (34.0 GiB of 41.8 GiB) in 8m 40s, read: 81.0 MiB/s, write: 81.0 MiB/s
INFO:  82% (34.6 GiB of 41.8 GiB) in 8m 43s, read: 198.7 MiB/s, write: 197.3 MiB/s
INFO:  83% (35.0 GiB of 41.8 GiB) in 8m 46s, read: 136.0 MiB/s, write: 136.0 MiB/s
INFO:  84% (35.4 GiB of 41.8 GiB) in 8m 49s, read: 145.3 MiB/s, write: 141.3 MiB/s
INFO:  86% (36.2 GiB of 41.8 GiB) in 8m 52s, read: 293.3 MiB/s, write: 292.0 MiB/s
INFO:  87% (36.8 GiB of 41.8 GiB) in 8m 55s, read: 174.7 MiB/s, write: 174.7 MiB/s
INFO:  88% (37.0 GiB of 41.8 GiB) in 8m 58s, read: 82.7 MiB/s, write: 82.7 MiB/s
INFO:  90% (37.8 GiB of 41.8 GiB) in 9m 1s, read: 284.0 MiB/s, write: 284.0 MiB/s
INFO:  91% (38.1 GiB of 41.8 GiB) in 9m 4s, read: 92.0 MiB/s, write: 92.0 MiB/s
INFO:  92% (38.9 GiB of 41.8 GiB) in 9m 7s, read: 261.3 MiB/s, write: 260.0 MiB/s
INFO:  94% (39.3 GiB of 41.8 GiB) in 9m 10s, read: 160.0 MiB/s, write: 160.0 MiB/s
INFO:  95% (39.9 GiB of 41.8 GiB) in 9m 17s, read: 80.0 MiB/s, write: 77.1 MiB/s
INFO:  96% (40.4 GiB of 41.8 GiB) in 9m 21s, read: 127.0 MiB/s, write: 126.0 MiB/s
INFO:  97% (41.0 GiB of 41.8 GiB) in 9m 24s, read: 204.0 MiB/s, write: 202.7 MiB/s
INFO: 100% (41.8 GiB of 41.8 GiB) in 9m 27s, read: 290.7 MiB/s, write: 288.0 MiB/s
INFO: backup is sparse: 8.00 MiB (0%) total zero data
INFO: backup was done incrementally, reused 448.56 GiB (91%)
INFO: transferred 41.83 GiB in 813 seconds (52.7 MiB/s)
INFO: adding notes to backup
storing login ticket failed: $XDG_RUNTIME_DIR must be set
INFO: prune older backups with retention: keep-daily=30, keep-last=36
INFO: running 'proxmox-backup-client prune' for 'vm/101'
storing login ticket failed: $XDG_RUNTIME_DIR must be set
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 101 (00:13:39)
INFO: Backup finished at 2023-03-02 17:13:42
storing login ticket failed: $XDG_RUNTIME_DIR must be set
INFO: Backup job finished successfully
TASK OK

At 17:14:56, the reset button was pressed to restart the hanging vm

Linux Host602 5.15.35-3-pve #1 SMP PVE 5.15.35-6 (Fri, 17 Jun 2022 13:42:35 +0200) x86_64
proxmox-ve: 7.3-1 (running kernel: 5.15.35-3-pve)
pve-manager: 7.3-4 (running version: 7.3-4/d69b70d4)
pve-kernel-helper: 7.3-2
pve-kernel-5.15: 7.3-1
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.15.35-3-pve: 5.15.35-6
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.3
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-1
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.3.2-1
proxmox-backup-file-restore: 2.3.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.0-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-2
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-2
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.7-pve3

In amoy vm, there is only such an error record

- System


   - Provider


    [Name] Microsoft-Windows-Kernel-Power
    [Guid]{331c3b3a-2005-44c2-ac5e-77220c37d6b4}
 
    Event ID 41
 
    Version 6
 
    Level 1
 
    Task 63
 
    Opcode 0
 
    Keywords 0x8000400000000002
 
   -TimeCreated


    [SystemTime] 2023-03-02T14:15:55.552126100Z
 
    EventRecordID 1080654
 
    correlation
 
   - Execution


    [Process ID] 4
    [ThreadID] 8
 
    Channel System
 
    Computer WIN-2019
 
   - Security


    [UserID] S-1-5-18
 


- EventData


   BugcheckCode 0
   BugcheckParameter1 0x0
   BugcheckParameter2 0x0
   BugcheckParameter3 0x0
   BugcheckParameter4 0x0
   SleepInProgress 0
   PowerButtonTimestamp 0
   BootAppStatus 0
   checkpoint 0
   ConnectedStandbyInProgress false
   SystemSleepTransitionsToOn 0
   CsEntryScenarioInstanceId 0
   BugcheckInfoFromEFI false
   CheckpointStatus 0

 

[t.lamprecht]

Can you also please post the VM config, e.g., qm config 101

In general, it helps to reduce IO pressure for the VM by using IO-Threads on the disks (and virtio-scsi-single as IO controller), as that gives the QEMU main thread more time to process some VM events/work.

 

[Maksimus]

root@Host602:~# qm config 101
agent: 1
boot: order=virtio0;net0
cores: 8
description: PB022-DS4%0A%0A11/01/23 +20Gb disk1
hotplug: disk,network,usb,memory,cpu
machine: pc-i440fx-6.2
memory: 45056
meta: creation-qemu=6.2.0,ctime=1656499164
name: BaltFresh
net0: virtio=0E:E2:38:9C:3C:75,bridge=vmbr0,firewall=1
numa: 1
onboot: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=1fa95fcb-98fd-4581-95bd-30160b66aef5
sockets: 1
virtio0: disk2:101/vm-101-disk-0.qcow2,format=qcow2,size=170G
virtio1: disk2:101/vm-101-disk-1.qcow2,format=qcow2,size=120G
virtio2: disk2:101/vm-101-disk-2.qcow2,backup=0,format=qcow2,size=15G
virtio3: disk2:101/vm-101-disk-3.qcow2,format=qcow2,size=200G
vmgenid: 2504364e-cd19-44e1-8d37-3d09aa7f3906