Another problem with backup

[slot]

Hello,

I have problem with backup VM. I got error:

start failed: command '/usr/bin/kvm -id 100 -name Win11 -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=4d5ab3be-4ccc-4320-b065-98546e3676eb' -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,size=540672,file=/dev/zvol/rpool/data/vm-100-disk-0' -smp '16,sockets=1,cores=16,maxcpus=16' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/100.vnc,password=on' -no-hpet -cpu 'kvm64,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep' -m 65536 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device 'vmgenid,guid=4d36a335-7702-4346-84ec-aa6484476cf0' -device 'nec-usb-xhci,id=xhci,bus=pci.1,addr=0x1b' -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'usb-host,bus=xhci.0,vendorid=0x2357,productid=0x0601,id=usb0' -chardev 'socket,id=tpmchar,path=/var/run/qemu-server/100.swtpm' -tpmdev 'emulator,id=tpmdev,chardev=tpmchar' -device 'tpm-tis,tpmdev=tpmdev' -device 'VGA,id=vga,bus=pcie.0,addr=0x1' -chardev 'socket,path=/var/run/qemu-server/100.qga,server=on,wait=off,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:496f3084b2d4' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/zvol/rpool/data/vm-100-disk-1,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,rotation_rate=1,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=B6:7A:6C:6E:5C:ED,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=101' -rtc 'driftfix=slew,base=localtime' -machine 'type=pc-q35-6.1+pve0' -global 'kvm-pit.lost_tick_policy=discard' -S' failed: got timeout

Backups are making everyday at 1:00. Error show random.

Server supermicro, 128GB RAM, 2 disks SSD Samsung PM883 1.92TB (MZ7LH1T9HMLT-00005) with ZFS RAID 1 (rpool) and 2 disks HDD 10TB also ZFS RAID 1 (dane). Server age is about 2 weeks. I checked all disks with short and extended SMART.

VM I got Windows 11 with 64GB ram, 1TB disk on SSD pool.

Backups are making on pool dane (HDD disks).

I got few supermicro servers with the same hardware on Proxmox 6 and I have no issues (they are working about 1 year without any problems with backup).

I wrote another thread with backup problems on 7.x

https://forum.proxmox.com/threads/backup-failed-timeout-waiting-on-systemd.107057/

Maybe something is wrong with version 7.x?

pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-7 (running version: 7.1-7/df5740ad)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3

qm config 100
agent: 1
bios: ovmf
boot: order=scsi0;net0
cores: 16
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
machine: pc-q35-6.1
memory: 65536
meta: creation-qemu=6.1.0,ctime=1649253483
name: Win11
net0: virtio=B6:7A:6C:6E:5C:ED,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
scsi0: local-zfs:vm-100-disk-1,discard=on,size=1000G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=4d5ab3be-4ccc-4320-b065-98546e3676eb
sockets: 1

()
INFO: starting new backup job: vzdump 100 --quiet 1 --compress zstd --prune-backups 'keep-last=7' --mailnotification always --mode stop --node proxmox --storage dane
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2022-04-21 01:00:04
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: Win11
INFO: include disk 'scsi0' 'local-zfs:vm-100-disk-1' 1000G
INFO: include disk 'efidisk0' 'local-zfs:vm-100-disk-0' 1M
INFO: include disk 'tpmstate0' 'local-zfs:vm-100-disk-2' 4M
INFO: stopping virtual guest
INFO: creating vzdump archive '/dane/dump/vzdump-qemu-100-2022_04_21-01_00_04.vma.zst'
INFO: starting kvm to execute backup task
swtpm_setup: Not overwriting existing state file.
INFO: guest is online again after 81 seconds
ERROR: Backup of VM 100 failed - start failed: command '/usr/bin/kvm -id 100 -name Win11 -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=4d5ab3be-4ccc-4320-b065-98546e3676eb' -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,size=540672,file=/dev/zvol/rpool/data/vm-100-disk-0' -smp '16,sockets=1,cores=16,maxcpus=16' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/100.vnc,password=on' -no-hpet -cpu 'kvm64,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep' -m 65536 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device 'vmgenid,guid=4d36a335-7702-4346-84ec-aa6484476cf0' -device 'nec-usb-xhci,id=xhci,bus=pci.1,addr=0x1b' -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'usb-host,bus=xhci.0,vendorid=0x2357,productid=0x0601,id=usb0' -chardev 'socket,id=tpmchar,path=/var/run/qemu-server/100.swtpm' -tpmdev 'emulator,id=tpmdev,chardev=tpmchar' -device 'tpm-tis,tpmdev=tpmdev' -device 'VGA,id=vga,bus=pcie.0,addr=0x1' -chardev 'socket,path=/var/run/qemu-server/100.qga,server=on,wait=off,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:496f3084b2d4' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/zvol/rpool/data/vm-100-disk-1,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,rotation_rate=1,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=B6:7A:6C:6E:5C:ED,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=101' -rtc 'driftfix=slew,base=localtime' -machine 'type=pc-q35-6.1+pve0' -global 'kvm-pit.lost_tick_policy=discard' -S' failed: got timeout
INFO: Failed at 2022-04-21 01:01:25
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[mira]

Do you have additional VMs running? How much free memory do you have?
A timeout issue could be because memory can't be allocated in time because of fragmentation and buffers/cache that needs freeing. It's a common issue with ZFS ARC.

How much memory does the ZFS ARC use? arcstat

 

[slot]

I have only 1 VM

for now when people are working is:

server with proxmox 7:

arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
11:43:52     6     0      0     0    0     0    0     0    0   57G   57G    22G

RAM usage is about 80% (99GB of 125GB)

server with proxmox 6:

arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c 
12:02:39    34     0      0     0    0     0    0     0    0    62G   62G

RAM usage is about 80% (100GB of 125GB)

so its similar and on proxmox 6 I have no issues.

I will check ARC usage at night when backup is making.

But ARC is set dynamic by default? so when VM is idle Proxmox get more free memory.

 

[mira]

ARC by default uses up to half your memory. But compared to kernel buffers/caches, ARC requires more time to be freed when demanded.
This can lead to timeout issues then.

You could try to limit ARC size to less than half your memory: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_zfs (3.8.7)

 

[slot]

I will try, thanks for help.