Backup of my VMs stalls I/O (network and VM)

[promoxer]

My PVE runs off a mirrored ZFS on 2 SATA SSDs, when my nightly backup job starts, the whole system including the VMs will eventually grind to a crawl.
During this time, even SSH connections have trouble getting through to PVE, however, SSH to VMs are fine.
At the same time, my Windows VM also becomes very unresponsive, in particular the file explorer.

It looks like my I/O is being clogged up, any idea how I can troubleshoot?

 

[sb-jw]

Please post your VM configs (qm config VMID) and more details about your system.

 

[promoxer]

I have mirrored pairs of SATA (for PVE) and NVMe (for VMs)

Below is an extract of my backup output of a VM to PVE's SATA

During the 23% and 28%, those drops results in
1. my Windows VM UI becoming unresponsive
2. unable to SSH to PVE (to stop the backup)
3. able to SSH to Linux VM

it goes back to normal at those speeds when write is above 100MiB/s

2 and 3 are interesting because it seems the network card hardware has the capacity to connect, but just wouldn't connect for PVE

INFO:  21% (13.5 GiB of 64.0 GiB) in 1m 12s, read: 189.7 MiB/s, write: 125.2 MiB/s
INFO:  22% (14.1 GiB of 64.0 GiB) in 1m 17s, read: 137.8 MiB/s, write: 113.6 MiB/s
INFO:  23% (14.7 GiB of 64.0 GiB) in 1m 29s, read: 52.0 MiB/s, write: 45.2 MiB/s
INFO:  24% (15.5 GiB of 64.0 GiB) in 1m 34s, read: 149.3 MiB/s, write: 129.6 MiB/s
INFO:  25% (16.1 GiB of 64.0 GiB) in 1m 38s, read: 152.2 MiB/s, write: 140.2 MiB/s
INFO:  26% (16.8 GiB of 64.0 GiB) in 1m 43s, read: 145.0 MiB/s, write: 132.8 MiB/s
INFO:  27% (17.5 GiB of 64.0 GiB) in 1m 47s, read: 187.6 MiB/s, write: 153.6 MiB/s
INFO:  28% (18.1 GiB of 64.0 GiB) in 2m 6s, read: 31.7 MiB/s, write: 22.0 MiB/s
INFO:  29% (18.7 GiB of 64.0 GiB) in 2m 10s, read: 150.4 MiB/s, write: 123.8 MiB/s
INFO:  30% (19.3 GiB of 64.0 GiB) in 2m 14s, read: 150.4 MiB/s, write: 128.2 MiB/s
INFO:  31% (19.9 GiB of 64.0 GiB) in 2m 19s, read: 126.9 MiB/s, write: 124.7 MiB/s

# qm config 102
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0
cores: 2
cpu: x86-64-v2-AES
efidisk0: vpool:vm-102-disk-0,size=1M
localtime: 1
machine: q35
memory: 4096
meta: creation-qemu=7.1.0,ctime=1681652725
name: homeassistant
net0: virtio=08:00:27:14:FB:5A,bridge=vmbr0
numa: 1
onboot: 1
ostype: l26
scsi0: vpool:vm-102-disk-1,discard=on,size=64G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=ee5003f0-4dcf-45b2-baab-971a086414cb
sockets: 1
tablet: 0
vmgenid: a83d7fb4-8ff2-4cb5-910d-7df2945b1fda
root@pve:/scripts#

 

[sb-jw]

Unfortunately, you continued to withhold the details of your system, but I suspect that you don't need the NUMA flag. Theoretically you could also switch to host CPU.

I have mirrored pairs of SATA (for PVE) and NVMe (for VMs)

Below is an extract of my backup output of a VM to PVE's SATA

Do I understand correctly that you are backing up locally and backing up from an NVMe to an SSD? I'm not surprised that your node then crashes, you can't push the OS disks to the limit and then expect nothing to happen.

You could limit the bandwidth and thus mitigate the effects.
pvesm set STORAGEID --bwlimit KIBs

But you still shouldn't make backups locally, then you can basically leave it alone because if there's a defect, theft or ransomware, they'll be gone.

 

[promoxer]

Unfortunately, I have no idea what you want to see, if there is something specific you want to see, then please tell me what it is. I have already given you `qm config 102`, the other VMs are the same, just different storage capacities.

I have been backing up this way on PVE7 since start of 2022 on my last machine with no issues, this slow-to-a-crawl started only on PVE8 and this new machine.

I also prefer that you don't assume my backups stop locally.

I will give `pvesm set STORAGEID --bwlimit KIBs` a try. Thank you.