luks vm backup job thrashes io 90%

[Kodey]

One specific vm always causes the io to start thrashing at 90%.
As soon as the backup job starts, everything slows until it finishes.
It has a small 30G sdd on a host zpool 10 array.
It was imported from an old vbox long ago and there doesn't seem to be anything else wrong with it.
The sdd is luks which may have something to do with it.
I have several other backup jobs for vms on that zpool, none of those involving luks, and they all perform nicely.
They have similar disk configurations, but the disk configuration doesn't seem to matter.
I'm not sure exactly when the problem started. I upgraded proxmox from 6.4 originally and never noticed a problem then.

 

[Kodey]

Some more background information.

$ proxmox-backup-manager versions
proxmox-backup-server 3.0.2-1 running version: 3.0.2
$ pveversion
pve-manager/8.0.4/d258a813cfa6b390 (running kernel: 6.2.16-12-pve)

These are running side-by-side

vm:

agent: 1
bios: seabios
boot: order=scsi1;ide2
cores: 2
cpu: x86-64-v2-AES
ide2: isos:iso/manjaro-xfce-21.0.7-210614-linux510.iso,media=cdrom
machine: q35
memory: 16384
name: ManjoProx
net0: virtio=6E:26:10:B8:06:E6,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-zfs:vm-100-disk-0,aio=threads,backup=0,cache=writeback,discard=on,iothread=1,size=124G,ssd=1
scsi1: zfs16Tr10:vm-100-disk-2,aio=threads,cache=writethrough,discard=on,iothread=1,size=30G
scsi2: /dev/disk/by-id/nvme-Fanxiang_S500PRO_512GB_FXS500PRO232495397-part5,aio=threads,backup=0,cache=writeback,discard=on,iothread=1,size=64G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=6f774a39-5744-4db3-a91a-87e6dd80dd14
sockets: 1
vga: qxl
vmgenid: 7283ee8b-819c-483e-8d93-80c336028d9f