VM Hang during high data transfer

[casalicomputers]

Hi,
we have a Proxmox server built with ZFS that hosts various vms, including one that is a Debian Bookworm with a Samba domain controller.

We experienced a weird problem where if we transfer data greater than 5GB, the vm could completely hang and become unresponsive, even from the built-in Proxmox console.

Is there something we could to to understand the root of the problem?

Thanks a lot for the support!

 

[shanreich]

Can you post the VM configuration (obtainable via qm config <VMID>)? The storage configuration would be interesting as well (located in /etc/pve/storage.cfg)

 

[casalicomputers]

We just discovered this that could be helpful, while the vm was hanging, ALL the Linux vms inside the same node logged this in the dmesg:

[ 2055.223604] INFO: task btrfs-transacti:399 blocked for more than 120 seconds.
[ 2055.224006]       Not tainted 6.1.0-9-amd64 #1 Debian 6.1.27-1
[ 2055.224323] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

This is captured from the that vm, but the same message about a different process appear on all the linux vms.

 

[casalicomputers]

Can you post the VM configuration (obtainable via qm config <VMID>)? The storage configuration would be interesting as well (located in /etc/pve/storage.cfg)

qm config 100:

agent: 1,fstrim_cloned_disks=1
boot: order=scsi0;ide2;net0
cores: 4
ide2: none,media=cdrom
memory: 4096
meta: creation-qemu=7.0.0,ctime=1680183385
name: dc01
net0: virtio=72:01:F6:5C:1B:5A,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
parent: pre_update
scsi0: raid5-sas-ssd:vm-100-disk-0,cache=writeback,discard=on,size=30G
scsi1: raid5-sas-ssd:vm-100-disk-1,cache=writeback,discard=on,size=8G
scsi2: raid5-sas-hdd:vm-100-disk-0,cache=writeback,discard=on,size=1000G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=9a7645f9-2167-4ecc-85bc-aeef689d7415
sockets: 1
vmgenid: 680c4fb6-a71d-4e64-97f1-faa5df00af3b

cat /etc/pve/storage.cfg

dir: local
    path /var/lib/vz
    content iso,vztmpl,backup

zfspool: raid5-sas-ssd
    pool rpool/data
    content rootdir,images
    mountpoint /rpool/data
    sparse 1

cifs: bckpve
    path /mnt/pve/bckpve
    server 10.100.98.250
    share bckpve
    content backup
    prune-backups keep-last=2
    username ****

zfspool: raid5-sas-hdd
    pool raid5-sas-hdd
    content rootdir,images
    mountpoint /raid5-sas-hdd
    sparse 1

pbs: bckpbs
    datastore bckpbs
    server 192.168.1.41
    content backup
    fingerprint ****
    prune-backups keep-all=1
    username ****

pbs: bckrdx
    datastore bckrdx
    server 192.168.1.41
    content backup
    fingerprint ****
    prune-backups keep-all=1
    username ****

 

[shanreich]

Can you try turning on the option 'IO Thread' for all your disks, as well as change the Controller to 'VirtIO SCSI Single'?

This should have the effect that each disk gets its own I/O thread rather than the QEMU process handling I/O (which can lead to freezes like you are experiencing).

 

[casalicomputers]

Can you try turning on the option 'IO Thread' for all your disks, as well as change the Controller to 'VirtIO SCSI Single'?

This should have the effect that each disk gets its own I/O thread rather than the QEMU process handling I/O (which can lead to freezes like you are experiencing).

We applied the change, we do not get that warning anymore, but problem still persist, all the vms become unresponsive while moving large files into the vm, even tho when we stop the transfer the system started working again after some time, so look like there is still an improvement.
We applied the change only to the linux vms, do we need to do the same also on the windows ones?

 

[_gabriel]

ssd model ?
be careful if zfs is behind raid5 hw controller