8.4 / High IO / ZFS or kernel issues?

JordansGhost said:
I didn't try with compression disabled because I rebuilt the volume to LVM-Thin and had the same IO issues. The main nvme also uses the default file system rather than ZFS.
Click to expand...
You can insert another SSD/NVMe and create a pool with no compression enabled and transfer the VM to that pool. Or transfer only the VM's virtual hard drive to its pool (uncompressed).

1745795958857.png
 
I asked Chat GPT and it created a script for me to disable it on all disks. Does this not achieve the same thing as backup and restore?
#!/bin/bash

POOL="zfs-local"

echo "Disabling compression on all VM disks under $POOL..."

# List all ZFS volumes in the pool
zfs list -H -o name | grep "^${POOL}/" | while read -r dataset; do
echo "Disabling compression on $dataset..."
zfs set compression=off "$dataset"
done

echo "All done!"


heres part of the shell output when i ran it

Disabling compression on zfs-local/vm-900-disk-0...
Disabling compression on zfs-local/vm-900-disk-1...
Disabling compression on zfs-local/vm-999-disk-0...
Disabling compression on zfs-local/vm-999-disk-1...
Disabling compression on zfs-local/vm-999-disk-2...
All done!

I also disabled it on the pool for new disks. Overall the IO overhead has dropped but as soon as I try anything on a server it spikes again. I'll migrate a disk to NFS store and back again to ZFS and see if that helps.
 
Last edited:
that definitely seems to have disabled it on all drives. but with ZFS the changes are not applied to files until they are rewritten to disk, so it basically just told it not to compress future files, it doesnt change the current file state.
 
  • Like
Reactions: news and Macross
zenowl77 said:
Isso definitivamente parece tê-lo desativado em todas as unidades. mas com o ZFS as alterações não são aplicadas aos arquivos até que sejam reescritos no disco, então basicamente apenas disse para não compactar arquivos futuros, não altera o estado atual do arquivo.
Click to expand...
Exactly!
zenowl77 said:
that definitely seems to have disabled it on all drives. but with ZFS the changes are not applied to files until they are rewritten to disk, so it basically just told it not to compress future files, it doesnt change the current file state.
Click to expand...
Exactly!
 
ok, thanks.. I wanted to understand if there was a way to disable compression without destroying the pool, I was thinking of using RAID with the ARECA controller, as I had before with VMW, instead of using ZFS, it seems to me that before with VMW the performances were better with the same HDDs
 
There is a tradeoff to pve operating storage design model each which is own pros and cons to performance, features and possibilities.
So it depends what one will use or has to use and important I find how anybody is comfortable with in case of any problems which might come up.
Good luck anyway and have fun with pve etc :)
 
The usage of cpu cycles and needed blocks of the pool to store an initiated write (here by running a vm).
 
so is it better with or without compression? I tried running a vm without compression, but the IO delay remains, it increases a lot when I go to write, while when I read it is low
 
toto-ets said:
so is it better with or without compression? I tried running a vm without compression, but the IO delay remains, it increases a lot when I go to write, while when I read it is low
Click to expand...
Move the vm to a different disk, then if need be back to the original disk. You will see zero changes until data is rewritten to disk. ZFS also in my opinion unless properly tuned with many disks, is better for backups, etc, and not VMs (again unless you have many disks setup properly)
toto-ets said:
what exactly changes if I use zfs with compression, and without compression?
Click to expand...
It shrinks files on disk which causes extra cpu use and sometimes extra IO wait so they take up less space, it will often also make reads faster but writes can be slower.
 
Maybe your SSD entering powersaving mode and reducing performance ?


Code: 
https://nvmexpress.org/resource/technology-power-features/

$> nvme list

# Check status #
$> nvme get-feature /dev/nvme0n1 -f 2 -H

# Limit PowerMangement #
$> nvme set-feature /dev/nvme0n1 -f 2 -v 2
or
# Disable PowerMangement #
$> nvme set-feature /dev/nvme0n1 -f 2 -v 0
 
Last edited:
Just to update on the original issue. After testing a number of different storage types I found two things that have allowed me to work around the issues but keep proxmox.

1) change the SSD from ZFS to BTRFS - this resolved the vast majority of IO issues but retained my capability for snapshots
2) enable SCSI Single on the VMs with only one disk and enabled IO Thread - I dont have numbers but this seemed to help reduce it a little too

Following on from this I also took the opportunity to update my NAS and this helped with shared storage latency (not that I really use this for VMs). The only issue here was I could not figure out how to get ZFS over ISCSI working so everything is using NFS and the underlying OS is BTRFS on the NAS so snapshots still work.
 
JordansGhost said:
Just to update on the original issue. After testing a number of different storage types I found two things that have allowed me to work around the issues but keep proxmox.

1) change the SSD from ZFS to BTRFS - this resolved the vast majority of IO issues but retained my capability for snapshots
2) enable SCSI Single on the VMs with only one disk and enabled IO Thread - I dont have numbers but this seemed to help reduce it a little too

Following on from this I also took the opportunity to update my NAS and this helped with shared storage latency (not that I really use this for VMs). The only issue here was I could not figure out how to get ZFS over ISCSI working so everything is using NFS and the underlying OS is BTRFS on the NAS so snapshots still work.
Click to expand...
Ok... but did you end up testing ZFS without compression? I solved the I/O problem this way. I am also using it without thin provisioning.
 
be careful with running BTRFS with VMs, in my experience it doesn't take to kindly to many small writes/file updates and i have had it crash on me a couple times where i lost everything on the drive.

usually its fine for static files, etc but i haven't had any good luck with using it with VMs, its not as stable as ZFS.

i usually use LVM/EXT4 on NVME/SSDs then ZFS or XFS on HDD.
 
zenowl77 said:
be careful with running BTRFS with VMs, in my experience it doesn't take to kindly to many small writes/file updates and i have had it crash on me a couple times where i lost everything on the drive.

usually its fine for static files, etc but i haven't had any good luck with using it with VMs, its not as stable as ZFS.

i usually use LVM/EXT4 on NVME/SSDs then ZFS or XFS on HDD.
Click to expand...
Appreciate the advice. I'll be prudent on replication and backup just in case. I can cope with that level of RTO / RPO as its a cluster which combines home production and home lab
 
Hi all,

unfortunately I have made the mistake of upgrading from 8.3.5 to 8.4.1 too. I run ProxMox on a Mac mini 2014 for home(lab) use. This has worked fine for months, until the moment the upgrade was done. I see all NIC's stalling every few hours. Since I use pfSense in a VM it also breaks the internet connection for about a minute.

The local storage is a single disk EXT4. For network there are 3 NIC's, one internal Broadcom and two of them are Thunderbolt Ethernet adapters (for pfSense). It all runs on a Mac mini 2014, which was fine on 8.3.5

With 8.4.1 is see all NIC's stalling causing the broken internet connection, time outs of DNS requests, file shares etc.
The log does show this on those moments:

Code: 
pve pvestatd[1197]: status update time (15.251 seconds)

These entries appear two of three times in a row, with delay varying from 5 to 30 seconds.
I can't see a pattern in the time between the huge IO delay causing the NIC's to stall. It doesn't match anything in CRONTAB or other processes I know.

Currently I have pinned kernel Linux 6.8.12-8-pve, but that doesn't seem to help at all.
 
Last edited:
You must log in or register to reply here.
Top Bottom