8.4 / High IO / ZFS or kernel issues?

J

JordansGhost

New Member
Nov 1, 2024
12
0
1
I have upgraded my cluster to 8.4 and since then all my VMs are running badly with 50-80% IO delay on the nodes. I have read that this issue doesnt happen with kernel 6.2 si I might need to downgrade just to get it working again which will then allow me to troubleshoot the issue.

Can someone point me in the right direction to running 6.2 or 6.5 Kernels.
 
pretty sure its just install new version:
apt install proxmox-kernel-6.*

then:
apt remove proxmox-kernel-6.*
to remove current version
(replace * with version number obviously)

but i am unsure if this is right for your case or the implications for compatibility, etc, may want to wait for someone who knows more about the subject and the specifics of trying to install an older version of the kernel on a more current proxmox version, there may be complications there.
 
Last edited:
Thanks. After more testing the kernel doesnt seem to help as the issue appears to be related to ZFS. If I remove the ZFS pool and use just LVM-Thin all the IO issues go away and my system is alive again. As soon as I deploy a ZFS pool in the cluster it runs like garbage. But prior to the upgrade it was working just fine. Fresh install on one of the nodes reproduces the same issue and same fix. Now I am testing deploying LVM-Thin on my 1TB SSD but its taking over 50 mins to run so far.
 
You’re welcome, ZFS can certainly be a pain, i have IO wait issues and stalls with mine too, not bad enough to bog the whole system down completely thankfully but it certainly would if i ran it off of ZFS (VMs sure run like garbage off of it), but i am only running it on a single drive.

I am hoping the coming ZFS 2.3 with Direct I/O and fast dedup fixes issues like this, because it really seems to be a potential buffer issue, i cannot for the life of me figure out how to fix but then again i am no expert on ZFS.
 
JordansGhost said:
Thanks. After more testing the kernel doesnt seem to help as the issue appears to be related to ZFS. If I remove the ZFS pool and use just LVM-Thin all the IO issues go away and my system is alive again. As soon as I deploy a ZFS pool in the cluster it runs like garbage. But prior to the upgrade it was working just fine. Fresh install on one of the nodes reproduces the same issue and same fix. Now I am testing deploying LVM-Thin on my 1TB SSD but its taking over 50 mins to run so far.
Click to expand...
Do the test with DD with compression enabled (this can be any algorithm - zstd, lz4, gzip, etc) and then repeat the test without compression. Both on the host and on the VM, if you are using ZFS on the VM as well.

# dd if=/dev/zero of=/rpool/file.out bs=4096 count=10000000 status=progress
# dd if=/rpool/file.out of=/dev/null bs=4096 status=progress

PS: Just be careful with the DD command. Not to direct the output to the wrong place (devices, etc).
 
waltar said:
That's useless as zeros are compressed to nothing if enabled so take /dev/urandom as "inputfile".
Click to expand...
Ok! Thank you for the correction. I did so according to a post about performance of multiple RAID systems in ZFS on FreeBSD.
I'm going to use urandom from now and check the results.

In fact, even using zeros, I got very different results in read and write rates. I turned off the compression and read rate increased a lot. But I wish it works with compression enabled.

I can post the screenshots of the tests if you want to analyze them. It will be a pleasure to collaborate and help.
 
If you want ... any performance numbers are anytime to general interest for all to any config to compare withself.
 
Hello Waltar.

The following is a screeshot of a virtual machine with FreeBSD 13.5 on a Proxmox 8.4.1 host using a pool with uncompressed ZFS. In fact, booting our VMs uncompressed into ZFS on the host has greatly improved boot performance. Still, we were experiencing slowdown issues with virtual machines, especially those with FreeBSD.

We performed several tests and came to the conclusion that with the compression turned off and using virtual SCSI (not VirtIO Block) disks, it was the best we could get to improve the read rate. All tests were performed on an HP Proliant 380 G9 (HBA mode) with 4 NVMEs in RAIDZ-2.

I am available to perform any tests on the host as well as on the FreeBSD virtual machine that you or another member suggests. I reiterate that slowdowns occur in any type of virtual machine (Linux, Windows, FreeBSD) that we use. Especially on the compression-enabled FreeBSD (zstd) VM.
 

Attachments

  • VM_FreeBSD_13.5_.PNG
    VM_FreeBSD_13.5_.PNG
    125 KB · Views: 18
Try with lz4 instead of zstd (=zstd-3 I mean is default level from 1..19) as zstd needs more cpu power for compress and decompress.
Even your freebsd vm's maybe use inside zfs too which gives you a cow on cow overhead ?
 
To my original post. Further testing has shown 8.4 has dreadful performance on the internal SSD (1TB consumer grade as its a small 3 node cluster system in the house) however it has worked fine for 6 months. It does not matter whether I format it ZFS or LVM-Thin. I get 40-60% IO issues and the server is slow to being unusable with just one VM on the disk.

However, the internal NVME has no performance issues when I move the VM onto this storage. All the IO issues go away.

So today I rebuilt one of the nodes to Windows 2022 server so i could run the micros/crucial SSD diagnostics and it shows the drive as totally healthy. No firmware updates required.

As further test I installed Hyper V and spun up two servers on the same SSD. No issues whatsoever with performance.

So have boiled this issue down to something specific with proxmox, 8.2 and above running on my opiplex, 6 core, 32 gig home lab on crucial SSD. Unless anyone has an idea as to why this is suddenly an issue when the hardware is clearly fine, I guess ill just have to convert to hyper V (which I really dont want to do)
 
waltar said:
Try with lz4 instead of zstd (=zstd-3 I mean is default level from 1..19) as zstd needs more cpu power for compress and decompress.
Even your freebsd vm's maybe use inside zfs too which gives you a cow on cow overhead ?
Click to expand...
Yes, I did a test previously with lz4 (screenshot attached), but the read rate was still terrible. The problem is with the reading rate. I accept the drop in write rate. In fact, the best arrangement so far has been to disable ZFS compression on both the host and the guest (FreeBSD VM).

In fact, as mentioned, the other VMs with Linux and Windows load much faster without compression in ZFS on the host. Especially with VMs with FreeBSD with ZFS inside. About the cow on cow overhead, I don't know how to answer you about this... But, would it still affect the reading rate so much?

The next test I'm going to perform is to passthrough two SSDs or NVMEs in RAID-1 to the FreeBSD VM.
 

Attachments

  • VM_FreeBSD_LZ4.PNG
    VM_FreeBSD_LZ4.PNG
    62.9 KB · Views: 4
JordansGhost said:
To my original post. Further testing has shown 8.4 has dreadful performance on the internal SSD (1TB consumer grade as its a small 3 node cluster system in the house) however it has worked fine for 6 months. It does not matter whether I format it ZFS or LVM-Thin. I get 40-60% IO issues and the server is slow to being unusable with just one VM on the disk.

However, the internal NVME has no performance issues when I move the VM onto this storage. All the IO issues go away.

So today I rebuilt one of the nodes to Windows 2022 server so i could run the micros/crucial SSD diagnostics and it shows the drive as totally healthy. No firmware updates required.

As further test I installed Hyper V and spun up two servers on the same SSD. No issues whatsoever with performance.

So have boiled this issue down to something specific with proxmox, 8.2 and above running on my opiplex, 6 core, 32 gig home lab on crucial SSD. Unless anyone has an idea as to why this is suddenly an issue when the hardware is clearly fine, I guess ill just have to convert to hyper V (which I really dont want to do)
Click to expand...
Hello JordansGhost.

I think I should have opened a topic for my question and not inserted it in your post! I'm sorry.

But, did you test with the compression on ZFS turned off? I had significant gains with the compression turned off in ZFS on the host. In my case, especially with a VM with FreeBSD+ZFS. But, I also noticed a big improvement in the VM with Windows Server 2025. The reading rate has improved a lot!

I recommend testing with CrystalMarkDisk ( https://crystaldiskmark.org/ ) on the Windows Server VM with compression on the host ZFS enabled and then disabled. After dumping/restoring the respective VM, of course.
 
JordansGhost said:
To my original post. Further testing has shown 8.4 has dreadful performance on the internal SSD (1TB consumer grade as its a small 3 node cluster system in the house) however it has worked fine for 6 months. It does not matter whether I format it ZFS or LVM-Thin. I get 40-60% IO issues and the server is slow to being unusable with just one VM on the disk.

However, the internal NVME has no performance issues when I move the VM onto this storage. All the IO issues go away.

So today I rebuilt one of the nodes to Windows 2022 server so i could run the micros/crucial SSD diagnostics and it shows the drive as totally healthy. No firmware updates required.

As further test I installed Hyper V and spun up two servers on the same SSD. No issues whatsoever with performance.

So have boiled this issue down to something specific with proxmox, 8.2 and above running on my opiplex, 6 core, 32 gig home lab on crucial SSD. Unless anyone has an idea as to why this is suddenly an issue when the hardware is clearly fine, I guess ill just have to convert to hyper V (which I really dont want to do)
Click to expand...
Oh, and I remind you that in Windows Server there is no compression enabled by default!
Don't give up on Proxmox!

The staff has done an extraordinary job and we are confident that if there is indeed a problem, they will solve it!
We must support them, including by subscribing to the system!
 
I also experience a lot of delay in my nodes with ZFS, I would like to try to disable compression, but I don't know the command to send, can someone tell me how to do it? but in my pool there are already 20 active VMs, what can I do? but what is compression actually for?

IO delay increases a lot when starting backups especially
 
Last edited:
I didn't try with compression disabled because I rebuilt the volume to LVM-Thin and had the same IO issues. The main nvme also uses the default file system rather than ZFS.
 
toto-ets said:
I also experience a lot of delay in my nodes with ZFS, I would like to try to disable compression, but I don't know the command to send, can someone tell me how to do it? but in my pool there are already 20 active VMs, what can I do? but what is compression actually for?

IO delay increases a lot when starting backups especially
Click to expand...
You can disable compression at any time, however, VMs that were in the pool before will have no effect as they have already been stored in the pool with compression enabled. You need to dump and restore via vzdump, PBS, migration, etc of the VMs.

To find out if compression is enabled (default) and which algorithm you are using, use the following command:
# zfs get compression <pool-name>

The command to disable compression in ZFS is:
# zfs set compression=off <pool-name>

Test with a VM and run the read and write tests. Getting positive results (without compression) do the same in the others. In my case, I had significant gains in the reading rate.

You can also dump (backup) all VMs and rebuild the pool again without compression and restore them.


1745794961900.png
 
You must log in or register to reply here.
Top Bottom