Proxmox x Hyper-V storage performance.

[ucholak]

Update: I rebuilt the VMs using Virtio SCSI, and performance again reached roughly double that of Virtio Block (matching the results seen when the drives were formatted with 512-byte blocks).

This indicates that switching the NVMe block size from 512 to 4096 has minimal impact, or at least an effect that is statistically negligible.

That seems reasonable to me, as modern NVMe controllers likely handle most of the complexity of managing the underlying physical page sizes internally (which are -- actually -- larger than 4K).

yes, but probably it mainly depends on controller. controller can inform you about optimal format. yours is ok with both (512e = 4kn = best).
example of difference from my system:

# nvme id-ns /dev/nvme3n1 -H | grep '^LBA Format'
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best (in use)

i think 4kn on all layers should be default goal in modern setup. it can have also slightly lower performance in some cases (as you tested different combinations).

file-system has (4x?) smaller inode table, fragmentation, hw alignment, compression advantage....

 

[ucholak]

you are welcome.
i forget to mentioned that all my hosts are linux OS (non-windows). i was really curious how windows will stand.

thanks much for quick check/benchmark, it will be helpful for many people.
Virtio SCSI should be used for windows OS.

i have also important information from my team, that has many problems with controllers other than scsi type in windows environment.
cluster services need SCSI persistent reservation feature.

 

[ucholak]

@fdcastel
i have remembered, that before proxmox v8 i need to set also zvol block size, because default was not optimal. from v8 it is optimal for most users (16k).

proxmox uses zvols on zfs pools (default name local-zfs) here blocksize is applied.
if you use zfs dir (default local-dir) here recordsize is applied. maybe you should also try disk on local-dir (raw?).

if you have windows hyper-v environment you can fast check hyper-v/VHD and NTFS block size.
exploring this, or just checking 64k can be hopeful.

i think in vmware environment default VMFS block size is 1MB.
it looks like there is also newer VMDX format. ai: The block size for VHDX files is typically 32 MB.

pvesm set local-zfs --blocksize 64K
mkfs.ntfs --cluster-size 64K...

1. ZFS Volblocksize for VM Workloads
2. ZFS Volblocksize Round 2

Well, it's still pretty confusing. 64k seems to be universally faster than 8k, but probably causes a decent amount of write amplification for small writes typical of VM workloads.

 

[cwt]

64K volblocksize is the sweet spot for Windows VMs execpt database or file servers. Pure AD controllers should also use zvols with 16K blocksize. MS SQL server for example: 1x vssd with 64k for the OS installation only. 2nd vssd for SQL data files with 16k and 3rd vssd for SQL logfiles with 8k. That’s a setup where we received the optimal performance on customer VMs running MS SQL. We also combined ZFS RAID with 2x Optane NVME as a SLOG mirror for „extra firepower“.