Average "Load Average" with ZFS (ZVOL)

[melanch0lia]

Hello,

Guys, share your average "Load Average" or iowait on production host with ZFS (ZVOL) as storage for 10+ VMs?

Also, what "Load Average" should be considered as critical or problematic?..

 

[melanch0lia]

Also, should cache=writeback and discard=on be used with ZVOL?

 

[mir]

discard=on be used with ZVOL?

Actually, you should never use discard=on since this dramatically reduce performance. Use a tool to manually run discard from cron.

 

[LnxBil]

@mir I think he meant discard=on on PVE, not as ext4 parameter, but you're totally right.

@melanch0lia: What does VM 10+ mean? Depends heavily on what they're doing. I'm running really a lot of VMs, but mainly they're idling almost all the time and then comes the $sqlserver, which has heavy load. I does not have any big iowait, because my pool consists of only enterprise SSD, so I'm not a great help on this topic. Yet I can say that our backup server has a big iowait time when performing backups, but that wasn't different when it wasn't on ZFS in previous years.

 

[melanch0lia]

@mir Thanks.

@LnxBil yeah, I mean discard=on on PVE.

@melanch0lia VM 10+ means more than 10 VMs regardless of the load. I don't have any big iowait as well (0-8% depending on load) and load average 2-5, but I have some peaks to 7-10 while I'm doing dd and something else or just booting up Proxmox and starting VMs.

Also, arcstat show misses on write. Should I add some SSD for ZIL?

 

[mir]

yeah, I mean discard=on on PVE.

I do not have the discard option enabled in any of my VM's and never the less each time I run fstrim inside the VM's the thin provisioned zvol's size decreases with the amount reported by fstrim. So I think the discard option is only in use if the storage backend is Qcow2?

 

[LnxBil]

I do not have the discard option enabled in any of my VM's and never the less each time I run fstrim inside the VM's the thin provisioned zvol's size decreases with the amount reported by fstrim. So I think the discard option is only in use if the storage backend is Qcow2?

I haven't tried that one before but you're right. Even for normal SATA it works out of the box. Maybe some "behind-the-scenes" magic.

 

[mir]

For Comstar iscsi this is because Comstar supports the iscsi command unmap and is parsing this to the zvol. When zvol receives an unmap and provided the zvol is thin provisioned then the block(s) is freed and the zvol shrinks accordingly.