Improving I/O on Windows VMs via ZIL/LOG ZFS SSDs to my existing pool

G

Giovanni

Renowned Member
Apr 1, 2009
109
11
83
Looking for some thoughts and feedback before I make any changes to my system.


At home I have proxmox with several VMs, some of them are windows for my homelab testing needs, there is often I/O delays and slowness when updating / installing programs. I have a pair of SSDs on the system already ('rpool') with a different custom partition scheme that I want to dedicate to ZIL/SLOG (dev/sdh3 anddev/sdg3)

This is my zpools:
Code: 
root@pve:~# zpool status
  pool: gdata
 state: ONLINE
  scan: scrub repaired 0B in 19:31:16 with 0 errors on Sun May  9 19:55:18 2021
config:

        NAME                                   STATE     READ WRITE CKSUM
        gdata                                  ONLINE       0     0     0
          mirror-0                             ONLINE       0     0     0
            ata-WDC_WD80EFAX-68LHPN0_y  ONLINE       0     0     0
            ata-WDC_WD80EFAX-68LHPN0_x  ONLINE       0     0     0
          mirror-1                             ONLINE       0     0     0
            wwn-0x5000cca28edd5f75             ONLINE       0     0     0
            wwn-0x5000cca2a3dd6598             ONLINE       0     0     0
          mirror-2                             ONLINE       0     0     0
            wwn-0x5000cca273cf31b3             ONLINE       0     0     0
            wwn-0x5000cca273d46c45             ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 00:03:06 with 0 errors on Sun May  9 00:27:10 2021
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdh2    ONLINE       0     0     0
            sdg2    ONLINE       0     0     0

errors: No known data errors
root@pve:~#


Here is my current "sync" settings for all disks, the windows VMs are using ZFS storage backend on under 'gdata'

ID 103 and ID 113 datasets are windows vms.

Code: 
root@pve:~# zfs get sync
NAME                                              PROPERTY  VALUE     SOURCE
gdata                                             sync      standard  default
gdata/containers                                  sync      standard  default
gdata/data                                        sync      standard  default
gdata/docs                                        sync      standard  default
gdata/fit                                         sync      standard  default
gdata/movies                                      sync      standard  default
gdata/movies@b4rename                             sync      -         -
gdata/music                                       sync      standard  default
gdata/pve                                         sync      standard  default
gdata/pve-vzdump                                  sync      standard  default
gdata/pve/subvol-100-disk-0                       sync      standard  default
gdata/pve/subvol-101-disk-1                       sync      standard  default
gdata/pve/subvol-102-disk-1                       sync      standard  default
gdata/pve/subvol-104-disk-1                       sync      standard  default
gdata/pve/subvol-105-disk-1                       sync      standard  default
gdata/pve/subvol-105-disk-1@before_upgrade        sync      -         -
gdata/pve/subvol-106-disk-0                       sync      standard  default
gdata/pve/subvol-106-disk-1                       sync      standard  default
gdata/pve/subvol-106-disk-2                       sync      standard  default
gdata/pve/subvol-106-disk-3                       sync      standard  default
gdata/pve/subvol-106-disk-4                       sync      standard  default
gdata/pve/subvol-106-disk-5                       sync      standard  default
gdata/pve/subvol-106-disk-6                       sync      standard  default
gdata/pve/subvol-106-disk-7                       sync      standard  default
gdata/pve/subvol-106-disk-8                       sync      standard  default
gdata/pve/subvol-107-disk-1                       sync      standard  default
gdata/pve/subvol-108-disk-0                       sync      standard  default
gdata/pve/subvol-110-disk-1                       sync      standard  default
gdata/pve/subvol-111-disk-1                       sync      standard  default
gdata/pve/subvol-117-disk-1                       sync      standard  default
gdata/pve/subvol-118-disk-0                       sync      standard  default
gdata/pve/subvol-120-disk-0                       sync      standard  default
gdata/pve/subvol-121-disk-0                       sync      standard  default
gdata/pve/subvol-122-disk-0                       sync      standard  default
gdata/pve/subvol-123-disk-0                       sync      standard  default
gdata/pve/subvol-124-disk-0                       sync      standard  default
gdata/pve/subvol-126-disk-0                       sync      standard  default
gdata/pve/subvol-128-disk-0                       sync      standard  default
gdata/pve/subvol-129-disk-0                       sync      standard  default
gdata/pve/subvol-140-disk-0                       sync      standard  default
gdata/pve/vm-103-disk-1                           sync      standard  default
gdata/pve/vm-103-disk-1@dec2020_updates           sync      -         -
gdata/pve/vm-103-disk-2                           sync      standard  default
gdata/pve/vm-103-disk-2@dec2020_updates           sync      -         -
gdata/pve/vm-109-disk-0                           sync      standard  default
gdata/pve/vm-112-disk-1                           sync      standard  default
gdata/pve/vm-116-disk-0                           sync      standard  default
gdata/pve/vm-119-disk-0                           sync      standard  default
gdata/pve/vm-119-disk-0@beforeupdate111720        sync      -         -
gdata/pve/vm-119-state-beforeupdate111720         sync      standard  default
gdata/pve/vm-141-disk-0                           sync      standard  default
gdata/pve/vm-141-disk-0@netdata_sudoers           sync      -         -
gdata/pve/vm-141-disk-0@portainer_installed       sync      -         -
gdata/pve/vm-141-disk-0@portainer_issues          sync      -         -
gdata/pve/vm-141-state-netdata_sudoers            sync      standard  default
gdata/pve/vm-141-state-portainer_installed        sync      standard  default
gdata/pve/vm-141-state-portainer_issues           sync      standard  default
gdata/tv                                          sync      standard  default
gdata/xenu                                        sync      standard  default
rpool                                             sync      standard  received
rpool/ROOT                                        sync      standard  inherited from rpool
rpool/ROOT/pve-1                                  sync      standard  inherited from rpool
rpool/data                                        sync      standard  inherited from rpool
rpool/data/subvol-106-disk-1                      sync      standard  inherited from rpool
rpool/data/vm-133-disk-0                          sync      standard  inherited from rpool
rpool/data/vm-133-disk-0@vanilla                  sync      -         -
rpool/data/vm-133-disk-0@basic_bridge_transition  sync      -         -
rpool/data/vm-133-state-basic_bridge_transition   sync      standard  inherited from rpool
rpool/data/vm-133-state-before_wgscriptexec       sync      standard  inherited from rpool
rpool/data/vm-133-state-beforentopng              sync      standard  inherited from rpool
rpool/data/vm-135-disk-0                          sync      standard  inherited from rpool
rpool/swap                                        sync      always    received
root@pve:~#

Here is my custom partition of my pair of SSDs (installed awhile back and forgot to enable ZIL - but intentionally left partion 3 open and ready for add):

Code: 
root@pve:~# fdisk -l /dev/sdh
Disk /dev/sdh: 238.5 GiB, 256060514304 bytes, 500118192 sectors
Disk model: Samsung SSD 840
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 08D97179-2EC2-4FAD-BDDE-B9F58B687021

Device         Start       End   Sectors  Size Type
/dev/sdh1         34      2047      2014 1007K BIOS boot
/dev/sdh2       2048 156252048 156250001 74.5G Solaris /usr & Apple ZFS
/dev/sdh3  156252049 205080174  48828126 23.3G Solaris /usr & Apple ZFS
/dev/sdh4  205080175 205096559     16385    8M Solaris reserved 1
root@pve:~# fdisk -l /dev/sdg
Disk /dev/sdg: 238.5 GiB, 256060514304 bytes, 500118192 sectors
Disk model: Samsung SSD 840
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 66EB71E4-0357-4197-B034-D8168D6C26B4

Device         Start       End   Sectors  Size Type
/dev/sdg1         34      2047      2014 1007K BIOS boot
/dev/sdg2       2048 156252048 156250001 74.5G Solaris /usr & Apple ZFS
/dev/sdg3  156252049 205080174  48828126 23.3G Solaris /usr & Apple ZFS
/dev/sdg4  205080175 205096559     16385    8M Solaris reserved 1

I'm reading https://linuxhint.com/configuring-zfs-cache/ as a guide, it seems to recommend to add ZIL devices to my pool, in my case add to 'gdata' pool :

$ zpool add gdata log mirror sdg3 sdh3

The guide also recommends changing ZFS settings for zfs volume used by VMs (currently set to sync=standard) to sync=always -- is this recommended to achieve my goal of improved I/O speeds on Windows guests when there is heavy disk writes?

If there's anything else I am missing or that I should do (such as a proxmox setting in case of creating future VMs on ZFS filesystem to always use 'always') let me know
 
The setting "sync=always" will force all the fast but not that secure async writes to be executed as slow but secure sync writes. You can force everything to be a sync write so that it can be cached but that don't have to be faster. You can test if it helps in your case but as far as I know this will most of the time slow down everything.

Also a write cache will kill your SSDs super fast so its not a good idea to use a partition on the SSDs you are using for your root pool (rpool). Moreover it will cause a lot of IO on your root pool and that might drop the performance of all VMs when every VM needs to wait for proxmox to do something but proxmox is stuck because the SSD is busy.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom